A generation of spies had lost an enemy but not yet found a role. Then came 9/11.

A decade after the dissolution of the Soviet Union, al-Qaida’s devastating attacks on New York and Washington on 11 September 2001 announced to the Central Intelligence Agency (CIA) that the long era of battling communism, for which it had been founded, was officially at an end.

It was to be replaced by a no-holds-barred “war on terror” that would leave the CIA with a soiled reputation over falsified intelligence that led to the invasion and occupation of Iraq, and an extraordinary rendition program in which suspects were subjected to “enhanced interrogations” at secret sites that official US inquiries subsequently defined as torture.

The US intelligence community was dramatically transformed after two planes flew into the World Trade Center, a third smashed into the Pentagon, and a fourth crashed in the Pennsylvania countryside after passengers overcame the hijackers.

Accounts of what unfolded inside the CIA’s headquarters in Langley, Virginia, suggests an organization unprepared for the drastically changed world the attacks ushered in, despite an awareness among some in the intelligence community that al-Qaida and its leader, Osama bin Laden, represented a clear and present danger.

Margaret Henoch, then a counter-intelligence officer with little knowledge of terrorism, recalls confusion, chaos and an initial lack of direction from senior leadership as staff struggled to process what seemed like a bolt from a clear blue sky on that balmy late summer day.

“One of my officers came in and said to me: ‘A plane just ran into the World Trade Center,’” she said. To this day, Henoch – who had previously been stationed in Vienna, where she focused on the International Atomic Energy Agency and nuclear threats – is haunted by the apparent vacuum that followed.

About 40 minutes after the first attack, with no clear direction from the top of CIA, she decided to go up to the seventh floor, which accommodated the agency’s senior figures, including its then director, George Tenet.

To her surprise, the area was deserted, apart from three assistants.

“It was reasonable that they would have moved those guys someplace else,” said Henoch. “What was unreasonable, in my humble opinion, was that no one thought about all those people sitting in that building. Because if the Pentagon was hit, there was a chance that we, too, were a target – and it would be nice to believe that the people who left the building rapidly had a thought for all those who worked for them.

“It was not clear that they did.”

The Guardian contacted Tenet, now working in the private sector, for comment. In an emailed response, Bill Harlow, the CIA’s public affairs director at the time of the attacks, cited details in the 2007 memoir, At The Center Of the Storm, which he co-wrote with Tenet, and said orders were issued for the majority of the staff to leave the building at “about 10am” – nearly an hour after the second plane struck at 9.03am.

Harlow added: “We were aware at the time of intelligence gathered around 1995-96 of terrorist desires to attack CIA HQ with an airplane.

“Director Tenet and the senior leadership moved to another building on the compound that presented a lower profile and would have a greater chance of survival if we were hit by a plane.

Notably, a decision was made that the personnel of the counter-terrorism center remained in the original headquarters building throughout that day … accepting the risk. The senior leadership team returned to the headquarters in the middle of the afternoon on 9/11.”

‘I had a briefing on who the hell al-Qaida was’

Some time afterwards, an order was given for the thousands of personnel in the building to evacuate. Some, including Henoch, were told to relocate to alternative sites to address the emergency.

But by the time of that instruction, nearby roads were choked with traffic as commuters returned home from Washington DC after being released from work early. That prevented agency staff from departing, and triggered fears that they could be sitting ducks for the doomed fourth plane, which was already known to be in the air.

“Large segments of the CIA operating staff were trapped,” Henoch said. “One guy, who was senior to me, was standing next to me in the car park and he said, ‘I have served in multiple war zones. I was a combat soldier in Vietnam and I’m going to die in the parking lot of the CIA.’”

The dysfunctional scene was soon replaced by one of much greater conviction – leading to permanent changes in US intelligence.

“I just remember loads of cable traffic coming in that said: it’s got to be al-Qaida,” said Henoch. “It wasn’t my world, so I had to have a quick briefing on who the hell al-Qaida was, because I just didn’t know anything about it.

“The change was immediate. The next morning, everybody at the Central Intelligence Agency had to know who they were, and how it intersected with everything you were doing.”

Steven Cash, a former CIA counter-terrorism officer who had left the agency to join the staff of the US Senate intelligence committee six months beforehand, faults the CIA’s pre-9/11 terrorism strategy as being too narrowly focused “looking to find bad guys” while neglecting the “political dimension”.

“There was an emphasis on looking a terrorists as if they were a motorcycle gang,” said Cash, who had specialized on the Lebanese Shia group, Hezbollah, a close ally of Iran. “A decision was made in the first [George HW] Bush administration to take terrorism away from the people who looked at geographic areas and separate it out. [It was] illustrative of a mistaken way of looking at the world, because terrorists don’t exist in a vacuum.”

Cash moved on to focus on the Balkans before leaving the agency, but still harboured guilt after the attacks. “Was this our fault? Did we fail? That kind of guilt felt, at least for me, really personal.”

Cash had been on Capitol Hill, attending a committee meeting, when news of the attacks filtered through. “Somebody said, put on the TV and we saw the second plane hit,” he said. “And then they started evacuating the building.

“It was a mess. People were in the hall screaming, running around. The third plane had hit the Pentagon by that point, because we could see smoke across the National Mall.”

The following day, Cash and his then boss, the late Democratic senator Dianne Feinstein, visited CIA headquarters and found a mood of determined resolve. “It was all hands on deck. The people I knew were getting deployed very quickly to Afghanistan,” he recalled. “These people are pros. They didn’t care whether they were blamed or not. They wanted to do something.

“I have a little bit of survivor guilt here. Maybe I should have been shipped out to Afghanistan.”

Political pressure

On their visit to CIA headquarters on 12 September, senators asked agency operatives what legislation and resources they would need to respond to the attacks of the previous day, according to Cash.

Two early – and controversial – results were the USA Patriot Act of 2001, which expanded law enforcement and surveillance powers, and the Homeland Security Act of the following year, creating the US Department of Homeland Security, which houses agencies including Immigration and Customs Enforcement (ICE).

Later changes established the Office of the Director of National Intelligence, with the goal of coordinating 18 different intelligence bodies and eliminating the lack of coordination widely held to blame for warning signs of the 9/11 plot going unheeded.

“DNI was formed to make sure that we actually share everything with everybody. We don’t just keep it inside CIA,” said Susan Miller, a retired CIA counter-intelligence officer.

“What’s really cool is we’ve changed how we do things across the board, not just for terrorism. It’s really improved things. There was no coordination before.”

Yet the reaction to the attacks changed intelligence in another way – by making it more susceptible to political pressure, as President George W Bush’s administration searched for information that could be used to justify an invasion of Iraq to topple Saddam Hussein.

In September 2002, Henoch – who had previously studied Russian weapons capabilities working at the Stanford Research Institute – was asked to assess intelligence provided by an Iraqi defector codenamed Curveball, who had sought asylum in Germany, and provided its intelligence agency with what he said was proof of Saddam’s biological weapons program.

She quickly concluded the claims were fabricated, and passed her conclusions up the chain of command. They were not well received. When Colin Powell, the then US secretary of state, visited the CIA months later to review the agency’s intelligence on Iraq, Henoch and her fellow doubters were not informed. Powell’s subsequent speech to the United Nations security council in February 2003, in which he made the case for war, relied heavily on Curveball’s testimony – despite Henoch’s insistence when the address was checked with her and other officers beforehand that his assertions were not credible.

“We took a look at the Powell speech and I said: ‘You can’t use this. We think this is garbage,” she said. “I did write somewhere – I can’t remember on what page – ‘How can you send someone to die for something like this? No one ever answered that for me.

“The next thing I heard was, ‘Well, when we get in there, we’ll find something.’”

After the invasion, inspectors found no evidence of biological weapons. Curveball was later debunked as a fraud.

Miller, too, encountered the administration’s zeal for intelligence that could justify invading Iraq when she was posted to Prague as CIA station chief in 2002. Czech intelligence had investigated – and, initially, tentatively confirmed – a claim that one of the 9/11 ringleaders, Mohammed Atta, had travelled to Prague the previous year and met an Iraqi embassy official, Ahmad Samir al-Ani, months before the attacks.

The report was seized on by the hawkish US vice-president, Dick Cheney, as evidence of Saddam’s possible links to the attacks, thus providing another justification for war. But Czech intelligence officials retreated from their earlier stance, and the CIA concluded there was no evidence that the meeting happened.

Nevertheless, Donald Rumsfeld, then the US defense secretary, brazenly tried to revive it when Miller briefed him – along with George W Bush and Condoleezza Rice, the national security adviser – when they visited the Czech capital for a Nato summit in 2002.

“‘Young lady, you forgot about this other thing’,” Rumsfeld told Miller, she said. “I played dumb and said: ‘What other thing?’ He replied: ‘Mohammed Atta was here in Prague meeting the Iraqi consul.’

“Bush was looking really uncomfortable and Condi was looking straight ahead. I simply said, ‘Mr President, Madam Secretary, and Mr Rumsfeld, the CIA, the Czech service and I don’t make things up. We give you the facts, and the fact is that he was not here.’

“Bush and Rice nodded in acceptance and the ambassador supported me. But Rumsfeld threw his pen down in disgust.”

The invasion the neocons craved

Whether the supposed meeting happened or not, Rumsfeld and his neoconservative allies got the invasion they craved – leading to a protracted conflict that tarnished the CIA’s reputation and whose length, along with the coinciding US-led military campaign in Afghanistan, spurred popular opposition to “forever wars”.

Despite the blow to the agency’s standing, Cash insisted that its intelligence-gathering and functioning radically improved, thanks to the changes instituted after the attacks. “We have improved our intelligence capabilities significantly from what they were before 9/11,” he said, citing the success in preventing repeat attacks on US soil.

But with the Trump administration now engaged in a war with Iran, roiling the Middle East, such security could no longer be guaranteed, he believes.

“I’m not saying it was a great time,” Cash said, of the CIA’s record in the months and years after September 2001. “But the sense of purpose of being at a place in history where we – as a group, as a society – can do something, was pretty intoxicating.

“I was very proud to have been part of it. It sucked – but proud to have been part of it. And I think everything we’re seeing now is the antithesis of that.”