The BBC's new documentary series, Putin, Russia & The West, was more than two years in the making – much of it spent trying to persuade senior Russian politicians to take part.

This style of documentary relies heavily on eye-witness accounts – insiders recalling the key moments in diplomatic battles and international crises. We needed as many top Russians (plus Americans, British, French, Germans and others) as we could muster. And we needed them twice: once to tell us off the record what they remembered, and again – after we squared their accounts with the other side – to retell the stories on camera. That way we could build up a fair, balanced account of the last decade's diplomatic encounters, as the West came to terms with the tough man who came to power in the Kremlin in 2000, Vladimir Putin.

Persuading the Americans was easy. Condoleezza Rice, Colin Powell, and a host of other top politicians and advisers spoke to us several times and at length, giving us remarkably frank accounts of their meetings with Putin and senior Russians.

It was principally my job, as chief consultant on the series, to persuade the Russians to take part. For the previous three years I had worked as an adviser to Putin's press secretary, Dmitry Peskov, and I embarked on the project feeling sure I could deliver every Russian up to and including "the Boss", as Peskov calls him. Almost all the media advice I had given him over the three years had boiled down to one thing: if you think you are getting a bad press in the West, open up, engage with journalists, give interviews and let them understand you better.

The first shock came right at the beginning – before we had done a single interview – when Peskov told me there was no way Putin (who had now "demoted himself" to prime minister) and president Dmitry Medvedev would both appear in the same series. Why not, I asked. "Because I will make sure it will never happen," came Peskov's reply.

He didn't trust the BBC, he said, and was sure that we would "compare" them, and he did not want that to happen.

It was a bad start. But over the next year we made huge progress. Peskov did help us gain interviews with most of the other Russian politicians we needed. We also made tentative approaches to Medvedev's own press secretary, Natalia Timakova, who I knew was not on good terms with Peskov, hoping that at the right moment – once we had Putin in the can – we would persuade her that Medvedev must also be in the film, despite Peskov's objections.

Slowly, Peskov came round to the idea of his boss giving us the interview we needed. We submitted detailed questions, discussed them over many hours, and even pencilled in a date for the interview to be filmed – April last year.

But that was precisely the moment when Russia entered a period of uncertainty, as Medvedev began making a pitch to remain as president for a second term, and Putin, evidently, was plotting ways to prevent that from happening. Now Peskov came up with excuse after excuse: he didn't want Putin to appear in a film about the past (a clear hint that Putin was looking to a long future in power); he didn't want him appearing alongside "has-beens" like Rice and Powell; he didn't want the great man to be a mere talking head on a par with diplomats and ministers.

At least he did not mention Medvedev any more, so we began to step up our approaches to Timakova – who turned out to be even more unco-operative than Peskov. She refused to meet us, or even to reply to our letters and emails.

I suddenly realised that we were caught in the very worst moment of recent Russian history – nobody wanted to go on record about anything, because nobody knew what the next few months would hold. Would Timakova still have a job next year? She did not know. Neither she nor Peskov, nor any other adviser dared put a toe out of line. Such was (and is) the atmosphere of fear in Putin's Russia.

AS it turned out, we got neither Putin nor Medvedev, but still managed to piece together an amazingly fresh and new history from interviews with their closest advisers – and from the Americans. The series casts fresh light on many hidden episodes. I particularly love the moment where Condoleezza Rice describes a tense meeting with Putin where they argued over the right of Georgia and Ukraine to join Nato. Putin suddenly jumped to his feet to make his argument more forcefully; Rice reflexively also stood up – and, in her high heels, found herself towering over Putin. "It was not a nice moment," she told us.

In another telling episode, we were able to gauge just how close the Russians and Americans were in the first year or so of Putin's rule. Rice and her Russian opposite number, Sergei Ivanov, both told us with delight about an evening in St Petersburg in May 2002. Putin was showing Bush around his home town, and took him to see the ballet The Nutcracker. Rice and Ivanov were with them. Ivanov turned to Rice and asked: "Do you really want to watch this? I have a much better idea - Have you heard of the Eifman ballet?"

Rice had indeed heard of the avant-garde choreographer and was delighted to break free. The pair shook off their bodyguards and snuck off to watch a rehearsal at the Eifman studio, as the sole guests. "I could see she loved it," Ivanov told us. "You can't fake that sort of thing."

They got back to The Nutcracker just as the lights went up, and joined the official delegations for a midnight canal trip around St Petersburg.

It was above all Putin's backsliding from democracy that put an end to the honeymoon.

Our work on the series was an excellent reminder of how far behind the West Russia remains in terms of understanding how the free media works, despite the best efforts of the Kremlin's Western PR experts to school them.

When I began working with them as an adviser in 2006 I was astonished to hear Peskov and his colleagues indicate that we should not merely lean on Western correspondents to give more positive coverage of Russia, but if necessary buy that coverage! They were convinced that a positive editorial in a respected paper such as the Wall Street Journal, for example, had a certain price – just as they were used to paying journalists off in Russia.

Putin's view of the media appears to come straight from the KGB handbook. The Soviet communist party used to regard the press as a "transmission belt" – a means for the party to control society. Putin says he thinks television should merely "explain" government policies. There is no suggestion that ideas should be freely debated and dissected before they become government policy, or that the wisdom of government decisions should be challenged by the opposition.

While working on the television series, and researching the accompanying book, I came across many examples of Putin's skewed view of the West – a view that makes it hard to believe he would ever turn Russia into a democracy. I get the impression that he sincerely believes his form of rule is as democratic as the West, because he simply does not understand how the West works.

For example, he once replied to President George W Bush's criticism of the lack of media freedom in Russia by saying that he knew Bush had ordered the sacking of a CBS news anchor, Dan Rather, for criticising him. Bush explained that it wasn't possible for a president to sack a private company's newscaster, but Putin would have none of it.

He went on to argue that Americans didn't even elect their presidents – it was done by an electoral college. Bush leaned close to him and said: "Vladimir, please don't repeat that in public, you'll only show that you don't understand our system at all."

Putin also falls for bizarre conspiracy theories. He once said that the war between Georgia and Russia in 2008 was started to help John McCain in his presidential campaign against Barack Obama. He complained to Bush that America had special poultry factories producing substandard chickens for export to Russia. Presumably these things remind him of the kind of thing the KGB would do.

In Putin's vision of democracy, nothing happens without his personal intervention or supervision. When wildfires devastated many villages in the summer of 2010 he had CCTV cameras installed so that he could monitor the rebuilding work from his desk. He has Cabinet meetings televised – ostensibly as a sign of democracy, but in fact it stifles meaningful discussion and allows him to browbeat colleagues in public. I once saw him change the purchasing policy of Aeroflot, the state airline, in the course of a few seconds. He told the director general of Aeroflot – as TV cameras rolled – that he wasn't buying enough Russian-made airliners. "But we are buying them,' the director said. "Not enough of them," retorted Putin. "OK," said the director, "we'll start now. I'll report back." "Good," said Putin.

This is what Putin calls the "vertical of power". Originally it was a political realignment intended to centralise power and prevent Russia's disparate regions from flying off into their own orbits. But now it has become a parody of itself. Putin controls not just the politics, but almost everything else too. So it was little wonder that his aides got cold feet when dealing with the BBC's requests for interviews.

All we wanted was for witnesses to tell the inside story about events that were already in the past. We did not intend to misrepresent anyone. The result is a fascinating tale of Russia's failure, over the past 12 years, to make friends with the West – and the West's failure to bring Russia in from the cold.

Angus Roxburgh's book The Strongman: Vladimir Putin And The Struggle For Russia, is published by IB Tauris. The BBC series Putin, Russia & The West, begins on Thursday, on BBC2 at 9pm.

Angus

Roxburgh

spent three

years working for the Russian

president and was

chief adviser on a major documentary season about the man behind the myth.

This is his fascinating behind-the-scenes story

of how the film was made

'I get the impression that Putin sincerely believes his form of rule is as democratic as the West, because he simply does not understand how the West works'