“Stand back! It’s da Pope!” The stern warning from a police officer on traffic duty greets our arrival at Penn Station. Moments later the Papal cavalcade speeds through towards Harlem and a meeting with leaders drawn from New York’s less affluent communities. Three giggling Muslim girls in colourful headscarves snap pictures on their phones as it passes.

The past few days in New York have witnessed an even greater buzz than usual. The streets have been heavy with armed police and security, first for that Papal tour, and then for the gathering of more than 150 world leaders for the UN General Assembly. Protest groups – from civil rights in Bangladesh to the environment in Latin America – are in full voice.

Pop’s first lady Beyoncé topped the bill at a free “Global Citizen” concert in Central Park. She was joined by the real First Lady, Michelle Obama. Yes, that’s how important a weekend normally too-cool New Yorkers have just had. Local TV reporters have been breathless in their excitement.

The contrast with what is going on over at the United Nations could not be more striking. There, Barack Obama and his Russian counterpart met for the first time in more than a year for a grim piece of arm-wrestling that neither side seems likely to win.

At stake is, well, just about everything if you live in the Middle East, and a great deal too for the rest of us. The Syrian crisis – with its 200,000 dead and who knows how many more injured, displaced, radicalised, hurt – highlights the cynicism and despair of what we used to describe as “east-west relations”.

Life was simpler during the Cold War era of them and us. The global arms race, whose real futility was the promise of “mutually assured destruction”, or “MAD”, was punctuated regularly by talks featuring serious men in dark suits and brokered by sinister figures like Alexei Kosygin or Henry Kissinger.

The myth of this “era of peace” remains strong. In fact, the two super-powers were engaged, usually clandestinely but spectacularly not in the case of Vietnam, in dozens of proxy “wars” all over the world. Like pawns on a chess-board, small countries all over Africa, Latin America and Asia were manipulated cynically by the US and Soviet Union.

How the spooks must have missed all that action after Ronald Reagan and Mikhail Gorbachev met in Reykjavik in 1986, and the fall of the Berlin Wall that presaged the collapse of the Soviet Union. The next two decades – wars in the Balkans, genocide in Africa, upheavals in the Indian sub-content – were driven more by nationalism, tribalism and issues of territory than good old-fashioned superpower ideology.

American foreign policy used to be driven by its venal pre-occupation with oil. That obsession with hydrocarbons – keeping US industry working and its gas-guzzlers on the road – justified more death and wars than any other conflict since the Second World war. Exceptions such as Vietnam were driven by paranoia of Communism.

The first Gulf War stemmed from protecting Western interests in the region. We are still left wondering what the second Gulf War was about, although it sure wasn’t much to do with “restoring democracy” in Iraq.

The rise of militant Islam is a reaction to the superpowers’ meddling in world affairs. From Afghanistan across the Middle East, Western interests are as much at the root of the conflict as regional intrigue or someone else’s jihad. The barbarous so-called Islamic State finds support amongst a generation disillusioned by capitalism and seeking some sense of identity, however macabre the means.

We have truly failed in the West to understand what is going on. In fact we have barely attempted to do so. As the “Arab Spring” of 2010-11 took hold across North Africa and the Middle East, it was cheered on by politicians and media as a great breakthrough for “democracy”. Down with those damnable dictators (many of them bolstered and armed at various stages by Russia and the West)! Hurrah for democracy! It was as if the demolition of the Berlin Wall was being enacted all over again. Another victory for the Western democratic model. For Tbilisi read Tripoli, for Prague read Palmyra.

Well, not quite. The Wikipedia entry for “Arab Spring” is revealing: “Arab Spring refers to the democratic uprisings that arose independently and spread across the Arab world in 2011. The movement originated in Tunisia in December 2010 and quickly took hold in Egypt, Libya, Syria, Yemen, Bahrain, Saudi Arabia, and Jordan.”

The Wikipedia author may be a sceptic with a very dry humour. The entry adds: “The term [Arab Spring] was previously used … by numerous media commentators to suggest that a spin-off benefit of the invasion of Iraq would be the flowering of Western-friendly Middle East democracies.”

Today Libya remains in chaos and Yemen is in similar straits. While Jordan and Tunisia are stable, Bahrain and Saudi remain authoritarian and Egyptian democracy was only welcomed by the West until the “wrong” party won an election and the military stepped in once more.

Syria is run by a brutal and autocratic regime, for many years bolstered by Moscow. President Assad’s government has a lot to answer for, in terms of its brutality and indifference to democracy. It has been bombing its own people for years, as Syria’s “Arab Spring” turned into outright civil war, displacing an estimated four million people mainly to neighbours like Lebanon, Jordan and Turkey, but also to the EU border where they have met Hungarian hostility and German guilt. Human concern about refugees can waver when they turn up on your doorstep looking for a home.

So Mr Obama and Mr Putin sat down to consider anew the great chess-board of international power-play. Before them is a set of pieces blown apart by what people and nations actually do to each other. Attended by the various placemen, dictators and democrats of the UN General Council, the great men agree that shoring up Mr Assad may be the least-worst means of defeating Isis. Mr Putin may even join in the Nato bombing raids.

We are as far from the “flowering of Western-friendly Middle East democracies” as at any time since 1946.

This grim scenario results from decades of cynicism. Our outlook on the Middle East, the Arab world, and the role of the great powers in so often making a bad situation worse, is punctuated occasionally by our collective conscience; for example when a three-year-old boy is found drowned on a Turkish beach.

The name of Aylan Kurdi has just about disappeared from the headlines, barely a month after his death. He won’t be the last victim of Middle Eastern duplicity and the amoral machinations of the great powers. And this will not be the last UN meeting to consider what on earth can be done.