Over the last four decades the Scottish Catholic community has been transformed.

As late as the 1960s, Catholics, the vast majority the descendants of Irish immigrants of the 19th century, remained a marginalised group. Overwhelmingly working class, they experienced routine prejudice, discrimination in the workplace and poor educational standards.

The story since then has been one of rapid emancipation and acceptance into the mainstream of society. Catholics finally achieved occupational parity with other Scots by 2001.

Labour-market discrimination against them, once so endemic, withered away. A new Catholic professional middle class emerged, no longer simply consisting of schoolteachers as in the old days, but of doctors, lawyers, accountants, academics and more.

The huge expansion of the universities in the later 20th century provided the ladder of opportunity. Denominational schools now attract praise from the inspectorate and serious commentators. The media, when matters of moral controversy arise, virtually ignore the national Church of Scotland and instead prefer to focus on the pronouncements of the more media savvy Catholic hierarchy – an astonishing change.

Today all the top jobs in civic Scotland are open to Catholics. Several university principals, senior law officers and leading politicians are members of the community.

Of course, the whole picture is not quite as rosy as we suggest. Anti-Catholic bigotry and anti-Irish racism survive in some areas. Scotland is the only jurisdiction where emigrants from Ireland have settled which feels it necessary to impose laws against sectarianism in the 21st century.

Despite its proven educational successes, the denominational school system attracts sustained attack. Some of the poorest and most disadvantaged communities in Glasgow and the West still contain large numbers of Roman Catholics with significantly higher than average rates of mortality and morbidity.

Paradoxically, with material and social amelioration have come major problems and challenges for Scottish Catholicism. The Church is not at peace with itself. Partly this stems from an ageing, declining priesthood. The crisis in vocations has been exacerbated by the celibacy rule. The next decade will see parish amalgamations and closures.

But the problem is more than manpower. Discord is also located in a series of "culture wars". The opening shots sounded as long ago as 1966. Mindful of the conclusions of the majority of the Pontifical Commission on birth control, many had expected a sea-change in Church policy. Instead, Rome imposed the status quo.

In 1968, with the release of his encyclical, Humanae Vitae, Pope Paul VI closed the doctrinal debate. For a while, an uneasy truce was established. Catholic women joined their other Scottish sisters in use of the pill. Priests noticed the three children now forming "good Catholic families" and said nothing. The laity reciprocated with uneasy silence.

But that breach between altar and pews has deepened. Whereas their grandmothers perhaps approached contraception with some embarrassment and shame, today's young Catholic women would be incredulous if asked the question the commission posed: "Is contraception always seriously evil?" The modern Scottish Church has suffered the indignity of rendering itself irrelevant.

Further fissures between the laity and leadership opened in the recent botched campaign against equal civil marriage. A confident episcopate would have been reluctant to take on a battle it knew it could not win. With the days of dog-whistle loyalty long gone, Scotland's Catholics simply refused to get in line.

This was not only a failure of leadership but judgment. A community which itself encountered deep prejudice was unlikely to respond to shrill, fundamentalist rhetoric from the bishops. Younger Scots Catholics tended not to frame the question as bishops might: "Are gay marriages seriously evil?" Instead, well-versed in their own Church's rights discourse, the question for them was essentially one of human rights. Their answer was clear – they were no less tolerant than their peers.

Further polite standoffs seem unlikely. Most recent papal and episcopal pronouncements suggest little scope or readiness for compromise on these moral "fundamentals". Without a new pontiff somehow rekindling their hopes for change and open-mindedness, the West's most enduring institution is faced with an existential struggle if its younger members judge it not merely irrelevant but unfair. A good start might be a little more listening and a little less preaching.