By Lindsay Paterson, professor of education policy, Edinburgh University

THE perennial controversy surrounding Scottish Catholic schools erupted again this week with a report that seemed to show that they do not do any better than other schools. Analysis of Scottish Government data by the Institute for Public Policy Research, a think tank, found that the average attainment of pupils in Catholic schools was very similar to that by pupils in non-denominational schools. The headlines which this provoked were emphatic: Catholic schools are no better than any other. The implication was that bringing religion into school governance is an expensive luxury.

This is odd. One of the repeated findings of good quality research in Scotland and in many other countries is that Catholic schools are highly effective. They are distinctive in ways that give them an edge in attainment. Their ethos, their being embedded in a community, their attention to civic responsibility: these qualities are part of the very identity of Catholic schools. Other high quality schools have similar characteristics too, of course. But Catholic schools have them in greater abundance, and that is why they achieve so well.

So why the different findings? The immediate explanation of the apparently contradictory research is quality. To be rather brutally frank: the IPPR work would not pass muster in any rigorous academic context. But the reason why it could not be any better is a failure of Scottish government.

First, though, what did the IPPR report say? The researchers noted that Catholic secondary schools in Scotland have almost the same proportion of pupils passing three or more Highers as other schools run by the local authorities. They also noted that poverty is a bigger issue in Catholic schools than in other schools. A higher proportion of their pupils receive free school meals. On the face of it, these patterns of attainment and of poverty might lead us to say that Catholic schools are remarkably successful in overcoming some of the effects of deprivation. Indeed, that’s what all the previous research has found.

But the IPPR report did not reach that conclusion because it found that the difference in attainment was not “statistically significant” after allowing for the percentage taking free school meals. And the reason it reached that conclusion is that the data it used was not fit for this purpose.

Explaining why is unavoidably technical, but here’s a flavour of it. The IPPR work had just 51 Catholic schools. These were treated as just 51 independent observations on Catholic schools, and IPPR’s technique was to ask how much chance variation we would expect in such a tiny sample. Since the answer is quite a lot, the difference between Catholic and non-Catholic schools could be readily attributed to chance, which is what IPPR duly did and the headlines repeated.

But that’s nonsense. Inside the 51 schools are somewhere in the region of 6,000 pupils sitting Highers. A survey with 6,000 observations would give us a much better fix on the evidence than one of just 51. If we could take that fact into account, the IPPR headlines would be indefensible.

But why couldn’t IPPR just go right down to the level of these individual pupils, and analyse their exam results? That is what the previous research on this has done, concluding that attainment is higher in Catholic schools. IPPR seems oblivious to any of this history of research – whether in Scotland or elsewhere – but, apart from that, it also as stymied in what itcould do by the destruction of the relevant survey resources.

The previous research on Scottish Catholic schools used the Scottish School Leavers’ Survey, which ran every two years from the 1960s to about a decade ago. If it still existed, it could have been used now. But it was abolished. That was a political decision by the then Scottish Executive which was echoed in a similar decision by the Scottish Government a few years later to withdraw Scotland from several important international surveys of education. So we now have no survey evidence that would allow us to address with appropriate rigour the important question which IPPR raised.

Catholic schools have been unfairly traduced this week, which is bad enough but is only a microcosm of a much bigger problem. In the data desert which now curses Scottish education, no worthwhile policy question about pupil attainment can be answered rigorously. For a nation that pioneered the use of surveys to inform education policy, that is tragic.