The final figures are in. And they don’t look good. Eighty-eight per cent. That’s six per cent lower than the Scottish Government’s own target and nine per cent lower than the result achieved in England and Wales. The Tories call it a shambles. Labour call it a catastrophe. The SNP calls it a success. But the real question is: what can the census teach us?

But first, a little more detail of what the figures actually show. The 88% figure I’ve quoted is the average response for Scotland as a whole, although in many areas it was much lower. Dundee, Glasgow, and West Dunbartonshire for example barely scraped past 80% and only 12 local authorities out of 32 got more than 90. Interestingly, what we don’t have are highly localised figures which I suspect would be even worse. Labour’s Sarah Boyack says she visited a high-rise block of flats where the rate was 57.

In response, National Records of Scotland, the government department that runs the whole thing, urges caution. When I spoke to them the other day, they told me a few more postals might still trickle in; they also pointed out that they’re sending staff door-to-door to some 53,000 households to check the accuracy of the information. But even so, the data collection phase is finished. To all intents and purposes, these are the final figures and there are, I think, quite a few lessons we can learn from them.

The first is that the biggest failure of the census was in the parts of the country where it’s needed most – in prosperous areas, the response rate was high whilst in relatively poorer areas, the rate was low. What this means is there’s a serious chance that in deprived areas we will under-estimate the need for public services and the services will be inadequate. The alternative is that we over-estimate the need and waste money. Either way, it’s bad.

The second lesson we’ve learned is that, if Scotland does things differently, it will often cost way more than it would if we’d sensibly pooled resources with the rest of the UK and done it together. The original plan was that the Scottish census would be part of the UK operation in 2021 but the Scottish Government decided to go it alone and delay for a year – allegedly because of the pandemic – which cost us an extra £21.6m. The fact that the census then had to be extended because of the shockingly low responses then cost us another £6m on top of that. And none of it needed to happen.

The third lesson is even more important: the Scottish Government needs to check its attitude. Take a look at Angus Robertson’s ministerial statement at Holyrood and you’ll see what I mean. There was no mention of the government’s 94% target; instead Mr Robertson hailed the fact that every area achieved over 80%, which is a completely arbitrary figure. He also said the extra £6m was only 4.3% of the total costs and that the return level was “good”. The NRS also told me that the results they’d achieved were a “solid base”.

I’m tempted to call this approach the Scottish problem, or at least the Scottish problem while Scotland is run by the SNP. First, do things differently to the rest of the UK. Second, muck it up and cost us heaps of money. Third, call it a success. And fourth, sink your teeth into anyone who criticises you. Mr Robertson showed us how it works when he spoke in parliament. The botched and useless census was good, he said, the extra money wasn’t that much really, and the attacks on the government were false and misleading. So there you have it: everything’s fine and it’s you that’s got the problem.

Fortunately, I spoke to Professor Lindsay Paterson, professor of education at Edinburgh University whose expertise includes data-led research such as censuses, and he is hugely impressive at cutting through the government guff. The results weren’t good, he said, they were poor and they are poor because the government will have to use “imputation” (as it’s called in the data trade) to suggest what the people who didn’t complete the census might have said. The problem is that National Records Scotland cannot know for certain and imputation will introduce a random error and in some parts of the country – probably the poorest – the error will be large.

Professor Paterson was also deeply unimpressed with the government response to its failures. Quite fairly, he asked how researchers like him or people who come up with public policy or make decisions about where public money should be spent are meant to rely on data that’s as flawed as the data collected by the Scottish census. He was also pretty disgusted at the government and the NRS presenting their failures as successes – a distortion of reality Professor Paterson called it.

For what it’s worth, I agree with the professor. I also agree with him when he says that a real nasty streak has crept into the debate and that nobody is apparently allowed to criticise National Records of Scotland because they, and no one else, are the experts. This is part four of the Scottish problem under the SNP: sink your teeth into anyone who criticises you.

There is a better way to do all of this however. First of all, the Scottish Government could have just done the grown-up thing and taken part in the UK-wide census which would have saved time and money and produced better, more reliable results. But heh ho, the SNP wanted to do their own thing and it went wrong and we’re stuck with it now so that’s that.

But now that it’s gone wrong, shouldn’t we expect some honesty or even some humility? This happens lots and lots of time with the SNP and to be fair many other modern politicians – we can see that the thing we’re talking about is a spade. I know it’s a spade. You know it’s a spade. They know it’s a spade. And yet they don’t call it a spade. Stop doing that. Call it like it is. Scotland’s census was a disaster. I know it was a disaster. You know it was a disaster. And they know it was a disaster. And yet they call it a success.

Behaving in this way will not only undermine the process of honestly and realistically assessing how the census went and how we could do it better next time, it also undermines the idea of politics itself. Scotland’s census was about asking all of us a series of questions so let me ask this one of Angus Robertson: do you really think the census was a success? And if you don’t but keep saying it was, what do you think that does for the reputation of politicians and politics? And more importantly, what do you think it does for your chances of achieving that other big plan you have for Scotland? Please, just answer me that.

Our columns are a platform for writers to express their opinions. They do not necessarily represent the views of The Herald.