A new paper from an educational lobbying group argues that a lack of student data makes it "effectively impossible" to measure learning and teaching or evaluate government education policies in Scotland.

In "Why Educational Data Matters", the Commission on School Reform (CSR), the educational arm of the lobbying group Reform Scotland, described the Scottish school system as a "black hole out of which no information escapes that might be useful to immediate policy."

The paper highlights distinct gaps in school data, including a lack of statistical evidence on teaching strategies, student experiences in schools and the transition from school to work or higher and further education. 

A throughline of the CSR paper is the argument that where high-quality data exists, it is often limited to describing individual students and subjects or is too inconsistently gathered to make meaningful comparisons.

Lindsay Paterson, professor emeritus of education policy in the School of Social and Political Science at Edinburgh University and a contributor to the paper, said this is concerning because incomplete data not only prevents meaningful learning but can also be actively misleading.

“Scottish schooling is a black hole out of which no information escapes that might be useful in assessing progress and informing policy.

“This absence of data means that it is effectively impossible to track learning or teaching, to assess the outcome of schooling for leavers, and therefore to evaluate government policies intended to improve schooling."

Prof Paterson pointed to existing sources of information on students and their performance, which fall short of the high standards that the CSR paper advocates. 

These include Scotland's ongoing participation in the global Programme for International Student Assessment (PISA) studies; attainment evidence from the Scottish National Standardised Assessments, the Scottish Qualifications Authority and Achievement of Curriculum for Excellence Levels (ACEL); and the Growing Up in Scotland Survey, which has been tracking a single cohort of pupils since 2005.

The CSR describes the above data sources as incomplete for various reasons.

For instance, although ACEL data measures the percentage of pupils who have attained each level of the Scottish curriculum every year, the measurements are based on teachers' judgments and not standardised tests. 

Read more:

Data from the SQA, on the other hand, is an objective measurement using standardised assessments, but information on specific students is limited to single subjects. 

Prof Paterson said Growing Up in Scotland is a particularly telling example of data that could be incredibly useful but falls short, primarily because it only provides data on one cohort.

"Growing Up in Scotland is a very good survey. It's of the highest standard and widely used by researchers in other countries.

"But without a comparison, you can't really say anything meaningful at all.

"It's sheer chance that these children entered primary school just when Curriculum for Excellence was being extended across all schools. 

"So we have no prior measure of what it was like to be in a primary school just prior to that, nor indeed have we any capacity to measure how the various iterations of Curriculum for Excellence and various reforms have had an impact."

Professor Lindsay Paterson has examined the role that students once played in shaping education policy, and is helping make the case that they should do again.Professor Lindsay Paterson has examined the role that students once played in shaping education policy, and is helping make the case that they should do again. (Image: Newsquest)

The solution argued for in the CSR paper is to integrate data from existing Scottish National Standardised Assessments and SQA assessments and introduce regular surveys to capture the experiences of multiple cohorts of students.

In many ways, this builds on previous pressure from Reform Scotland to return to a more robust data collection in the country.

It is also a call for reintroducing data that disappeared in 2002, when the Scottish School Leavers' Survey ended, and again in 2016, when the Scottish Survey of Literacy and Numeracy ended.

Prof Paterson said the CSR's proposed survey would be administered every three years at P1, P4, P7 and S3, followed by a questionnaire when students leave school. 

The surveys would collect information about the curriculum, attainment, and school experience and track demographic information.

The CSR argues that information on students' home lives, which the Scottish School Leavers' Survey previously gathered, provides crucial insight for implementing fair and effective educational policy.

"There is no evidence on the relationship between children’s progress and anything more than rudimentary aspects of their psychological challenges and their social circumstances.

"Students face enormous and unfair extra difficulties if they have limited space at home in which to work, and if their parents cannot afford such educational resources as good internet access, a range of books at home, and stimulating excursions."

Critically, he added, the survey would need to be administered independently and not handled by schools or local authorities to avoid placing added pressure or creating temptations to skew the results.

To ensure this, the CSR is also proposing the establishment of a new Office for School Education Data, which would be funded by and answer to the Scottish Parliament rather than the government.

Prof Paterson said that the foundation for these changes exists if policymakers are willing to act.

“The good news is that it would not be particularly challenging to put a credible data structure in place.

"We already have good-quality evidence from the Scottish National Standardised Assessments and the SQA assessment process, and by bringing these together and adding a regular sample survey at key stages, we can generate critical data."

A Scottish Government spokesperson said that these existing assessments provide the basis for ongoing reform efforts and that the CSR's proposals will be considered.

“Scotland already has an extensive range of assessment measures to monitor and inform the progress of pupils. Our programme of education reform will improve outcomes for learners of all ages, while ensuring the education workforce is better supported.

“We will consider the proposals of the Commission on education reform as part of our ongoing efforts to ensure the best data analysis is available to inform policy.”