The uphill battle of establishing accurate performance trends for schools
The South African Institute of Race Relations, in its February 2016
Fast Facts publication, concludes that “schools drag South Africa down” and that
pupil performance is declining substantially.
The latter conclusion is based almost entirely on two numbers: the
number of Grade 12 pupils obtaining a score of 70% or more in mathematics in
2008, which was 25 027, and the corresponding figure for 2015, 17 452. The two
figures point to a decline of around 30%. At first glance, this appears to be a
However, analysis done by myself and others in the basic education
department, to be published shortly, paints a completely different picture, of
large and encouraging improvements in mathematics in Grade 12. Many of the
challenges facing the schooling system are accurately described by the
institute’s report, but I disagree completely that the numbers should be
pointing to a deterioration.
The problem is, first, that the institute looked selectively at the
numbers, and ignored important figures, including a few appearing in their own
report. This seems irresponsible.
Second, analysing pupil performance trends is exceedingly complex,
not just in South Africa. The British education analyst John Jerrim has written
extensively about how the data on mathematics trends have been spectacularly
misinterpreted in his country. In South Africa, the complexities are
particularly daunting in the case of Grade 12 mathematics.
So what did the institute’s analysts get wrong? They failed to
point out that all of the 30% decline they refer to happened between 2008 and
2009. From 2009 to 2015, the trend, using values from all years (as one should),
is a weakly positive 2% overall. The number of passes at the 70% level in 2008
was exceptionally high relative to all other years, something which should make
any analyst suspicious.
The institute’s analysts also fail to point out that the overall
increase in the number of physical science passes at the 70% level, over the
entire 2008 to 2015 period, was a whopping 85%. In contrast to mathematics, what
appears suspicious here is at least one exceptionally low value at the start of
the period, in 2009. The question is why two such closely related subjects would
move in completely opposing directions.
Another suspicious trend discussed in the analysis to be released
by the basic education department is that the percentage of white and Indian
pupils achieving high marks in mathematics has declined markedly over the 2008
to 2015 period. There appears to be no plausible explanation for this trend
among these two relatively advantaged groups.
We zoomed into a sample of particularly stable and well-performing
schools, with about 4000 mathematics candidates each year, to find explanations
to the apparent anomalies. What emerged clearly is that variations across years
in the difficulty of obtaining certain marks, for instance 70% in mathematics,
explain most of the anomalies.
Mostly these variations are small, but for certain years they are
large. Pupils who obtained 69% in the years 2012 to 2015, when levels of
difficulty appeared particularly stable, would have obtained 68% in 2011, a
marginally more difficult year, and at least 72% in the years 2008 to 2010. 2008
was a particularly easy year for obtaining high marks.
Changing the criteria for our sample did not change the picture
substantially. Over the years, the mathematics examination became more
difficult, while for physical science the opposite was true.
So is the problem then poor standardisation in the examinations
system? Yes and no. There appears to be scope for improving the comparability of
marks across years and this is receiving the attention of the basic education
department and the council for quality assurance in general and further
education and training, Umalusi.
At the same time, it is technically impossible to achieve anything
approaching perfectly comparable marks, at all mark levels, in an examination
system such as ours, or in similar systems in other countries. We need to learn
to live with some variation over the years and rely on other systems, such as
the international testing programmes, for more rigorous assessment of
When we recalibrated results for all pupils over the 2008 to 2015
period, using what we found to be equivalent scores, we found that the number of
pupils achieving a 70% level of performance in mathematics increased by 27%
For black African pupils the increase was 61%. Physical science
improvements, on the other hand, were found to be smaller than what published
statistics would suggest, but were still encouraging. By far the largest
improvements were in historically disadvantaged schools and top mathematics
performers are spread across more schools in 2015 than they were in 2008.
We do not dispute that the under-performance of schools is a key
factor holding the country’s development back. This is made clear in the
National Development Plan. However, where we do disagree strongly with the
institute for race relations is the direction the schooling system has been
taking in recent years.
If the movement has been in the right direction and improvements as
large as one might realistically expect, then one could hardly hope for more.
The evidence suggests the quality of school education is improving,
that the improvements have been substantial and encouraging, and that they are
helping to overcome historical race-based inequalities.
But trends seen in a few other countries, such as Brazil, suggest
we should be aiming for an even steeper improvement.
This is what ongoing changes to our interventions, of which there
are many, should aim to achieve. We also need a more rigorous national debate,
involving a wider range of stakeholders, about the actual performance trends of
» Martin Gustafsson is
based at the department of basic education and a member of the ReSEP group at
Stellenbosch University. The opinions expressed in the article are his personal
We live in a world where facts and fiction get blurred
In times of uncertainty you need journalism you can
trust. For only R75 per month, you have access to
a world of in-depth analyses, investigative journalism,
top opinions and a range of features. Journalism
strengthens democracy. Invest in the future today.