Most of us think we can spot a trend in school data when we see one but increasingly I’m not so sure. The problem is not that trends don’t exist; some schools will genuinely be improving and others declining. The problem is not even a failure to recognise that some trends might be completely outwith the control of the school. The problem is that what looks like a trend, might be no such thing.
Actually, I think the deeper problem is that most people tend to accept that schools have blips in their data for reasons that are almost completely random but that looking at data over several years gets past this problem. Reaction to the Cramlington Learning Village Ofsted report (Outstanding to Special Measures) is a good example.
Some of the arguments against this kind of interpretation have been about the validity of statistical methods – @Jack_Marwood has blogged about this extensively and there has been a little frisson of excitement about the way that the clustering effect has been used to overule the apparent statistical significance in the recent RCT on reception baseline testing, with speculation about how this might be applied to RAISEonline etc.
But I want to take a different tack. Have a look at this graph.
Over ten years, the red school data falls pretty steadily from the national average of 56% 5A*-CEM – what do you think Ofsted would make of this? And the green? Well it’s not as dramatic as the improvement at Huntington School but as long as they survived the first four years I think most headteachers would give their right arm for this data.
The thing is, about once a year I give myself a little VBA project. I am totally crap at writing code but I enjoy the challenge (and the immediate, if often entirely unhelpful feedback!) This year, the red and green graph is the outcome. I have taken a spreadsheet with 26 identical ‘schools’ with a 56% A*-CEM score and applied an algorithm that randomly allocates anything up to a 4% variation each year, for ten years. The spreadsheet then graphs the best and worst performing ‘school’. If you want to have a go for yourself then here is the Excel spreadsheet
You’ll need to enable macros when you open it, then click the RESET button, and then click the ADD 1 YEAR… button. As a bit of fun I’ve added some of the things that are sometimes cited as key reasons for a school’s success (or not), but I promise you the data is completely random.