My first post on Ofsted inspections of ITE set out where I am coming from, and considered the purpose of these inspections. It concluded:
“So if Ofsted were to step back from reporting on good practice, and if the difference between Grade 1 and 2 (over 80% of providers) has a rather arbitrary effect on available provision, that leaves Ofsted as an effective enforcer of absolute minimum standards and a possible pressure, and possible guide, to improving the quality of training. The former role requires reliable differentiation between Grade 1/2 and Grade 3/4; the latter two require valid measurement of training quality, and the ‘guide’ bit requires accurate identification of strengths and weaknesses. In this second post, I’ll try to dig into the issues of reliability, validity, and accuracy that my original comment alluded to.”
For those of you who are not aware of how an ITE inspection works, the call comes first thing Thursday, with the inspection starting on Monday. For the two years the previous Framework operated, this could be at any point in the academic year. The inspectors look at statutory requirements; data (on outcomes and tracking of progress, and NQT surveys about their training); observe training sessions (if there are any); observe trainees teaching to get an idea of their progress, to look at the quality of mentoring, and for evidence of good training showing in their teaching; and observe NQTs (and maybe RQTs) teaching to judge the quality of the final product. That’s my summary, for more information check the Handbook.
When we were inspected, secondary trainees were observed right at the begining of their second placement i.e. day 3, so the ones affected only found out with a weekend’s notice that not only were they going to be teaching a class on Monday, in a school they didn’t know, but it was going to be with an Ofsted inspector observing. I thought that was an unacceptably awful thing to do to trainees. The inspection team handled it sensitively but I just felt grossly unprofessional about the whole thing. It’s less important, but clearly there is also a major issue with reliability here, too. How can inspectors reasonably be expected to judge trainee progress if one lot are observed in their first placement, others on day 3 of their second placement, and at another provider they are observed after many weeks of teaching?
This is one of the main drivers behind my original comment about the way in which Ofsted inspects ITE. However, under the new framework this has been sorted out. Hurrah! The changes are probably best summarised in the revisions to the framework but under the newest framework, there will be a summer inspection which will include observation of training sessions (if there are any) and trainees teaching; and an autumn inspection which will focus on observation of NQTs (and maybe RQTs) teaching. There is still an issue with HEI courses finishing placements at Whitsun, and SDs going to the end of the summer term, and some weeks having training sessions, and some weeks none, so I do think Ofsted really need to get calendar info before setting dates if they want to improve reliability by comparing like with like, but it is a solid step away from ‘dreadful’ and actually I think quite a bold and imaginative idea.
The second thing that really upsets me about Ofsted is the pressure on ITE providers over grading trainees. Under the Grading Descriptors on p.33 the Handbook states that for Grade 1 or Grade 2 ITE providers “all trainees awarded QTS exceed the minimum level of practice expected of teachers as defined in the Teachers’ Standards”. That word ‘exceed’ is critical; in other words, if any trainee gets a Grade 3 then an ITE provider Requires Improvement. I think this is probably a remaining ripple from the big splash casued by the changing of Grade 3 from ‘Satisfactory’ to ‘Requires Improvement’. At Grade 3 a trainee meets the Teachers’ Standards and therefore will be awarded QTS but where once this was Satisfactory, it no longer is. Providers certainly ought to be trying to provide extended placements with extra support to reach Grade 2 before gaining QTS but it also ought to be acceptable for providers to work hard to support Grade 3 NQTs in schools. At the moment, this is a very dangerous strategy because a Grade 3 might not go into teaching (but will still be in the data, and qualified). The incentive to find some spurious evidence and chance upgrading them before awarding QTS is obvious. We have taken the right approach at my university; I will be fuming if that comes back to bite us.
Of course, the alternative is to find some spurious evidence and fail them. If we are really saying that we don’t want these trainees in the profession then, fine, but the Teachers’ Standards and/or award of QTS needs changing to reflect the standard required. Don’t just tell providers that Grade 3 meets the Standards for QTS but it isn’t acceptable to let anyone at this standard be awarded QTS. And, of course, completion rates are significant data in an inspection. Just like exclusion rates for schools, high completion rates might demonstrate excellent recruitment and training, but they could also reflect over-grading and low standards. Good recruitment decisions obviously help with completion rates but where is the evidence that there is a reliable way to discriminate all the potentially good teachers? Where are the science and maths teachers we need going to come from if we only take dead certs?
Anyway, those are the two points that led to my labelling ITE inspections ‘dreadful’, so it’s one down and one to go for Ofsted on fixing these. I will now try to get some perspective on the issues with reliability, validity, and accuracy, promised at the start of this post.
So here are some of the reliability problems with ITE inspections:
- Even under the new Framework, inspectors are likely to see different things at different providers depending on when in the summer term they visit. This is not easily resolved but I would like to see Ofsted acknowledging the challenge, at least.
- The amount of training observed is likely to be tiny (if any). I think the danger of a poor session from one trainer tarring the whole course with the same brush is too high.
- NQT observations are attempting to evaluate the quality of the finished product. There is no mention of individual lesson observation grades in the Handbook but our inspection team saw only seven secondary NQTs which leaves an awful lot riding on those individual performances. Hopefully the two-part inspection will increase this number but there is nothing in the Handbook to reassure me that Ofsted are clear about how many are required to ensure reliability isn’t affected by random variation.
- The same reliability issue affects any comparisons drawn between NQT quality when observed, and grading of trainees at the end of training. The Handbook doesn’t appear to require this but it was a clear feature of our inspection (so maybe the framework has changed).
- Any observation of NQTs is bound to be influenced by the quality of induction and training provided by the employing school, and their ability to pick NQTs that suit their school. Under the previous framework all schools involved would be in the ITE Partnership, so maybe that’s fair game; under the new Framework I’m not so sure that will be the case.
- Observation of RQTs is hard to justify (although interviewing them about their training may well be appropriate), because so much will have happened in schools since training. Maybe this won’t be a feature of inspections but the Handbook is a bit ambiguous on this. The phrase being “NQTs/former trainees”.
And here are some of the validity problems:
- There is no evidence-based way to determine the standard of trainees at the start of their training; so any measure of the quality of outcomes will reflect not only the quality of training but also the quality of applicants. It’s not currently possible to measure ‘value-added’ but there is a sense that this is nonetheless what Ofsted think they are doing. Maybe the argument is that recruitment and training quality together are being evaluated but this is pretty advantageous for the providers with the best reputations who get more applicants. Is reputation really a variable that Ofsted want to include in their inspection outcomes?
- Completion rates might demonstrate excellent training and support, but they could also reflect over-grading and low standards, as described above. ITE providers must, in the end, be gatekeepers to the profession – children are owed that.
- The Grade 3 penalty means, as described above, that if the best ITE provider in the country correctly grades a trainee 3 and hasn’t sorted it before inspection then that one piece of data will count more than everything else combined.
- The new framework places a big emphasis on behaviour. Inspectors won’t be seeing the training, only the performance of trainees and NQTs. What they see will depend an awful lot on context. The NQT having a ding-dong battle (that they will eventually win) with a truculent Y10 class could easily represent outstanding training, whilst the clockwork smoothness of another class might be due to smashing kids, or a trainee for whom good behaviour comes as easily as breathing.
- The NQT Survey data depends a lot on responses and there is no mechanism for validating the data; our inspection was possibly triggered by a drop in the previously high ratings from this survey but that data was flatly contradicted by our exit point survey data so what happened remains an unsolved mystery.
Finally, on the subject of accuracy, inspectors are in for three days maximum; during this time they may be able to make a fair stab at judging the quality of the provider but I really don’t think that they can achieve a level of understanding that would allow an accurate description of not only what, but why, the provider was doing well in certain areas, or not so well. I think inspectors will tend to see strengths and weaknesses in the presence or absence of the things they value in ITE – confirmation bias at work – and I don’t think that is good enough evidence on which to build world-class intial teacher education.
I’m not actually saying that I think Ofsted ITE judgements are necessarily unreliable or invalid, I’m just saying that there are all these issues that are fairly obvious and I have no sense that Ofsted are engaged in worrying about these things. Maybe it is possible for an inspection team to accurately grade providers on a 1-4 scale, but I think it’s ambitious, and if these judgements aren’t right then Ofsted could be failing to correctly identify providers offering poor quality training, and they could be creating pressure to improve, and offering guidance, that doesn’t actually lead in the direction of genuine improvements – the problem we’ve been seeing in schools until recently.
There have been some very sensible suggestions that school inspections should move to a three-tier grading system and I think this would make sense for ITE. I’m not sure that trying to distinguish Outstanding from Good is terribly helpful whereas getting really effective at distinguishing Requires Improvement from ‘Good or Better’ is terribly important so we don’t have badly trained NQTs entering the system. And this brings me to the massive elephant in the ITE inspection room.
I’m very aware that the effectiveness of the established system of training teachers has been a moot point but it has at least been pretty stable. Now, ITE is going through a massive upheaval. SCITTs are sometimes, effectively, single schools, and SD alliances can be very small too, or dominated by one school. I’m certain some brilliant things will be happening but also sure there will be some disasters. A lot of this new training is, on paper, quality assured by HEIs or well-established SCITTs but SD has put schools in an exceptionally strong position to plough their own furrows. The chaotic nature of all this is entirely the doing of the DfE but it is Ofsted that are ultimately responsible for enforcing standards. SD should have been introduced more gradually but, given that the seeds were all cast at once, it needs a bit of germination time and there may be a few sickly seedlings that will produce excellent crops so it seems a bit harsh for Ofsted to get the hoe out straight away. For this reason, the complete avoidance of SD in our recent inspection is possibly justified, but Ofsted need to quickly be exceptionally clear about how they are going to engage with SD. In particular, I don’t think it is acceptable to lump SD and provider-led training together. Yes, a provider that allows poor quality SD to run on their watch needs to be pulled up on this, but unless somehow this drills down to the decisions made at school-level, providers will be held responsible for decisions made at the periphery of their control (even when their own training is excellent) whilst the school leaders who should have done better (or stayed out of it if they weren’t sure they were going to get it right) remain largely unscathed. If Ofsted tame the elephant, we might all come out of the SD revolution in some semblance of order and then be able to get on with the question of how to make our NQTs even better-prepared for civilisation’s most essential profession. If Ofsted don’t get this right, children will suffer.