MCQ – going all in

I think it was at ResearchEd Durrington, that Peps McCrea talked about making bets. If I remember correctly he was suggesting that, having digested both research and experience, there was a point at which you needed to make a bet, and go all in. If you don’t do this then you end up trying to do a bit of everything, which isn’t as effective.

A recent thread on CogSciSci about the use of multiple-choice questions (MCQs) was asking for thoughts on MCQ design and particularly the problem of distractors – exposing children to wrong answers, and the danger that these get learned instead of (or as well as) the right answers.

I’m no expert here but I’ve read some of the literature. There is an extensive, recent review by Andrew Butler and some interesting work on learning from errors by Janet Metcalfe. Both those links come from Pooja Agarwal, and there is lots more useful and really accessible information on her Retrieval Practice website.

I’m also old enough to have both sat and taught national exams with lots of MC components, as well as being around for their more recent resurgence.

These are my bets.

If the purpose is mainly retrieval practice then MCQs should be kept fairly easy, with the number of alternative answers adjusted to try to make retrieval successful but effortful e.g.
The part of the electromagnetic spectrum that has the longest wavelength is:

1. infra-red,
2. radio waves.

The part of the electromagnetic spectrum that has the longest wavelength is:

1. infra-red,
2. radio waves,
3. gamma radiation,
4. visible light,
5. microwaves.

BUT I think short response is better than MCQ if children can handle the demand. I’m aware that some literature suggests MCQs might be better but I think that might be because the success rate on the short response questions was low i.e. the students couldn’t handle the demand.

If the aim is to hammer misconceptions then MCQs might be best down at just two answers (unless there is more than one thing to get confused) e.g.

Weight is measured in:

1. kg,
2. N.

When two 10 ohm resistors are connected in parallel the total resistance is

1. less than 10 ohms,
2. more than 10 ohms.

I think feedback is needed without delay, and for misconceptions I think it should be immediately after a single question.

 

If you want something closer to the thinking required for short and extended responses, then MCQs might benefit from more complexity –

– here you need to choose more than one answer – but I would see this sort of question as a starting point, leading to “why did you pick that?” oral questioning, and then maybe on to an extended response. Or you could do a cool poster to elaborate on this!!!

And if you are actually preparing for an MC exam – there are some fabulous questions out there from IB and A-Level Physics – then the final preparation should reflect the nature of the exam questions. But again I would see this sort of question mostly as a starting point. As long as you are successful in the endless campaign to show working, then popping up wrong answers to the MCQ below, on a visualiser, could be really valuable use of lesson time.

I am pleased that my dislike of 30-odd years for complex answers seems to be vindicated. However, obviously if students have these in a national exam then they need to learn the exam technique. I shudder whenever I see roman numerals in a question…

So, I think I’m coming to the conclusion that there are two purposes for MCQs.
One is to nail down declarative knowledge. Here the distractors are a potential problem so keeping the number of answers lowish, and aiming for a highish success rate, and providing swift feedback (and absolutely immediate for misconceptions) would be my bet. I think most of the research will have been on this first purpose so it’s about finding the best ‘fit’ with what that says.
The other purpose is to provide a scaffold for practising application (procedural knowledge and inferences from declarative knowledge) and here MCQs and ‘normal’ questions are similar in the way it’s best to use them in teaching. MCQs have the advantage of being easy to mark, though 🙂

 

Advertisements

Amanda Spielman @SotonEd

Last Wednesday, Amanda Spielman, HMCI, delivered the annual @SotonEd lecture. This was followed by a really interesting panel discussion with Amanda, erstwhile colleague Daniel Muijs – now seduced by the dark side – and HMI and SE Regional Director, Chris Russell.

Amanda Spielman is a refreshing change from Michael Wilshaw, who seemed to think that if only all school leaders and teachers were even half as capable as him, every school would be Outstanding. Remarkable, isn’t it, how few very successful people can see that a different sequence of events would have left them looking on with irritation as someone else peddled that line. I think we are lucky to have someone as HMCI with an open mind and no sign of hubris.

She has certainly been on a bit of a (very welcome) mission to talk to a wide range of teachers and others involved in the school system about the way forward for the new Inspection Framework. It was good to hear from her directly. She spoke openly and convincingly and had some good things to say. Does she really understand, though, at a visceral level, what it’s like to have marginal data, a disadvantaged cohort, and a looming inspection. I’m not sure she, or other senior Ofsted figures, really get this. I think that’s a massive problem.

When Ofsted focused on judging quality of teaching through individual lesson observations, the school system developed a massive and largely unhelpful system of high-stakes graded observations, “Perfect Ofsted lessons” and so on. These may or may not have been a fair reflection of the inspection framework but they were definitely a reaction to it – an attempt to reassure school leaders that inspection teams wouldn’t see ‘poor’ lessons.

When Ofsted abandoned all that twaddle and moved to a focus on outcomes, schools fell over themselves to develop hideously detailed data and intervention systems, increasing day-to-day workload and eating into teachers’ after-school PPA and holidays. Some schools achieved Ofsted success by genuinely doing a great job of teaching but there was also a whole lot of gaming of the system to nudge A*-CEM or P8 over whatever benchmark senior leaders thought would please an inspection team. Many secondary schools have been slashing at non-EBacc provision, and it’s a lucky Y6 child that doesn’t spend an entire year on 3-step maths problems and fronted adverbials. At the same time, it seems very likely that a poor, white working-class catchment, has been a disadvantage in trying to keep a school out of the murky waters of RI or a category.

You see, I don’t think it matters how much myth-busting Sean Harford does. I don’t think it matters that there are no longer active inspectors coaching schools. Apply any set of metrics to a high-stakes accountability process, and schools will improve performance on the metrics and not necessarily on what they are supposed to be measuring: Campbell’s Law.

And so, I worry about a change of focus from Ofsted. What new perverse incentives will it unleash? In particular, having got a sense that curriculum will be at the heart of the new framework, I have been really concerned that this will result in subject leaders being required to either re-write, or justify in minute detail, each little step in every SoW. Bearing in mind most of them are only just emerging from the workload hell of the new National Curriculum, GCSE and A-Level requirements, I think that would be a disaster.

Perhaps it’s just that the renewed focus on curriculum (the last edition of Impact is an excellent reflection of the Zeitgeist) had me thinking this is what Amanda has in mind. But that’s not it, as far as I could tell from her speech. It seems much more as though she is focused on the problem of EBacc GCSEs and Y6 SATS dominating choices made in schools. Better that, than the curriculum micro-justification I’d vaguely been envisioning, but I still have three issues with this.

Firstly, I do think the quality of the curriculum, by which I mean detailed, subject-specific planning of schemes of work, is really important. To talk about curriculum and mean something closer to ‘broad and balanced’ is to fail to recognise the central importance of a genuine curriculum focus. I’d rather Ofsted stayed out of the latter, but where Ofsted lead, schools tend to follow. I think there’s a danger of sidelining the current impetus to improve curriculum design within subjects by shifting the focus to the curriculum across subjects instead.

Secondly, how will inspectors decide if a school’s curriculum offer is meeting the needs of their children? It seems like a rather nebulous thing. I may be underestimating inspectors, and obviously a leadership team that have thought really hard about curriculum needs should do better than one that has not. However, I wonder whether the ability to talk the talk (or write the write) will influence judgements. Maybe Daniel Muijs can convince me that valid and reliable judgements can be drawn from the sort of qualitative data that’s going to be available but he has some work to do.

Finally, are we just looking at something that is sort of like the current inspection framework, but with an attempt to punish those making outcome-driven decisions, rather than making a broad, balanced and fair curriculum offer? Will it come down to adding some non-EBacc time back in, and providing a mixture of subjects in Y6? Maybe some better primary science and a genuine open option to do triple at GCSE for anyone who’s keen? These would be excellent things but it begs the question, is this Ofsted deciding they don’t like the government agenda and imposing their own view? Where will that end up?

So I’m reserving judgement. Possibly a “Good” for Amanda, from me but not until I’ve seen what actually happens. The people she needs to talk to are the teachers and headteachers that have been on the wrong end of this process – the bombers that didn’t come back. If she did, I think she might still be interested in trying to reverse some of the damage caused by the combination of recent DfE policy and an outcomes-focused inspection framework. But I think she might also be looking at softening the high-stakes, cliff-edge nature of inspection. That would be a massive step forward.

Deliberate Practice

My son is a keen but fairly mediocre member of an U9 football team. However, he has great hand-eye co-ordination (potentially an excellent cricketer) and has posted 38s for 50m freestyle (he’s been swimming from very young, is big and strong, and seems to have a remarkable ability to listen and respond to coaching when it applies to body movement). So, generally sporty, but why not so good at football, despite wanting to be?

His problem is that his decision-making under pressure is not great on the football pitch. Now I really know diddly-squat about invasion games so this is all speculative, but here’s what I think is going on and how it relates to teaching.

I know when I play football, my problem is that I am good at thinking about my position on the pitch when I don’t have the ball. I’m often in great space and very rarely get caught out of position in defence. When I’m watching I can see the play and call the pass. My skill level is so-so but I can control and kick a ball alright. What I cannot do for the life of me is put any of those things together when I have the ball. The moment I get it I just have this sense of ‘man-on’ and I can maybe get a pass away but it tends to be pretty blind. My son’s the same, plus a little bit of fear of giving the ball away, that means he often makes a pass to nothing rather than travelling with the ball or making a better pass. But he is improving. The thing that is having an impact is a coaching session he goes to that’s completely separate from his team. A typical sequence involves having 4 groups of 3 players, all on a 20mx20m square. The drill is to form a triangle and pass in and out to the middle player, constantly moving between passes. If you picture that, there is a massive amount of movement to process with passes having to be spotted and threaded through the other 9 players on the pitch. They do this again, and again – maybe 20 minutes with breaks for feedback. Slowly, but surely, my son is becoming a better player than me (not a high bar but still…).

What has this to do with teacher training? I’m well aware the physical skill analogy has been around for a good while now. As @LornaShires pointed out, just because Deliberate Practice (DP) works for shooting basketball free throws, doesn’t mean it will work for teachers. Also, it’s pretty clear that whilst DP is important in reaching a high level in golf, music, chess, high jumping, catching and throwing or whatever, both teaching and invasion games are more dynamic than any of those things. Also, the 10,000 hours rule Malcolm Gladwell popularised isn’t supported by more recent evidence. Some people seem to reach very high levels of performance much quicker than others.

Nonetheless, my hunch is that if we could find ways for novice teachers to get more of the right sort of DP at an early stage, I think they might progress faster than is typically the case now. I know some people have been working on this – see Deans for Impact, and work by Harry Fletcher-Wood – and TLAC training is largely predicated on this. However, whilst I can see some value there’s a problem I’m not sure we’ve really got to grips with, yet.

To play football my son needed to spend a good bit of time getting good at kicking and controlling a ball. That’s quite clearly come from practice, at least some of which has been DP. However, it’s a pretty static skill if you just practise passing. I think it’s the same for teaching. Practising responses to specific disruptive behaviour, or clear explanations, or boardwork, seems to definitely be useful (one which the majority of current models of ITT don’t have enough time for). But however good my son gets at striking the ball, it isn’t going to make him a really good footballer. Same for the classroom. It’s those dynamic skills – responding to 30 children in real time – that matter.

I’ve been reading The Sports Gene, following a recommendation on Twitter – possibly from @DylanWiliam. The first chapter is about occlusion tests. The original research involved slides from volleyball matches. People were shown these for a very short time period and then asked whether or not the ball was visible in the frame. Similar research,  identifying the speed with which performers at different levels can identify salient information, has been carried out for a number of sports, for air traffic controllers, and for chess players. There has also been some work on this with teachers. Consistently, experts can extract information massively faster, in their own domain. It seems as though maybe two things are going on: experts focus on areas that provide key information – top tennis players identify the direction of a shot from the torso whilst merely good players have to wait for the arm movement; and experts have stronger schemas in LTM so can process complex situations more effectively.

This makes me wonder whether there is a stage for trainee teachers, beyond some basic DP of more static skills, where something more like my son’s football training would be useful. If you look at this framework for types of practice, it feels to me like this is sitting in a column like ‘scrimmage’ but in blue, not red. You see, the clever thing about those drills is that by having small groups all working in the same space, some of the complexity of a match is created whilst keeping the drill quite simple. I wonder about getting a trainee teacher cohort working in groups, with several ‘teachers’ having to talk to a group each, whilst scanning and spotting a primed low-level disruption in the room. However, I don’t think having multiple ‘teachers’ working in the same room is going to work for all classroom skills. I do wonder whether there is potential for classroom video and even Augmented Reality (AR) to play a role. Things like having very short video clips, where expert teachers have identified things to spot (someone turning to their neighbour, the unlikely chuckle in a pair discussion, the unopened book or pencil that’s still on the desk, or the child writing instead of listening). Or recordings of children’s oral answers that need instant feedback. I can think of quite a few possibilities like this. There are some GDPR and ethical challenges, particularly with use of video where children are identifiable but these can be overcome. It is also a lot of work to create high quality resources. I would be really interested to know if anyone’s doing, or thinking about doing, this sort of thing.

 

Behaviour

This is Part 3 of three quick posts for trainee teachers, about keeping the main thing the main thing, in the midst of all the complexity of learning to teach.

You’re going to get masses of, occasionally conflicting, advice on each of these themes, so I’m not going to say much – just a few observations from seeing a lot of trainee teachers at work.

This one is about Behaviour (TS3).

If you’re typical of someone starting a career in teaching, behaviour is your biggest concern. I’m not sure whether it’s unnerving or reassuring to know that you may well be overestimating your subject knowledge for teaching, and underestimating the challenge of planning, in comparison to the obvious problem of children just not co-operating with you. You’re certainly right that if you don’t get on top of behaviour, teaching will eventually become an unproductive experience.

The good news is that, whilst a few people have instant presence in the classroom when they start teaching, it’s rare. These are the people that calmly but assertively complain in restaurants – who does that? Most of us find confrontation stressful, negotiate around it in normal life, and tend to default to acquiesence or anger when it’s unavoidable. You have to change a lifetime’s habits here, and that can be hard.

The solution is routines and rehearsal. If you are trying to make up responses on the spot, you’ll go into default mode. You’ll maybe give warnings… and then more warnings… when you need to follow through. You’ll escalate into (real or fake) anger. You’ll start saying the things you learned from when you were at school “It’s your time you’re wasting, not mine!”

There are some excellent little primers out there. Try top ten tips from @tombennett71, or A Bill Rogers Top 10 from @teacherhead (or a Google for Bill Rogers videos should be productive).

I think the most useful thing a trainee teacher can do is to write down all the common things that children will do that you need to address. Children are both highly inventive, and creatures of predictable routine: deal with the routine things. Write down your standard response – script it – discuss with your mentor and other colleagues: refine. Practise and learn. Flashcards are good. Do it out loud as much as you can.

You also need to be crystal clear, and very consistent, about how you cut through (hopefully purposeful) activity and establish silence so you can talk to the class. In secondary, countdowns are very common but what are the children used to? What are other teachers doing (teachers like you – no point trying to copy the one with the booming voice if you can’t boom)? Primary are often awesome at this – clapping sequences are so cool but are still very rare in secondary. Whatever you’re doing, do it right from the start, explain what you’re doing, and make lots of easy opportunities for the class to practise getting it right early on, rather than only using it when things are more marginal.

If you’re having difficulty spotting misdemeanours (trying to identify talkers whilst you’re buried in the register is a classic problem), phone a friend. Get the class teacher (if you’re supernumerary) to give you the names and then you can easily hold them back after the lesson.

Here are some other things to do straight away:

Have a routine for classroom entry. Do it the same, every time. Always have something quiet and busy for them to do immediately. Make sure this has a very low bar for entry so no-one gets stuck, and has an open-end so they don’t quickly finish before you are ready (science example here).

Have a routine for finishing the lesson. Start this early – if they are ready quickly, do some swift oral questions to plug the gap. If you have to cut an activity, so be it; they’ll cope with that. This is so much better than having to rush and then letting them escape without properly tidying up; they’ll remember that! Dismiss gradually to avoid a scrum at the door, and only when they have done exactly what you want.

Where else are the easy wins? Keeping safety glasses on for science practicals is a gift for science teachers, for example. Make the first practical simple so the children don’t need lots of help, and then be Judge Dread on the glasses thing. This works because there is no ambiguity, it’s easy to monitor, and easy to ‘win’. Can you see that this is about planning for behaviour? What else is like this for your phase or subject? Silent working, perhaps?  Trying to manage behaviour, or even just spot low-level disruption, whilst fully extended with teaching is hard; give yourself some chances .

 

So that’s some suggestions for my three big ticket items for teaching: planning, subject knowledge, and behaviour. It takes time but when you get these right then you’ll be in fine shape. The children will learn stuff; teaching will be enjoyable. Good luck!

 

Subject Knowledge

This is Part 2 in three quick posts for trainee teachers, about keeping the main thing the main thing, in the midst of all the complexity of learning to teach.

You’re going to get masses of, occasionally conflicting, advice on each of these themes, so I’m not going to say much – just a few observations from seeing a lot of trainee teachers at work.

This one is about Subject Knowledge (TS3).

If you don’t know it, you can’t teach it. Why not? Well it’s obvious init? But worth analysing a bit, I think. If you don’t know it really well then it’s very hard to break something down into the little steps you need to teach it properly. If you don’t know it really well then even a well-prepared explanation is likely to go awry. The developing understanding in your children’s heads is ever so delicate, and their working memories are stretched to the limit, bless them. Just a little muddling up on your part and they may well end up further back than when you started. What can you do? No shortcuts, I’m afraid. For Primary it’s particularly challenging with an awful lot to learn about teaching reading and, in some cases, some deep conceptual challenges in maths, but the scientists I work with normally only have one science to degree level, and often a third science specialism they never took past GCSE themselves. But although there’s no magic bullet, you will get better and you’re surrounded by subject experts in school. Make use of them! Identify what you find hard, when it’s being taught and you are free, and get into those lessons. You’ll be learning subject knowledge and ideas about how to teach it.

Do consider that anything you can prepare before your own lessons rather than making up as you go, is likely to help. Short video clips, with their carefully edited script, followed by questions, can help to underpin things. Rehearse your explanations and use crib cards – nothing wrong with that. Read short sections from a textbook, then go carefully over the key vocabulary and ideas with the class. A subject expert with great communication skills is always a pleasure to observe but, if that’s not you, remember that it’s the learning that counts so just find ways to support yourself and don’t start to think that if you can’t do without, somehow that makes you a poor teacher.

There is evidence that suggests that knowledge of misconceptions – where the children are most likely to go wrong – and knowledge of particularly good ways of getting ideas across, are also an important part of knowing how to teach your specialism. Indeed it may be that this ends up being the more important knowledge in the end. Hopefully mentors and tutors will point you at appropriate sources of information, but also keep asking the experienced teachers in your school about sticking points with the different things you are teaching. Where the things you are teaching are abstract, think about providing concrete examples – from a number line to support working with negative numbers, to blowing through straws to illustrate why adding a resistor in parallel reduces resistance.

Mark written work as soon as you get the chance. There’ll be plenty of teachers willing to have you roped in. Do (and mark) some past papers; that will help you to know what to emphasise.

That’s about it, I think. Subject knowledge really matters. Sometimes that focus gets lost in all the complexity of classroom practice.

Behaviour, coming next…

Planning

This is Part 1 in three quick posts for trainee teachers, about keeping the main thing the main thing, in the midst of all the complexity of learning to teach.

You’re going to get masses of, occasionally conflicting, advice on each of these themes, so I’m not going to say much – just a few observations from seeing a lot of trainee teachers at work.

This one is about Planning (TS4).

The two things I see all the time that lead to disaster are (a) a vague destination and (b) steps that are not small enough.

Experienced teachers know what they are aiming for without all that much thought. That’s a problem because it’s hard for them to see how much knowledge they have, and how difficult new teachers find this process. Inexperienced teachers write down something like “Be able to describe how plants get their food by photosynthesis” or “Be able to add fractions” and then proceed to carefully plan a lesson without any analysis of what that actually means. For each thing you are trying to teach, you need to know what it will look like when the children have got there, and all the little steps along the way. To start with you probably need to produce an exemplar of what you want them to do and think very carefully about what you are doing as you produce it. Now you know exactly what your destination looks like, and how to get there.

If teaching was as simple as telling, we’d all be so smart it would hurt. Your destination needs to be challenging. If novice learners can get there in one leap then it’s not challenging enough. So you can’t just tell them the whole lot in one go and expect it to work. They just don’t have the capacity for that. It doesn’t matter that it seems easy to you. You need small steps. Tiny ones! … No – even smaller than that! If you make the steps too simple you can get away with it for quite a long time whilst you adjust. Get even one step too big, and all but the highest achieving children will come unstuck within seconds. Small steps! Short and sweet. Lots of checking before you move on. And then, step by step, brick by brick, build something good. Build the children up to match that exemplar of yours. Not a pale shadow of it; work that meets your highest expectations.

Clear destination

Small steps!

 

Subject knowledge coming next…

Keeping the Main Thing, the Main Thing

As another 40,000 trainee teachers gear up to begin treading the boards at the front of classrooms around the country, probably feeling a little like actors thrust in front of a critical audience without much of a script, it seemed a good time to weigh in with some gratuitous advice, to add to the ever-increasing pile that no doubt everyone is pushing from all directions.

At times, teaching can feel impossibly complex. Perhaps you’ve already encountered the quote from Shulman (1987 and a whole load of other places)

“After some 30 years of doing such work, I have concluded that classroom teaching… is perhaps the most complex, most challenging, and most demanding, subtle, nuanced, and frightening activity that our species has ever invented.”

Well, forget that. Teaching does get complex, but it should be perfectly manageable for anyone who has made it through the selection process. It just takes a bit of time to get the hang of it.

Actually, you only really have to get three things right. Within those three things there are lots of little bits, but that means you can get plenty of the little bits wrong without too much damage. An actor has to memorise the whole of a play and get it word perfect, along with all the rehearsed arm waving. You, my friend, can ad lib when you need to. So slap on the metaphorical greasepaint. Remember that you don’t have to let on that no-one gave you a script. Children have an astonishing tendency to overlook, or at least forgive, all sorts of errors if you keep your chin up, don’t flap, speak clearly, and treat them fairly.

Just three things:

• Planning
• Subject Knowledge
• Behaviour

Blogs on each one, coming soon…