Michael Fordham has written two interesting blog posts on research methodology (and methods) Part 1 here and Part 2 here. I started to write a comment to Part 1, realised it was a ridiculous length, and then wrote most of this, before spotting Part 2. So I added a bit, et voila! I may possibly be talking rubbish – I’m not a researcher – but this is my take on it, written for the sake of crystallising my own thinking. If you read it and disagree, there’s a fair chance you know more than me.
I think that there is quite a broad spectrum of approaches to education research. At one end is ethnographic research where the researcher becomes a part of the situation they are interested in – the anthropologist who goes through the manhood rituals of a remote tribe; the researcher who describes the politics of their own staffroom. There are accepted methods to this but it’s very subjective because different anthropolgists may experience rituals differently; in the staffroom it will depend on what others think of you. I don’t think this methodology has much to offer research aimed at improving education/teaching but it needs mentioning because I think there is sometimes a mistaken belief that this is what qualitative means, and this is used to dismiss qualitative approaches.
Next there is the action-research methodology where the researcher makes a change in their own practice (or that of their dept/school) and evaluates the effect. This has considerable value (I disagree with Ben Goldacre’s dismissal at ResearchEd2013) but only in improving things on the same scale and context as the research. I think the mistake here is to dismiss it because of the small scale but you have to remember that teaching IS small scale. That’s why the difference between teachers is so much more important than the difference between schools. If you and Y9Z really find working on test-taking skills productive than it’s irrelevant that Hattie’s effect size for this is low – he may have looked at a billion children but he didn’t look at yours! What does matter is that you don’t kid yourself, and that’s the hard bit. Again, there are ways of doing the research that will help you to avoid this.
Finally there is the methodology of research on the large scale – attempting to evaluate ‘what works’ using RCTs, meta-analysis, big cohort studies, mining data, regression analysis, large numbers of interviews, etc. Again, the scale matters. A high effect size or other statistical measure is an average. Effect size of 0.40 could be an effect of 0.80 on one bunch of students and 0.00 on another, or could be around 0.40 for everyone. If meta-cognitive strategies are going down like a lead ballon with Y9A then maybe that’s just Y9A for you. Perhaps their meta-cognitive strategies are well-honed and you’re trying to teach them to suck eggs. Or maybe it’s just your teaching of test-taking skills that sucks. The commissioner of an RCT can’t tell you the difference!
Maybe there are more methodologies along this spectrum; I’m not a researcher. I agree with Michael, that the qualitative/quantitative dichotomy is unhelpful but I think this is less about the blurred line between the two, and more because it is much less important than understanding where on my spectrum a piece of research stands. Michael has suggested that the key is to think about what discipline your study subject falls within, and that’s a good point, but I think deciding whether you are looking for a general effect, or one specific to your own small context, is actually the first step. If it’s M-Level research, it’s probably either action-research, or it’s mining data, for reasons to do with the size of the piece of work. I’ve seen a fair few M-Level assignments trying to do pre- and post- test statistics in action research settings where there is no chance of comparing like with like. This is applying large scale methods to small scale research. Now, quite often the problem is using a quantitative method with data that isn’t valid, and a possible solution is using a qualitative method that digs deeper, but it doesn’t have to be. Replacing pre- and post- test data with Likert-scale questions and something to confirm expected progress might well also be a good solution but that’s not changing from quantitative to qualitative, it’s changing from a large-scale method to a small-scale one, and I think that is what matters.