What Might Stop Excellent Academy Chains From "Scaling Up"?

The Coalition’s education ministers seem convinced that academy-chains are  “the next big thing”. Money is available for academy sponsors to take over failing(ish) schools, and chains are an increasing player in upcoming ‘Free Schools’. Theoretically, ‘successful’ chains will deliver the economies of scale and quality assurance of LEAs, while also being free of unions, pesky “regulation”, or requirements to go and account for oneself among local government representatives. You can see why Conservatives like the idea.
The problem? Evidence from the States suggests that really successful academy chains tend not to ‘scale up’. KIPP, the most-discussed, incredibly successful US chain, operates 125 schools. In a country of 100,000+ schools. Not because it “needs time to grow” – the chain started in 1993. No, smallness is a conscious choice.
Why? Steven Wilson, former Harvard fellow and CEO of a charter school chain, argues that chains are constrained by finances and human capital. Education reformers too often believe money and talented people are sitting around waiting to be found and used in a new school venture. But sadly, it’s not so.
After interviewing 10 chain leaders, Wilson found 5 things that limit school chain size. They’re here, along with how they might influence the UK:
(1) Political risk – This is less of a problem in the UK as Labour seem unlikely to reverse academy policy. But aggression at local government level still exists in some areas, particularly towards academy chains, and it can be off-putting.
(2) Unrealistic business plans – This has hampered almost every US chain. Again, it’s down to that false optimism about money and people.
(3) Start-up skills requirement – Opening schools takes a lot of skill and not every chain can afford in. In a country where schools have regularly been locally planned it’s also unlikely there will be enough people with these skills to share around.
(4) Undisciplined client acquisition – Chains take on schools without really knowing what they are getting themselves in for. Then they bomb. And then they get frightened off from ever expanding again.
(5) – Uneven design implementation – The chain takes over a school without a clear plan for explaining how it will change the school so it reflects the chain’s image.
So far I’m not hearing conversations about these barriers. Instead I’m hearing more and more of the false optimism: “Of course academy chains will spring up”…”There’s definitely enough talented people”…”The government has plenty of capacity money”… But it’s not true. There’s no definitely, plenty or ‘of course’ about this. Like a rebellious elder sibling, the US made these mistakes already so we don’t have to. If England wants academy chains it must work to get them and the sooner we get past fantasy and into detail the better off we’ll be.

Helping out @toryeducation

This morning @toryeducation asked me to say “something useful” about a piece of research they sent a link to.  I was surprised to find the piece is about physics lectures for 850 undergrads in a Canadian university. My blogging and tweeting is almost exclusively about secondary/FE education – not least because Higher Education policy is not a remit of the Department for Education – so this seemed a bit of a bizarre piece to be asked to comment on.
Still, I like education research. Currently I spend my days commenting on and synthesizing it, so I cast my eyes over the research and my views are below. I hope that @toryeducation is satisfied, but I remain surprised that they thought this paper to be ‘not trivial’. Perhaps it’s possible they do not have a subscription to the full magazine and so had not fully read it. Hopefully the below comments will help give a better understanding of the piece.

Improved Learning in a Large-Enrollment Physics Class – Deslauriers, Schelew & Weiman (It’s behind a paywall for most people sadly. Luckily I have a subscription)

“Sight Check” – i.e. What clues do I have for its validity before I start reading
– This paper is about a university physics lectures attended by large numbers of students. The likely relevance to secondary education policy is nil.
– It is in Science magazine. Science only peer reviews some articles, not all. This piece was in the ‘report’ and not ‘research’ section. I am not certain it circumvented peer-reviewing, but I anticipate this to be the case. Update: Report sections are peer-reviewed. This is good to know. Unfortunately because reports are only 2k words there is still a lot of information not available to the reader.
– There are two ‘associated letters’ highlighted in the sidebar of the full article (or below the article in the abstract-only version in the link above). When clicking on these, one is shown to be a letter from a Professor of Education from the University of Birmingham (UK), and the second one is a multi-author letter from Biology departments at several US universities. Both highlight significant design flaws: high attrition rates, treating the experiment as a randomized control trial when it was a quasi-experiment, the use of a single teacher in one condition but two teachers in the other condition, and an absence of validity and reliability checks.  While one can always pick holes in scientific research having so many problems makes me sceptical as I go into reading this article. The authors will need to do a really good job of explaining these issues (spoiler: they don’t).
– Finally, the report is only 2 pages long. To accept research as useful for policy I would usually want more detail than a 2 page report can give.
The Actual Report
Putting aside my concerns for now, let’s see what the paper says:
–  In our studies of two full sessions of an advanced quantum mechanics class taught either by traditional or by interactive learning style, students in the interactive section showed improved learning, but both sections, interactive and traditional, showed similar retention of learning 6 to 18 months later (10).” < So the argument is that interactive learning gives you a more improved ‘immediate’ sense of learning but traditional lecturing and interactive learning both have similar levels of retention later on.
– “The control group was lectured by a motivated faculty member with high student evaluations and many years of experience teaching this course. The experimental group was taught by a postdoctoral fellow using instruction based on research on learning” < How do we know that any variation isn’t down to the fact that two different people delivered the groups? Given they are quite different – in level of responsibility, experience, possibly age – there are confounding variables here that are not adequately dealt with in the later data analysis.
– In the experimental group, the students were “making and testing predictions and arguments about the relevant topics, solving problems, and critiquing their own reasoning and that of others”… “as the students work through these tasks, they receive feedback from fellow students” < A lot of this sounds like assessment for learning already embedded in most schools throughout the 2000s so there is no relevance to education policy except carrying on!
“We incorporate multiple “best instructional practices,” but we believe the educational benefit does not come primarily from any particular practice but rather from the integration into the overall deliberate practice framework” < This a bit worrying, it sounds like the instructor just did whatever they thought was best practice. Without more detail there is nothing here that could be repeated in education policy in the UK (HE or otherwise)
– (!) Something useful < I had suspected there would be some use of clickers in this article, and they are there. Clicker use is common in the US and is something that I think could be used fruitfully in the UK although we have a horrible history of ICT implementation. Hence the only ‘useful’ thing for the UK so far in this piece is that Clickers are something individual schools might wish to invest in (particularly among 16-19 yr olds)
– Figure 1 shows the test was out of 12 < The validity of checking knowledge with only 12 response items is likely to be very low.
 “The result is likely to generalize to a variety of postsecondary courses” < The final statement from the author is that this is a postsecondary piece. I therefore maintain my further claim that this research is largely irrelevant to UK education policy.
 What can I say that is “useful”?

  1. The authors do not think this is generalisable to teaching in schools. There is no evidence to suggest that it is generalisable to UK school policy. I therefore reassert that it is irrelevant.
  2. There is no ‘method’ tested here shown to be ‘better’ for learning however it would appear that (a) the use of clickers, and (b) more ‘working together’ on physics problems influences short-term learning as tested in a 12-mark multiple-choice test.
  3. The research design is not well explained given the 2k words, the experimental condition is vague and unrepeatable, and there are no published validity or reliability checks. Without further information I would not use the piece to inform UK HE education policy either.

Did Gove Implement Comparable Outcomes?

Part of the furor over the English GCSE Fiasco last summer was the use of something called ‘Comparable Outcomes’ – a method through which Ofqual limits grade awarding by requiring that the number of grades given are comparable to the previous year’s cohorts or to exams the current cohort took when younger.
Much was made of the role ‘comparable outcomes’ in the lower-than-expected numbers of students getting English C Grades in August 2012. Gove, however, was adamant that the adoption of comparable outcomes had nothing to do with him. In fact, in an Education Select Committee evidence session about this issue he said:

Q4 Michael Gove: The first thing that I would say is that the comparable outcomes framework was something that was designed and adopted before this Government came to power. It was a previous Government that, under the QCA, as it then was, and then subsequently under the previous leadership of Ofqual, adopted comparable outcomes and outlined how it should work, first of all with respect to A levels and then with GCSEs. So the current team at Ofqual are dealing with tools that were designed by the last Government, rather than tools they have had a chance to fashion themselves.

Except, a recent FOI release from the DfE seems to contradict this fact. In a huge (and very delayed) release of correspondence between the DfE and Ofqual, one of the documents has a ‘comparable outcomes’ timeline and specification box on its last two pages. This is what the box shows:
gcse adoption of comp outcomes
It’s a bit tricky to make out but the fifth column is titled “Decision on use of comparable outcomes”. The decision to use it in new GCSEs in “most subjects”, the “new English suite, maths, ICT” and “New science suite” were all taken on 6th December 2010. After the current government came to power.
Now, I want to make clear – this does not mean Gove is to blame for comparable outcomes. In the 7 months between taking office and this decision it’s entirely probable that Gove didn’t know what was going on at Ofqual. After all, he was busy trying to push free schools and academies as quickly as possible. Secondly, the decision appears to have been made at an Ofqual board meeting on 6th December and it’s not clear how the results of these meetings were being fed back to the DfE if at all. Gove’s ignorance of the matter is therefore plausible, although to argue that the previous government adopted comparable outcomes is perhaps over-stating the fact.
However, before the policy was fully implemented in the 2011 exam season there was a report to Ministers about what was going on (although the date is unknown). Then, the FOI release shows how in the Summer of 2011 there were some issues that foreshadowed what would occur in 2012 – particularly in AQA’s History GCSE. So concerned is the tone in some of the released emails they speak of “letting teachers know” in advance that results are likely to be lower in order to start managing expectations.  It therefore seems likely that Gove knew before Summer 2012 there would be lower grades. And though he didn’t introduce comparable outcomes, given that the policy was only documented (and not even yet fully implemented) at the end of December 2010 was it really impossible before Summer 2011 – or even after it – to stop the policy if one were really against it? Or is it possible that the ‘crisis of confidence’ that one email refers to as a possible consequence of comparable outcomes was, in fact, a great way to push for a new exam system?
The many many FOI releases are here if you want to look for yourself in order to come to a conclusion. For what it’s worth I don’t think there’s any evidence that Gove acted improperly during the GCSE Fiasco, but I do object to him blaming comparable outcomes on the previous administration when its implementation happened under his watch.