Friday, June 25th
How long did Department for Education officials and ministers spend cobbling together the official document they have published assessing the likely impact of their new academies policy?
Not nearly long enough, to judge from what looks like the thrown-together character of this paper, and some truly heroic assumptions on which some of the calculations it comes up with appear to be based.
The paper is an “impact assessment” of the academies bill, which is currently going through Parliament. The idea behind this sounds sensible: when policies come out, the government is supposed to show that it has thought through the implications of them, and considered alternatives.
So far, so good, then. But problems begin to appear for this document in the first paragraph. It opens with the questions: “What is the problem under consideration? Why is government intervention necessary?”
The answer begins: “As part of their [sic] pre-election commitments, the Government said that outstanding maintained schools would be allowed to adopt Academy status by September 2010.”
Erm…this wouldn’t be both parties within the coalition government, would it? In fact, the commitment appears to be a reference to the Conservative manifesto, which said “all existing schools will have the chance to achieve academy status, with outstanding schools pre-approved”. There is no mention, in the Conservative manifesto, however, of the September 2010 date, so the document seems to have got that factually wrong. Leaving that aspect of detail aside, clearly, the Liberal Democrat manifesto is not relevant here, as it says conventional academies should be replaced by “sponsor-managed schools”, accountable to local authorities. The equation of coalition government policy with what appear to be Conservative manifesto commitments occurs several times in the document, which states later on page 1 that the policy “is a Government manifesto pledge”.
Later on page 1, the question is posed: “When will the policy be reviewed to establish its impact and the extent to which the policy objectives have been achieved?” The answer? “It will be reviewed.” So that’s that sorted out, then. An annex at the back of the paper leaves space for the government to fill in a “Post Implementation Review Plan”, with information to be added under seven categories. All have been left blank.
There are other strange aspects of this 11-page paper. Its date is listed as effective from 01/01/2010, rather than the period on or after the coalition government’s formation, which is bizarre. (The document actually bears a signature which could be that of Michael Gove, education secretary, which itself is dated 26/5/2010, which was the day after the Queen’s Speech, when the Academies Bill began its passage through Parliament.)
Against a box in which the department is asked to sum up the policy’s impact on equalities – a note says that “race, disability and gender impact assessments are statutory requirements for relevant policies” – the document says there are no implications. Again, it seems bizarrely, and worryingly, to this observer, counter-intuitive to have reached this position with no evidence offered: what if “outstanding” schools being offered academy status first disproportionately educated children from one ethnic group, for example? To take this document at face value, the government deems this as not worthy of consideration. The assessment also ticks “no” in a section asking the government to identify nine other possible categories of effects, including on small firms, greenhouse gas emissions and human rights.
Under “evidence base”, the document says that academies are freer from regulation than conventional state schools, are independent of local authorities and thus have “far greater autonomy than maintained schools”. They are able to take decisions on priorities “to suit the pupils in the Academy”; are not constrained by national teacher pay structures; and can form partnerships that work for them and their pupils “without constraint”.
“These as part of a broader package of freedom flowing from autonomy, mean that Academies are freer than other schools to focus their efforts on teaching and learning and push up attainment as a consequence.”
Hmm, whatever the details of how particular freedoms may benefit academies, I don’t think it’s true in any general sense that academies have more freedom to “concentrate on teaching and learning” than other schools.
But it is the figures on the likely results improvements at academies, and the highly convoluted arguments about economic benefits that could follow that I want most to focus on here.
The paper attempts to come up with a cost-benefit analysis for the policy. On the debit side, it reckons that “total annual economic costs” flowing from setting up and running another 200 academies a year in 2010-14 will be £462 million. “Total economic costs” (I confess after reading the document I cannot tell the difference between these two terms) are put at £530 million.
But on the credit side, the document estimates that the academies will produce economic benefits – to those individuals educated within these institutions, I think – of £1.072 billion.
The chain of reasoning seems to go as follows.
First, academies have been shown to improve GCSE results, the document says. The paper assumes that a school changing its status from mainstream to academy will generate a 1.5 percentage point improvement in the proportion of its pupils achieving five or more GCSEs, including English and maths.
I cannot quite see, from the paper itself, where this figure has come from. Earlier in the document, it states that 63 academies which had results for both 2008 and 2009 had been found to have improved their five-or-more GCSE including E and M rate by five percentage points, which was twice the national average.
Explaining why the 1.5 percentage point figure had been chosen, the paper says that the figure had been “down-rated by half” to allow for the fact that new academies would be different from those covered by the 2008 and 2009 figures. 1.5 per cent, of course, is less than a third of the five percentage point increase shown by the academies in 2008 and 2009, so this is confusing-or have they taken the 2.5 percentage points that academies improved above the national average, and then more-or-less halved that? Having read the document several times, I don’t know.
The paper then attempts to assign an economic effect to an individual achieving five or more good GCSEs (there is no mention of this having to include English and maths in this section), as opposed, presumably, to not achieving this benchmark. It puts a figure on this as an extra £100,000 in “lifetime productivity” (earnings, I guess), for men, and £85,000 for women.
This, says a note, is based on an “internal DfE analysis” of three existing studies, which seem to focus mainly on the returns for individuals of gaining vocational qualifications.
It appears that this figure is then multiplied by the number of extra pupils that might be expected to gain five good GCSEs, including English and maths, as a result of the academies policy (the document assumes there will be 200 new academies each year to 2014), to produce an overall economic benefit arising from academies of £282 million per year, or £1.072 billion over four years, after taking into account inflation.
It is difficult to know where to start with all of this, without denouncing it as pie-in-the-sky rubbish.
The previous government’s own, largely sympathetic, evaluation of the academies policy, by PricewaterhouseCoopers, found, in November 2008, that “there is insufficient evidence to make a definitive judgement about academies as a model of school improvement”. How the new government can therefore assume that changing schools’ status to academies, in itself, would bring about a 1.5 percentage point improvement above what would have been achieved without it is puzzling.
And trying to predict improvements in academic results in the “new” academies – which seem likely to be drawn from among the more successful schools with, probably, better results as a starting point – on the basis of what predecessors which tended to come from the lower end of league tables achieved, seems especially perilous.
To be fair to the document, it does say towards the end that the figures “may be an overestimate” of the likely results gain, since the new academies “will have less scope for improvement than existing Academies” (presumably because their results are already higher) and “will receive much less start-up funding”.
Fair enough, except that on page 2, it stresses that: “Future benefits are assessed as only half those that have been achieved in past academies. Benefits are therefore likely to be underestimated.” (my italics)
Really, reading this document, you wonder why they even bother doing these impact assessments. I wonder if it has been done in a terrible rush in the early days of the new government, to try to get some political momentum behind the new policy. Whatever, if this is the level of analysis that has gone into this policy, there could be trouble ahead.
You can read the impact assessment in all its glory at: http://www.education.gov.uk/~/media/Files/lacuna/academiesbillimpactassessment.ashx