How Did ONS get the projections and estimates so badly wrong for Coventry?
How Did ONS get the projections and estimates so badly wrong for Coventry?
Notes provided by Merle Gering[1], Campaign for the Preservation of Rural England (CPRE)
Purpose:
This brief paper sets out a timeline of the events concerning the controversy surrounding the ONS estimates pf the population in Coventry from 2014.
The Census results published on 28 June 2022, estimated the population of Coventry at 345,000 This was far short of the most recent ONS estimate of 387, 000 and also short of the 2014 projection of 374,000. How did ONS get it so wrong, and why did they ignore critics?
Past History:
2001 – the census reported that Coventry grew very slowly in the 1991-2001 decade – by only 2.2%.
2001-2009. ONS annual population estimates had the city growing at a very high rate. Any thoughtful person would have asked, “Why should the city suddenly explode?” In the 2009 Examination of the then local plan, campaigners asked this question of the council planners, the planning inspector, and council members. None had an answer.
2010 – Coventry Council changed political control and rejected the Draft 2008 Local Plan.
2011 – The 2011 census showed the ONS population estimates has been wildly wrong. The city had grown only half as fast as ONS estimates claimed. There was no need, for even half of the 33,000 homes posited by the 2008 local plan.
Nov 2011. ONS publishes their future forecast, the Sub National Population Projections [SNPP2010] which still showed Coventry growing much faster than its neighbours as shown on this graph:
A priori, outliers like this are always suspect, especially given ONS failure to get population growth estimates right in the decade 2001-2011. In 2020, when CPRE made a formal complaint, Sir David Norgrove, Chair of the UK Statistics Authority, straight away said, ‘they should have looked at outliers and done a sense check” The mathematical chance of seeing Coventry growing like this, was very low – little better than 1 in a thousand in SNPP2010. Red Lights should have been flashing.
Anyone looking at the graph would also have to ask, what changed in 2010? Was there a change of methods or evidence gathering? We know that ONS made changes around this time, and thought they had fixed the problem of unattributed population change – which meant they felt justified in not continuing to apply the UPC they added to all the figures from 2001-2011. Did they make things worse?
2016 – Coventry publishes a new local plan, based on SNPP2014, with population growth vastly exceeding its neighbours, this time for 44,000 homes over 20 years, claiming that vast swathes of green belt were needed in Coventry, and in neighbour authorities to accommodate 100,000 additional residents.. At the local plan hearings, in 2016-17 objectors showed in detail, from official HESA and ONS data, how the ONS projections were wrong: Births were overestimated by 1,000/yr, deaths underestimated by 200/yr, and student emigration underestimated by more than 3,000/yr – taken together, population growth was grossly inflated. A 2013 Royal Society of Statistics Report confirmed the concerns raised about migration numbers:
“There is a continued problem with the quality and quantity of migration data available at a local level”
The local plan inspector, the planners and the council consultants asked not one question about the objectors analysis and evidence, which had been supported by two internationally eminent demographers [Prof Anthony Champion and Prof David Coleman]. They averted their gaze from inconvenient facts.
Objectors provided corroboration. If the population was exploding as claimed, one would have legitimately expected pressure on house prices, and schools, and an increase in jobs. In fact, house prices were among the lowest in the country,, numbers of homes delivered was just average, job creation was mediocre, and school admissions were actually falling. Objectors repeatedly urged the inspector, Rebecca Phillips, to order an independent review of the population numbers. She took no notice, never once asked the Council for an explanation of why there was no sign of extraordinary growth.
In 2017, the former leader of the Council, John Mutton, stated in public that the ONS numbers were “stupid”, but claimed that the Council was forced to use them. Objectors asked again and again to see the documents or guidance which ‘forced the council’, but it was never forthcoming. The NPPF and PPG simply stated that the ONS numbers were a starting point, and a council wishing to depart from them would need to provide robust evidence.
Recent History:
2018 – Giving a much expanded analysis, objectors appealed to ONS to correct their numbers. In addition to the evidence about student migration, births, and deaths,, they now showed that there was no sign of sharp growth in a whole range of administrative data,. There was no trace of hordes of extra people in A&E attendance, school admissions, voters registered, car ownership, pensions and benefits, births, deaths, house completions, house prices, jobs, domestic gas and electricity consumed, and overall tonnage of domestic waste collected.
ONS did agree to meetings with the objectors – in July 2018 -, but never engaged with the arguments and evidence presented. They never said, “you are wrong about this number. You have made a mistake in your calculation here. You have failed to take account of X Y or Z”. Their letters were a polite fob off. They refused to correct or qualify their own estimates or projections. Craig Tracy MP [in 2019] and Andy Street Mayor of the West Midlands, [in 2020 and 2021] also wrote to them, and got a similar response.
However, subsequent evidence supplied through freedom of information showed that internally the ONS had conceded that they had never really answered Andy Street’s letter.
Apparently, this was standard ONS operating procedure. A 2021 FOI request revealed this internal comment from a senior ONS manager to his colleagues: “are you merely phoning them to say “those were the numbers, live with it, there’s nothing more we can do”? The biggest qualm we’ve heard as feedback – for all ONS outputs not just yours – has been that “ONS is willing to hear what people say, but seems to do literally nothing with it, they just do what they were going to do anyway with a very broad-brush and little local insight”. In one instance with Manchester, that lead to somewhat vigorous bristled debate.
In the period 2018-2020, Objectors, MPs and Andy Street wrote at least twice to the Secretary of State, asking him to use his Section 31 powers to call in major applications in former green belt land, in and around Coventry, on the grounds that the population estimates were grossly exaggerated and the homes were not needed. The SoS declined to intervene, saying that decisions were best made at the local level.
In 2021, the Office of Statistics Regulation [OSR], the regulator for ONS, investigated the ONS population estimates and projections, at the request of CPRE, Andy Street, and 6 MPs. The complaint being that inaccurate population numbers were destroying the countryside – providing for vast numbers of homes that were not needed.
Initially the OSR appeared to be supportive of they wrote on 11 May 2021:
‘We found that the population estimates for some cities such as Coventry, did seem to be inconsistent with, and potentially higher than, local evidence would suggest.”
However, that observation did not end up in their report. Moreover the OSR weakened their final report by removing this passage from the draft at the request of the ONS: “fixes have not done enough to address the overestimation of these groups in some areas. This has led to an over reliance on insufficiently robust data to inform local planning decisions such as the need to build additional schools and housing.”[obtained through FOI]
They also omitted judgments from the national Statistician, Sir Ian Diamond and the Chair of the UK Statistics Authority, Sir David Norgrove: “On the issue of the Coventry casework, he [Sir Ian Diamond] said his initial reaction was that the people who’d written to us had raised a good point.” [Ed Humpherson email]
“It looks on the face of it as though they have a strong case and it is the kind of thing where ONS should be engaging, especially given the weight that MHCLG are putting on these numbers.” [David Norgrove email]
As a result Coventry Council brushed the OSR report aside, saying that it was “not definitive”, and did not require the council to modify its plans for building on land removed from green belt by the local plan.
OSR claimed that they did not have access to the data the objectors had used so they could not produce an alternative figure for Coventry population. This despite the fact that all the data was official information, readily available to ONS. Objectors had actually supplied two alternative estimates, based on housing completions and vaccinations, which when compared with the census, turned out to be reasonably accurate [they estimated Coventry population at 337,000, and 346,000 respectively, compared with the Census figure of 345,000].
6 weeks after the 2021 OSR investigation was published, the ONS published their latest estimate of 379,000 for the Coventry population MYE2020, without any warning or caveat to users that the number was in any way suspect. ONS had been informed of the disastrous consequences these numbers were having, and they did nothing to correct them.
By the time the 2021 census was published, at the end of June 2022, it was too late to retrieve the situation. Planning permissions had largely already been granted and there was no going back.
Unfortunately, this is not a new story. Lord Andrew Tyrie, then an MP and chair of the House of Commons Treasury Committee wrote in 2016
It’s clear from Sir Charles [Bean’s] interim report that the UK has fallen behind other countries, and that some of our statistics are scarcely fit for purpose.
“The ONS has fallen a long way short, lacking intellectual curiosity, prone to silly mistakes and unresponsive to the needs of consumers of its statistics.”
What has gone wrong?
There is a culture of complacency and self-satisfaction at ONS – When they were asked if they would correct the Mid-Year Estimates after the 2021 OSR report, Rich Pereira, Director of Population Division, wrote,
- “’Plans to correct’ implies an error…of which we have made none.” and another wrote,” It makes me wonder whether we really have a complaints procedure and if so, whether it could be improved to meet best practice, should that exist” elsewhere, the same manager comments, again to internal colleagues, “Maybe I’m getting too used to the weaselly words approach!”
- The regulator, OSR, is compromised. The same person, the national statistician, is chief executive of both ONS and OSR. The regulator will always have a conflict of interest, between criticisms of ONS, and protecting the reputation of ONS.
- ONS is not accountable to anyone.
- The public cannot intervene, even through the courts. The doctrine of planning judgments, means that Councils can accept any evidence, and the courts will not look at the quality of evidence
All FOI quotations in this paper are published online
and
[1] Contact: Merle Gering, [email protected]