Talk for RSS Code of Practice Round Table 30th November 2023 Tony Dent, Better Statistics CIC
The invitation to speak had suggested I should begin by describing some aspect of the code that I consider to be particularly useful. I am sorry that I cannot oblige, because I consider the whole of the Code to be of potential value and my primary concern is that the Code should be followed. Nevertheless Better Statistics fully supports this review, because we had suggested a review of the code in the spring of 2022. I quote from a letter sent to the Office for Statistics Regulation sent at that time: “we are primarily concerned with the Covid-19 Infection Survey (CIS) and asking whether … the approximate £1 billion cost of the study, combined with the 13% response rate, provides justification for an urgent review of the Code of Practice to include a specific reference to Value for Money.”
In reply, Ed Humpherson wrote: “We do not consider.. it.. necessary .. to include a specific reference to value for money.. because ..We would expect that any significant value for money issues would be picked up through regulatory reviews as they would be counter to the principles of the Code”
Well, the fact is that the subsequent review of the CIS, conducted by the OSR in the summer of 2022, made no reference to the Value pillar of the code. In particular, that review did not consider clause V5.3 of the code which specifies: “The suitability of existing data, including administrative, open and privately – held data, should be assessed before undertaking a new data collection.” Better Statistics consider the continuation of the CIS in 2022 should have been subject to a comparison of the CIS data with the extensive administrative data then available on the incidence of Covid-19, in particular hospital admissions data. We note that the ONS has now embarked on a Winter CIS, without refreshing the sample and without checking the sample against the 2021 census, let alone conducting comparison analysis with the hospital admissions data.
Also, our letter of March 2022 had pointed out that the Code does not provide any ‘good practice’ guides as to the minimum reporting details on response rates that should be provided for population surveys. More importantly, there are no guidelines as to how potential biases resulting from uneven response rates should be investigated and how they might be resolved through the weighting used to estimate the population data.
The OSR’s response to these observations was more encouraging, stating their intention to discuss such guidelines with the GSS. Whether discussions have taken place we do not know, but we believe that response rates should be calculated within each geographical area used in the sample design and should, as a minimum, cover sex, age, ethnicity, household size and some measure of deprivation and/ or income / wealth. Mandatory reporting of this kind will expose much of the potential for bias in survey responses and will greatly strengthen the Quality pillar, whilst contributing to the perceived Trustworthiness of the resultant data.
And so to the Trust pillar. The objectives of Better Statistics include: Challenging poorly prepared or presented published statistics. And we firmly believe that there is a need to strengthen the code to reduce the amount of published data that is inadequately prepared and is either difficult to understand or, on occasion, simply banal. If the reader does not understand what is being said they will not trust it and we believe that the increasing reliance upon obscure modeling to provide ‘experimental’ estimates, requires more oversight than it is presently receiving. Although the answer lies in improving staff training in report writing (and blogs), Better Statistics believe that an explicit reference to Occam’s razor within the code would reduce unnecessary complexity and improve understanding. That is the key to increased trustworthiness.
In summary we believe that Trust, Quality and Value all need strengthening. But more importantly, so does the regulatory regime and we are concerned that there are no penalties for breaching the code – we believe that also needs reviewing.
Thank you for your time.