Thursday, April 09, 2020

COVID-19 stats are actually next to useless

Like pretty much every one else, I have been religiously following the daily statistics on cases, deaths, etc, from COVID-19, deperately looking for that inflection point where things start to look better rather than worse. But I know in my heart that the statistics, such as the Johns Hopkins University Cornavirus Resource Centre, with its wealth of detailed international figures, are at best skewed and at worst downright useless. (The number of deaths attributable to the coronavirus, on the other hand, is a better-supported, and more reliable, measure of the real situation, although even that determination is far from infallible, and believed to be substantially understated.)
That's because, despite the sterling work being done by statisticians around the world, the figures for the number of cases in a population is based on the amount of COVID-19 testing being  that population. And the amount of testing being done varies hugely between countries, states, even cities. Given that an estimated 80% of COVID cases are mild and do not require hospitalization or medical intervention, it seems likely that the vast majority of cases are being neither tested nor recorded.
Setting aside the possibility - no, certainty - that some countries are massaging their official figures for political or face-saving reasons, if one country is doing a much better job of testing its population than another, then its case incidence is likely to be higher, even though it may be actually doing a better job of dealing with the outbreak.
And the amount of skewing we are talking about is huge. ALL countries are understating their number of cases, because nowhere is testing everyone in its population; that's just not practicable or possible. Estimates of the extent of the misstatement of cases suggest that they may be out by orders of magnitude, making any comparison of statistics all but meaningless. For example, two recent studies showed a 20-fold discrepancy in case detection rates between countries who are doing a good job of it (e.g. Norway) and those who are doing a bad job of it (e.g. UK). So, the UK's already horrendous stats could actually be at least 20 times worse than they appear! Other studies and surveys suggest that case totals in the US and UK could be out (understated) by anywhere from 2 times to 100 times! Not very useful data for comparison purposes, then, although arguably of some limited use for identifying trends.
For the same reason, most of those dire and panic-inducing models, projections and prognostications we've been hit with recently are equally meaningless, based as they are on meaningless current and historical figures. Not only are the predictions of future cases and deaths next to useless because they give such a huge range of figures depending on various assumptions (for example, Canada can expect 11,000 to 22,000 deaths in a best case scenario, and up to 100,000+ deaths with weaker controls ... or maybe not). But even those ranges are based on faulty historical data, as we have seen above. This comic strip gives a good idea of the uncertainties involved in modelling COVID-19.)
So, a pinch of salt is most definitely called for when looking at any of these statistics or models, and how things are going to go probably depends more on whether you're a glass-half-full or glass-half-empty kind of person. The only thing we can be pretty sure of, whatever kind of glass you see, is that it's going to get a lot worse before it gets better. And I'm probably still going to be obsessing over those case statistics, even knowing what I know.

No comments: