In this piece I argue that the scientific method failed during the Covid-19 pandemic – as it always will in rare and extreme events – and that we need other methods to guide our decision-making in crisis environments. There is no point funding ‘scientific’ measures to combat future outbreaks, when those measures are unlikely to be effective—if not totally counter-productive.
Definitions
What is Science? Science is a method of inquiry which uses experimentation, observation, and hypothesis testing to identify characteristics of the natural world (or some part thereof).
What is a Pandemic? A pandemic is an outbreak of a deadly, contagious pathogen which has infected many people across large geographic areas. A pandemic is kind of crisis: a rare and extremely harmful event.
Why does Science fail in a pandemic? Several reasons…
The Long-Term Data Problem
As a general rule in research, the more data you have, the stronger your theories will be. Equally, the less data you have, the weaker your theories – if indeed you can justify any theories all. Science has strict minimum requirements (which is why we can trust it) and a sufficiently large ‘representative sample’ is one of them.
What is our pandemic dataset? Is it representative? Does it allow us to draw scientific conclusions about pandemics and how they should be managed? Let’s consider the potential evidence…
How many outbreaks of dangerous pandemic-potential pathogens (‘PPPs’) have there been in recorded history? Quit a lot, actually. If we included every major outbreak in every country, all the way back to the Plague of Athens, we’d have hundreds. So that’s a good start.
For how many of them do we have robust data? Bar Covid-19, none. (And even Covid-19 is debateable).
Our ability to collect and store this kind of data was minimal for most of human history. Moreover, the germ theory wasn’t confirmed until the late 1800s so, in a way, we didn’t even know what the ‘data’ was. There are major epidemics which killed millions, and yet we’re still debating which pathogens might have caused them. While we can learn from these scraps of data, we have nothing like enough to reach robust scientific conclusions which can dictate our policy-making today.
If we did want to do robust science on pandemics, we would need comprehensive datasets from dozens of major epidemics – per pathogen! We’re nowhere near that point, and I hope we never get there either. Humanity is not long for this world is we have to live through dozens more Covid-19s before we finally learn how to protect ourselves from respiratory PPPs.
The Short-Term Data Problem
So that’s the long-term data problem, but what about when we are in the midst of an outbreak? What can we say about the data available to us and our ability to process it?
Our knowledge of the pathogen will always lag its spread. The first case we identify is rarely, if ever, the first case in the outbreak, so we know from the very start that we are playing catch up. The pathogen will always be spreading somewhere, out of sight, and the official number of cases identified will always the absolute minimum. This data deficit is just a fact of life in epidemic control.
If the pathogen is mutating, then our knowledge will also trail the evolution of its epidemiological characteristics and the symptoms it presents, which further complicates the game of catch up. Every variable needs its own representative dataset, so every mutation puts additional demands on our data collection and processing capacity. In these environments, the variables can multiply faster than the data can be collected, which means science cannot function.
We saw this during Covid, where the pathogen mutated and the strains multiplied. That was a major spanner in the works for the people trying to control the outbreak, but also for the scientists trying to establish its epidemiological features. Then the vaccines arrived, which was good for the people controlling the outbreak but worse again for the scientists, as their data requirements multiplied once more. And all of that was on top of the minimum data needed to stratify the results by country, age, sex, pre-existing conditions, etc.
Even if we did have perfect datasets, the process of scientific research is too slow to make it useful in an epidemic. It takes too much time to collect the data, analyse it, interpret it, publish it, and then get it in front of the relevant policy-makers. By then, the information is already months out of date. It’s hard enough playing catch up, but it’s near impossible when you’re looking in the rear view mirror. And yet, that’s the best that science can offer us.
To be clear, I’m not just saying that science is unsuited to a pandemic, or that it is inadequate and other methods are necessary. I’m saying that, in addition to those points, science cannot function at all! Scientific research stops being scientific in a pandemic. It cannot do what it is supposed to do, much less what we unreasonably expect of it. It’s like a car driving through 5 feet of water: first the engine fails, then the wheels to lose contact with the ground. You don’t have a car anymore; you have some kind of floating wreckage.
The effects of this can be seen in the published academic research.
Contradictions
We have academic papers telling us that masks worked during the Covid-19 pandemic and we have academic papers telling us that they did not (I am deliberately using the word ‘academic’ here, not ‘scientific’).
We have academic papers showing that lockdowns were effective and we have other papers proving conclusively that they were a waste of time.
To this day we have academics telling us that there is “dispositive” evidence that the pandemic began in a wet market in Wuhan, while other papers demonstrate that it cannot possibly be so.
Perhaps my understanding is simplistic or naïve, but if they’re all applying the same scientific method, to the same data… shouldn’t they all produce the same results? Even if we allow for a healthy dose of variation, true science should not produce conclusions that are diametrically opposed to each other. Scientists may disagree on the size of the effect, but not on the sign, surely?
There may be many different explanations for these contradictory results, other effects at work, but I’d argue that the possibility that science simply cannot function as intended in an extreme and dynamic environment like a pandemic would have to be one of them. We will never have enough data to reach a scientific consensus on any matter.
But even if we did, science would still fall short of what we need if we are to permanently protect ourselves from these kinds of threats.
Making Statements vs Making Decisions
Even if we did have exhaustive datasets on pandemics throughout the ages, and even if we did generate comprehensive datasets in real time as the current outbreak unfolded, and even if all of my aforementioned concerns could be allayed, we would still be left with the same question which spurred all that data collection and analysis in the first place: what should we DO with that information?
As has been discussed in this parish before, the purpose of science is to make statements, but in a crisis like a pandemic, we need to make decisions and take actions. Making statements and making decisions are completely different things, and the former does not automatically determine the latter.
Science tells us that the pathogen is becoming more contagious. OK, what should we do about that?
Science screams from the rooftops that ‘masks work!’, but I ask you again: what should we do with that information – wear masks 24 hours a day? The statement of fact does not lead us to a single course of action. Even perfect science still leaves us with the hardest part of the problem.
Some pieces of information can be decisive in determining our actions (e.g. there is cholera in the water ⇒ don’t drink it) but these situations are rare. So even when science is functioning as it intended by providing us with robust statements of fact, we will still need other methods to make the right decisions to safely navigate volatile and dangerous environments. Hence why we need to let go of the scientific
Conclusion
If science really held the answers, wouldn’t we have ‘solved’ Covid-19 long ago? There were so many scientists in the media, in the policy discussions, at the highest levels of influence, and yet the Covid-19 outbreak turned into a global catastrophe.
And if pandemics really were amenable to science, wouldn’t we have also reached a scientific consensus on pandemic policies by now? We’ve had centuries of practice, yet as things stand in early 2024, we seem further from consensus than ever. I’d argue that we are less capable of protecting ourselves from PPPs today than we were in 2019, or even 2003.
Science fails in a crisis, as the Covid-19 pandemic showed clearly. It could not function as intended and it failed to provide us with the answers and direction we needed. It should be clear now that other methods are required.If the world is to protect itself from future PPPs, it will need people who know how to plan and prepare and make big decisions in the moment guiding our responses, not more campus academics or medics prescribing cures. Science is fine in calm seas, but Risk Management is needed in a storm.
My next piece will be a case study of a decision from the Covid-19 pandemic which was of immense consequence. It will demonstrate that not only does the scientific method fail in a crisis, it can lead you to the worst possible conclusions.
3) Science Fails in a Pandemic
In this piece I argue that the scientific method failed during the Covid-19 pandemic – as it always will in rare and extreme events – and that we need other methods to guide our decision-making in crisis environments. There is no point funding ‘scientific’ measures to combat future outbreaks, when those measures are unlikely to be effective—if not totally counter-productive.
Definitions
What is Science? Science is a method of inquiry which uses experimentation, observation, and hypothesis testing to identify characteristics of the natural world (or some part thereof).
What is a Pandemic? A pandemic is an outbreak of a deadly, contagious pathogen which has infected many people across large geographic areas. A pandemic is kind of crisis: a rare and extremely harmful event.
Why does Science fail in a pandemic? Several reasons…
The Long-Term Data Problem
As a general rule in research, the more data you have, the stronger your theories will be. Equally, the less data you have, the weaker your theories – if indeed you can justify any theories all. Science has strict minimum requirements (which is why we can trust it) and a sufficiently large ‘representative sample’ is one of them.
What is our pandemic dataset? Is it representative? Does it allow us to draw scientific conclusions about pandemics and how they should be managed? Let’s consider the potential evidence…
How many outbreaks of dangerous pandemic-potential pathogens (‘PPPs’) have there been in recorded history? Quit a lot, actually. If we included every major outbreak in every country, all the way back to the Plague of Athens, we’d have hundreds. So that’s a good start.
For how many of them do we have robust data? Bar Covid-19, none. (And even Covid-19 is debateable).
Our ability to collect and store this kind of data was minimal for most of human history. Moreover, the germ theory wasn’t confirmed until the late 1800s so, in a way, we didn’t even know what the ‘data’ was. There are major epidemics which killed millions, and yet we’re still debating which pathogens might have caused them. While we can learn from these scraps of data, we have nothing like enough to reach robust scientific conclusions which can dictate our policy-making today.
If we did want to do robust science on pandemics, we would need comprehensive datasets from dozens of major epidemics – per pathogen! We’re nowhere near that point, and I hope we never get there either. Humanity is not long for this world is we have to live through dozens more Covid-19s before we finally learn how to protect ourselves from respiratory PPPs.
The Short-Term Data Problem
So that’s the long-term data problem, but what about when we are in the midst of an outbreak? What can we say about the data available to us and our ability to process it?
Our knowledge of the pathogen will always lag its spread. The first case we identify is rarely, if ever, the first case in the outbreak, so we know from the very start that we are playing catch up. The pathogen will always be spreading somewhere, out of sight, and the official number of cases identified will always the absolute minimum. This data deficit is just a fact of life in epidemic control.
If the pathogen is mutating, then our knowledge will also trail the evolution of its epidemiological characteristics and the symptoms it presents, which further complicates the game of catch up. Every variable needs its own representative dataset, so every mutation puts additional demands on our data collection and processing capacity. In these environments, the variables can multiply faster than the data can be collected, which means science cannot function.
We saw this during Covid, where the pathogen mutated and the strains multiplied. That was a major spanner in the works for the people trying to control the outbreak, but also for the scientists trying to establish its epidemiological features. Then the vaccines arrived, which was good for the people controlling the outbreak but worse again for the scientists, as their data requirements multiplied once more. And all of that was on top of the minimum data needed to stratify the results by country, age, sex, pre-existing conditions, etc.
Even if we did have perfect datasets, the process of scientific research is too slow to make it useful in an epidemic. It takes too much time to collect the data, analyse it, interpret it, publish it, and then get it in front of the relevant policy-makers. By then, the information is already months out of date. It’s hard enough playing catch up, but it’s near impossible when you’re looking in the rear view mirror. And yet, that’s the best that science can offer us.
To be clear, I’m not just saying that science is unsuited to a pandemic, or that it is inadequate and other methods are necessary. I’m saying that, in addition to those points, science cannot function at all! Scientific research stops being scientific in a pandemic. It cannot do what it is supposed to do, much less what we unreasonably expect of it. It’s like a car driving through 5 feet of water: first the engine fails, then the wheels to lose contact with the ground. You don’t have a car anymore; you have some kind of floating wreckage.
The effects of this can be seen in the published academic research.
Contradictions
We have academic papers telling us that masks worked during the Covid-19 pandemic and we have academic papers telling us that they did not (I am deliberately using the word ‘academic’ here, not ‘scientific’).
We have academic papers showing that lockdowns were effective and we have other papers proving conclusively that they were a waste of time.
To this day we have academics telling us that there is “dispositive” evidence that the pandemic began in a wet market in Wuhan, while other papers demonstrate that it cannot possibly be so.
Perhaps my understanding is simplistic or naïve, but if they’re all applying the same scientific method, to the same data… shouldn’t they all produce the same results? Even if we allow for a healthy dose of variation, true science should not produce conclusions that are diametrically opposed to each other. Scientists may disagree on the size of the effect, but not on the sign, surely?
There may be many different explanations for these contradictory results, other effects at work, but I’d argue that the possibility that science simply cannot function as intended in an extreme and dynamic environment like a pandemic would have to be one of them. We will never have enough data to reach a scientific consensus on any matter.
But even if we did, science would still fall short of what we need if we are to permanently protect ourselves from these kinds of threats.
Making Statements vs Making Decisions
Even if we did have exhaustive datasets on pandemics throughout the ages, and even if we did generate comprehensive datasets in real time as the current outbreak unfolded, and even if all of my aforementioned concerns could be allayed, we would still be left with the same question which spurred all that data collection and analysis in the first place: what should we DO with that information?
As has been discussed in this parish before, the purpose of science is to make statements, but in a crisis like a pandemic, we need to make decisions and take actions. Making statements and making decisions are completely different things, and the former does not automatically determine the latter.
Science tells us that the pathogen is becoming more contagious. OK, what should we do about that?
Science screams from the rooftops that ‘masks work!’, but I ask you again: what should we do with that information – wear masks 24 hours a day? The statement of fact does not lead us to a single course of action. Even perfect science still leaves us with the hardest part of the problem.
Some pieces of information can be decisive in determining our actions (e.g. there is cholera in the water ⇒ don’t drink it) but these situations are rare. So even when science is functioning as it intended by providing us with robust statements of fact, we will still need other methods to make the right decisions to safely navigate volatile and dangerous environments. Hence why we need to let go of the scientific
Conclusion
If science really held the answers, wouldn’t we have ‘solved’ Covid-19 long ago? There were so many scientists in the media, in the policy discussions, at the highest levels of influence, and yet the Covid-19 outbreak turned into a global catastrophe.
And if pandemics really were amenable to science, wouldn’t we have also reached a scientific consensus on pandemic policies by now? We’ve had centuries of practice, yet as things stand in early 2024, we seem further from consensus than ever. I’d argue that we are less capable of protecting ourselves from PPPs today than we were in 2019, or even 2003.
Science fails in a crisis, as the Covid-19 pandemic showed clearly. It could not function as intended and it failed to provide us with the answers and direction we needed. It should be clear now that other methods are required.If the world is to protect itself from future PPPs, it will need people who know how to plan and prepare and make big decisions in the moment guiding our responses, not more campus academics or medics prescribing cures. Science is fine in calm seas, but Risk Management is needed in a storm.
My next piece will be a case study of a decision from the Covid-19 pandemic which was of immense consequence. It will demonstrate that not only does the scientific method fail in a crisis, it can lead you to the worst possible conclusions.