In earlier posts I argued that pandemics could be solved, but only by using the methods of Risk Management, not Science. I further argued that Science fails in a pandemic: not just that it is an inferior or inadequate method, but that the method itself cannot function in a crisis like a pandemic. In these kinds of environments, a misplaced trust in science can do more harm than good. In this piece, I will present a case study from the Covid-19 pandemic which demonstrates the weakness of science as a decision-making tool. The WHO’s delay in confirming the presence of human-to-human transmission was one of the most consequential mistakes in the whole pandemic, and yet, as this piece will show, it was entirely avoidable – had they chosen the correct method of analysis.
Setting The Scene...
It’s the third week in January 2020. A novel coronavirus has been identified in China. Official reports suggest there is little cause for concern: local health officials are on top of the outbreak and there are no signs of human-to-human transmission.
Data from external sources suggests otherwise.
Cases have been identified in Thailand, Japan, South Korea, and the USA, and such diffusion would be very unlikely, were the virus not spreading from human to human (‘contagious’). The conflicting accounts created a tricky situation for the world’s policy-makers.
If the virus was not contagious, then the only people who would be at risk of infection would be those who were in close contact with the animal source in the Huanan Seafood Market (the presumed origin of the outbreak). In that case, the outbreak would be geographically restricted to Wuhan, with a handful of cases turning up elsewhere in China and perhaps even overseas if some of those infected just happened to have flights booked for the following weeks. The Chinese government would have some work to do suppressing the outbreak at home, but for the rest of the world, it would just be another ‘one of those things’ that only ever happens ‘over there’.
If the virus was contagious however, then all of that went out the window. With modern air travel, a contagious pathogen could be spreading on the other side of the world in under 48 hours. Given the smattering of cases already identified overseas, that process might have begun. In that case, national emergency protocols would have to be implemented immediately; workers and resources would need to be requisitioned for the task; every part of society would have to be informed and prepared… if it was contagious.
The difference between these two scenarios could not have been more extreme, and so the question of contagiousness took on monumental significance for the management of the outbreak – not to mention for the management of international relations, the stability of our societies, and the health of every human being on the planet. It was crucial that the world got the right answer, and sooner rather than later. The responsibility for answering that question, and the world’s attention, fell to the WHO.
To understand how the WHO approached their task, we will analyse the problem from a scientific perspective. That is, through the lens of the scientific hypothesis test.
The Scientist’s Perspective
When SARS-CoV-2 first appeared in January 2020, scientists knew little about its epidemiological characteristics. It was a novel virus – how could they? Lacking this key information however, they were in no position to advise the world on appropriate responses.
Adding to the scientists’ difficulties was the precariousness of the situation, the significance of any conclusion, and the implications for the world and everyone in it. For any scientist facing that dilemma, grounded in the philosophy of hypothesis testing, the appropriate course of action was clear: wait for the data to accumulate and let it make the decision for you. Of equal importance: do nothing until then.
The question was too consequential for the analysis to be rushed. It would be foolish to panic the world over what could turn out to be a minor outbreak, and not least because it would elicit global opprobrium for the WHO. Why take the risk? Getting the right answer was key, so the WHO would take their time to ensure that they did. As per the hypothesis test, they would wait until the evidence was pointing conclusively in the direction of contagiousness, and only then would they inform the world. Until then, the null hypothesis of no contagiousness would remain their default position.
And so, despite many early warnings from Wuhan and abroad, the WHO continued to urge caution into the second and third weeks of January 2020 while they waited for conclusive evidence to emerge. It was not until the fourth week of the outbreak that they relented and the world was officially notified that the early warnings were in fact correct: this novel coronavirus was transmitting from human to human, the world was facing a truly global pandemic, and everyone was at risk.
To the risk managers watching – gasping in disbelief – it had seemed an act of almost deliberate self-sabotage.
The Risk Manager’s Perspective
Given that this was a novel pathogen, we knew little about it, which meant that every possibility remained on the table, because if you don’t know what it is, then you can’t say what it isn’t, can you?
Since we didn’t know how virulent it was, it could have been very virulent. We didn’t know how deadly it was, so it could have been very deadly. We didn’t know how effective our medicines and therapeutics would be, so it was possible that they wouldn’t work at all. We didn’t know whether it transmitted from human to human, so it was possible that it did – and very well too.
We knew almost nothing about the virus at that time, so it was possible that it could have been anything, including all of the above and worse again.
Or, it could have been relatively benign. Again, we had no way of knowing either way.
But at least we knew that we didn’t know, and that was something (‘known unknowns’). We knew that high virulence and contagiousness were possibilities – we knew that they were risks – so we knew we needed a plan for them. By the principles of risk management: if the risk exists, then you must manage it; if it is possible, then you must prepare for it. This is how risk managers keep themselves (and the rest of us) alive.
So, as things stood on the 1st of January 2020 (the 1st, not the 22nd) a risk manager would have determined that this virus could have been highly-infectious, it could have been highly-virulent, and it could have been very efficient at transmitting from human to human. These risks would have been clear, and therefore the need for an immediate risk management response would have been too.
From the very first day of this outbreak, the risk manager would have seen the need for urgent action. Instead, three weeks later, the scientists at the WHO were still sitting on their hands waiting for the data to make the decision for them.
Science Fails Again
In science, the null hypothesis is of no effect: there no difference between the variables in question, there is no relationship between them, nothing going on. The base case assumption is always the ignorance of the status quo, and we need overwhelming evidence to convince ourselves otherwise. In the case of SARS-CoV-2 in January 2020, the WHO’s base case assumption was that there was no contagiousness, and it took until the 4th week of January before it became impossible for them to continue to deny the awful reality of the situation.
This may have been the ‘scientific’ thing to do, but the delay was catastrophic for the world. By the time we had discovered the pathogen, it had already been given a 10+ week head start on us, and the WHO’s unnecessary deliberations cost us at least another couple of weeks. Given that the outbreak could have been tripling in size every week at that time (and some estimates put that growth rate at 4x) this was equivalent to allowing the outbreak to increase by a factor of 10 or more while they waited. It’s hard enough playing catch up with a PPP at the best of times, but it’s near impossible when the WHO take so long to get out of first gear.
Had the risk managers been in charge, things would have been different.
The risk managers would have responded to the threat much quicker than the scientists because they only need confirmation of the risk to act, not overwhelming evidence that the risk has materialised. The mere existence a novel pathogen circulating in Wuhan was all the risk managers needed to know to justify an immediate response. Had the world applied their method of analysis to the question of contagiousness, the outbreak would have been 1/10th or even 1/30th of the size at the time, and a much easier problem to solve.
With the full benefit of hindsight, an immediate and forceful response on the 1st of January was the optimal course of action. As I have argued above, the risk managers would have provided it. It is a matter of fact that the scientists could not. The key takeaway is that risk management can be orders of magnitude more effective than science at guiding decision-making in situations like these.
Type 1 vs Type 2 Errors
The scientists and the risk managers are effectively testing the same hypotheses against the same datasets, but when we move from the scientific method to risk management, the null and the alternative hypotheses swap. For the scientist, the null is inaction and the alternative is action. For risk managers it is the reverse. The scientist needed proof that it was contagious to act, whereas the risk manager needed proof that it wasn’t contagious to stop acting.
In Science, it is worse to believe something that isn’t true (Type 1 error) than it is to not believe something that is true (Type 2 error) as anything built on falsity will eventually collapse, while the truth may yet be discovered another day. The same logic applies to criminal trials with the defendant’s presumption of innocence until proven guilty (in liberal societies at least).
In a pandemic however, the asymmetry in the errors flips. When the thing in question is a threat (a pathogen, wild animal, or storm) then the greater risk is in the Type 2 error: doing too little too late, not too much too soon. If it is possible that something very bad could happen, then you must prepare for that scenario, even if it is only a possibility. Over-protect yourself and you have wasted some resources; under-protect yourself and you die.
We know this already, which is why we put our seatbelts on before we start the car, install fire-doors during the building’s construction, and save funds for a rainy day. We don’t know whether the bad thing will happen, but we protect ourselves in advance, just in case. These are all forms of risk management, and they are so common and so deeply engrained in our culture that we hardly think about them anymore.
And yet the ‘scientific’ approach to epidemic control ignores this logic. The scientific method can’t deal with contingencies – that’s not what it’s designed to do. Science’s purpose is to know what can know for sure, but in a crisis, this need for certainty can yield the worst possible outcome: overlooking risks until it is too late to do anything about them.
Conclusion
The WHO’s mistakes in the early stages of the outbreak perfectly demonstrate both the failure the scientific method in a crisis, and the superiority of risk management (aka decision making under uncertainty) in volatile, uncertain, and dangerous environments. It also shows us what could have been, and how we could do better next time.
The WHO’s reluctant admission of human-to-human transmission on the 22nd of January was met with a flurry of border control policies, all over the world. This demonstrates the significance of the announcement and it also shows us that the nations of the world were paying attention to it. Had the risk managers been in charge, they would have recognised the problem sooner, the world would have responded sooner and more forcefully too, and health outcomes would have been better for everyone. Instead, the WHO tried to do science and… well, the rest is history.
Sadly, this faulty approach is so pervasive among the scientists in the WHO, that they made the same mistake when delaying the declaration of a PHEIC in January 2020, and again in the official declaration of a pandemic in March 2020. Too slow, too late, again and again. We simply cannot solve the problem of pandemics with this thinking. On the contrary, it will sleep walk us into one catastrophe after another.
If we are to protect humanity from PPPs – whether naturally occurring, lab leaks, engineered, bioweapons, bioterrorism, or any other – a wholesale change in philosophy will be required. We must let go of our implicit assumption that science is omnipotent and scientists infallible, or that the solution to pandemics can be found in a novel compound or a new piece of technology. Instead, we must recognise and embrace the inherent uncertainty of these problems, and with it, the philosophy of risk and the practice of risk management.
4) A Case Study in Pandemic Policy Failure: Covid-19 & Human-to-Human Transmission
In earlier posts I argued that pandemics could be solved, but only by using the methods of Risk Management, not Science. I further argued that Science fails in a pandemic: not just that it is an inferior or inadequate method, but that the method itself cannot function in a crisis like a pandemic. In these kinds of environments, a misplaced trust in science can do more harm than good. In this piece, I will present a case study from the Covid-19 pandemic which demonstrates the weakness of science as a decision-making tool. The WHO’s delay in confirming the presence of human-to-human transmission was one of the most consequential mistakes in the whole pandemic, and yet, as this piece will show, it was entirely avoidable – had they chosen the correct method of analysis.
Setting The Scene...
It’s the third week in January 2020. A novel coronavirus has been identified in China. Official reports suggest there is little cause for concern: local health officials are on top of the outbreak and there are no signs of human-to-human transmission.
Data from external sources suggests otherwise.
Cases have been identified in Thailand, Japan, South Korea, and the USA, and such diffusion would be very unlikely, were the virus not spreading from human to human (‘contagious’). The conflicting accounts created a tricky situation for the world’s policy-makers.
If the virus was not contagious, then the only people who would be at risk of infection would be those who were in close contact with the animal source in the Huanan Seafood Market (the presumed origin of the outbreak). In that case, the outbreak would be geographically restricted to Wuhan, with a handful of cases turning up elsewhere in China and perhaps even overseas if some of those infected just happened to have flights booked for the following weeks. The Chinese government would have some work to do suppressing the outbreak at home, but for the rest of the world, it would just be another ‘one of those things’ that only ever happens ‘over there’.
If the virus was contagious however, then all of that went out the window. With modern air travel, a contagious pathogen could be spreading on the other side of the world in under 48 hours. Given the smattering of cases already identified overseas, that process might have begun. In that case, national emergency protocols would have to be implemented immediately; workers and resources would need to be requisitioned for the task; every part of society would have to be informed and prepared… if it was contagious.
The difference between these two scenarios could not have been more extreme, and so the question of contagiousness took on monumental significance for the management of the outbreak – not to mention for the management of international relations, the stability of our societies, and the health of every human being on the planet. It was crucial that the world got the right answer, and sooner rather than later. The responsibility for answering that question, and the world’s attention, fell to the WHO.
To understand how the WHO approached their task, we will analyse the problem from a scientific perspective. That is, through the lens of the scientific hypothesis test.
The Scientist’s Perspective
When SARS-CoV-2 first appeared in January 2020, scientists knew little about its epidemiological characteristics. It was a novel virus – how could they? Lacking this key information however, they were in no position to advise the world on appropriate responses.
Adding to the scientists’ difficulties was the precariousness of the situation, the significance of any conclusion, and the implications for the world and everyone in it. For any scientist facing that dilemma, grounded in the philosophy of hypothesis testing, the appropriate course of action was clear: wait for the data to accumulate and let it make the decision for you. Of equal importance: do nothing until then.
The question was too consequential for the analysis to be rushed. It would be foolish to panic the world over what could turn out to be a minor outbreak, and not least because it would elicit global opprobrium for the WHO. Why take the risk? Getting the right answer was key, so the WHO would take their time to ensure that they did. As per the hypothesis test, they would wait until the evidence was pointing conclusively in the direction of contagiousness, and only then would they inform the world. Until then, the null hypothesis of no contagiousness would remain their default position.
And so, despite many early warnings from Wuhan and abroad, the WHO continued to urge caution into the second and third weeks of January 2020 while they waited for conclusive evidence to emerge. It was not until the fourth week of the outbreak that they relented and the world was officially notified that the early warnings were in fact correct: this novel coronavirus was transmitting from human to human, the world was facing a truly global pandemic, and everyone was at risk.
To the risk managers watching – gasping in disbelief – it had seemed an act of almost deliberate self-sabotage.
The Risk Manager’s Perspective
Given that this was a novel pathogen, we knew little about it, which meant that every possibility remained on the table, because if you don’t know what it is, then you can’t say what it isn’t, can you?
Since we didn’t know how virulent it was, it could have been very virulent. We didn’t know how deadly it was, so it could have been very deadly. We didn’t know how effective our medicines and therapeutics would be, so it was possible that they wouldn’t work at all. We didn’t know whether it transmitted from human to human, so it was possible that it did – and very well too.
We knew almost nothing about the virus at that time, so it was possible that it could have been anything, including all of the above and worse again.
Or, it could have been relatively benign. Again, we had no way of knowing either way.
But at least we knew that we didn’t know, and that was something (‘known unknowns’). We knew that high virulence and contagiousness were possibilities – we knew that they were risks – so we knew we needed a plan for them. By the principles of risk management: if the risk exists, then you must manage it; if it is possible, then you must prepare for it. This is how risk managers keep themselves (and the rest of us) alive.
So, as things stood on the 1st of January 2020 (the 1st, not the 22nd) a risk manager would have determined that this virus could have been highly-infectious, it could have been highly-virulent, and it could have been very efficient at transmitting from human to human. These risks would have been clear, and therefore the need for an immediate risk management response would have been too.
From the very first day of this outbreak, the risk manager would have seen the need for urgent action. Instead, three weeks later, the scientists at the WHO were still sitting on their hands waiting for the data to make the decision for them.
Science Fails Again
In science, the null hypothesis is of no effect: there no difference between the variables in question, there is no relationship between them, nothing going on. The base case assumption is always the ignorance of the status quo, and we need overwhelming evidence to convince ourselves otherwise. In the case of SARS-CoV-2 in January 2020, the WHO’s base case assumption was that there was no contagiousness, and it took until the 4th week of January before it became impossible for them to continue to deny the awful reality of the situation.
This may have been the ‘scientific’ thing to do, but the delay was catastrophic for the world. By the time we had discovered the pathogen, it had already been given a 10+ week head start on us, and the WHO’s unnecessary deliberations cost us at least another couple of weeks. Given that the outbreak could have been tripling in size every week at that time (and some estimates put that growth rate at 4x) this was equivalent to allowing the outbreak to increase by a factor of 10 or more while they waited. It’s hard enough playing catch up with a PPP at the best of times, but it’s near impossible when the WHO take so long to get out of first gear.
Had the risk managers been in charge, things would have been different.
The risk managers would have responded to the threat much quicker than the scientists because they only need confirmation of the risk to act, not overwhelming evidence that the risk has materialised. The mere existence a novel pathogen circulating in Wuhan was all the risk managers needed to know to justify an immediate response. Had the world applied their method of analysis to the question of contagiousness, the outbreak would have been 1/10th or even 1/30th of the size at the time, and a much easier problem to solve.
With the full benefit of hindsight, an immediate and forceful response on the 1st of January was the optimal course of action. As I have argued above, the risk managers would have provided it. It is a matter of fact that the scientists could not. The key takeaway is that risk management can be orders of magnitude more effective than science at guiding decision-making in situations like these.
Type 1 vs Type 2 Errors
The scientists and the risk managers are effectively testing the same hypotheses against the same datasets, but when we move from the scientific method to risk management, the null and the alternative hypotheses swap. For the scientist, the null is inaction and the alternative is action. For risk managers it is the reverse. The scientist needed proof that it was contagious to act, whereas the risk manager needed proof that it wasn’t contagious to stop acting.
In Science, it is worse to believe something that isn’t true (Type 1 error) than it is to not believe something that is true (Type 2 error) as anything built on falsity will eventually collapse, while the truth may yet be discovered another day. The same logic applies to criminal trials with the defendant’s presumption of innocence until proven guilty (in liberal societies at least).
In a pandemic however, the asymmetry in the errors flips. When the thing in question is a threat (a pathogen, wild animal, or storm) then the greater risk is in the Type 2 error: doing too little too late, not too much too soon. If it is possible that something very bad could happen, then you must prepare for that scenario, even if it is only a possibility. Over-protect yourself and you have wasted some resources; under-protect yourself and you die.
We know this already, which is why we put our seatbelts on before we start the car, install fire-doors during the building’s construction, and save funds for a rainy day. We don’t know whether the bad thing will happen, but we protect ourselves in advance, just in case. These are all forms of risk management, and they are so common and so deeply engrained in our culture that we hardly think about them anymore.
And yet the ‘scientific’ approach to epidemic control ignores this logic. The scientific method can’t deal with contingencies – that’s not what it’s designed to do. Science’s purpose is to know what can know for sure, but in a crisis, this need for certainty can yield the worst possible outcome: overlooking risks until it is too late to do anything about them.
Conclusion
The WHO’s mistakes in the early stages of the outbreak perfectly demonstrate both the failure the scientific method in a crisis, and the superiority of risk management (aka decision making under uncertainty) in volatile, uncertain, and dangerous environments. It also shows us what could have been, and how we could do better next time.
The WHO’s reluctant admission of human-to-human transmission on the 22nd of January was met with a flurry of border control policies, all over the world. This demonstrates the significance of the announcement and it also shows us that the nations of the world were paying attention to it. Had the risk managers been in charge, they would have recognised the problem sooner, the world would have responded sooner and more forcefully too, and health outcomes would have been better for everyone. Instead, the WHO tried to do science and… well, the rest is history.
Sadly, this faulty approach is so pervasive among the scientists in the WHO, that they made the same mistake when delaying the declaration of a PHEIC in January 2020, and again in the official declaration of a pandemic in March 2020. Too slow, too late, again and again. We simply cannot solve the problem of pandemics with this thinking. On the contrary, it will sleep walk us into one catastrophe after another.
If we are to protect humanity from PPPs – whether naturally occurring, lab leaks, engineered, bioweapons, bioterrorism, or any other – a wholesale change in philosophy will be required. We must let go of our implicit assumption that science is omnipotent and scientists infallible, or that the solution to pandemics can be found in a novel compound or a new piece of technology. Instead, we must recognise and embrace the inherent uncertainty of these problems, and with it, the philosophy of risk and the practice of risk management.