The Global Health Security Index looks like a misfire. This isn’t directly about performance during the pandemic, but Nuclear Threat Initiative, funded by Open Phil for this purpose (h/t HowieL for pointing this out) and collaborating with the Johns Hopkins Center for Health Security, made the 2019 Global Health Security Index which seems invalidated by COVID-19 outcomes and may have encouraged actors to take the wrong moves. This ThinkGlobalHealth article describes how its ratings did not predict good performance against the virus. The article relies on official death counts rather than excess mortality, but I made that correction and reached similar results.
Looking through the index, there are some indicators which don’t make sense, like praising countries for avoiding travel restrictions (which is perverse), praising them for having more ethical regulations against surveillance and clinical trials (which may be ethically justified but is more likely to make it harder to fight a pandemic), and praising them for gender equality (a noble sentiment but not directly relevant to pandemics).
Some people have also criticized the index for rating China moderately highly on prevention of pathogen release, considering that COVID-19 came from China, but considering that COVID-19 is just one data point of virus emergence or lab leak and that China is a very large country I don’t think this is right.
FYI—my impression is that pandemic preparedness is often defined broadly enough to include things like research into defensive technology (e.g. mRNA vaccines). It does seem like those investments were important for the response.
Hm, certainly the vaccine rollout was in hindsight the second most important thing after success or failure at initial lockdown and containment.
It does seem to have been neglected by preparation efforts and EA funding before the pandemic, but that’s understandable considering how much of a surprise this mRNA stuff was.
I think research into novel vaccine platforms like mRNA is a top priority. It’s neglected in the sense that way more resources should be going into it but also my impression[1] is that the USG does make up a decent proportion of funding for early stage research into that kind of thing. So that’s a sense in which the U.S.’s preparedness was prob good relative to other countries though not in an absolute sense.
In September 2017, CEPI requested proposals for vaccine platform technologies that enable rapid vaccine development, elicit rapid onset of immunity, and whose production can be scaled-up quickly to respond to outbreaks of Disease X. CEPI funded three platform technologies: a vaccine printer, molecular clamp platform, and a self-amplifying RNA vaccine platform.
I think it was a surprise that non-self-amplifying mRNA vaccines work as well as they do (mRNA is more immunogenic than predicted, I guess, at least for COVID?). 18 months ago, I don’t think I would have bet on mRNA platform vaccines as the future over DNA or adenovirus vaccines.
Hello, Thank you for the interesting thoughts. The comments on the GHS index are useful and insightful.
Your analysis of COVID preparation on Twitter is really really interesting. Well done for doing that. I have not yet looked at your analysis spreadsheet but will try to do that soon.
To touch on a point you said about preparation, I think we can take a bit more of a nuanced approach to think about when preparation works rather than just saying “effective pandemic response is not about preparation”. Some thoughts from me on this (not just focused on pandemics).
Prevention definitely helps. (It is a semantic question if you want to count prevention as a type of preparation or not). The world is awash with very clear examples of disaster prevention whether it is engineering safe bridges, or flood prevention, or nuclear safety, or preventing pathogens escaping labs, etc.
The idea that preparation (henceforth excluding prevention) helps is conventional wisdom and I think I would want to see good evidence against this to stop believing in this.
Obviously preparation helps in the small cases, talk to a paramedic rushing to treat someone or a fireman. I have not looked into it but I get the impression that it helps in the medium cases, eg rapid response teams responding to terror attacks in the UK / France seem useful, although not an expert. On pandemics specifically the quick containment of SARs seems to be a success story (although I have not looked at how much preparation played a role it does seem to be a part of the story). There are not that many extreme COVID-level cases to look at, but it would be odd if it didn’t help in extreme cases too.
The specific wording of the claim in the linked article headline feels clickbait-y. When you actually read the article it actually says that competence matters more (I agree) and also that we should focus more on designing resilient anti-fragile systems rather than event specific preparation. I agree but I think that designing systems that can make good decisions in a risk scenario is a form of preparation.
I do agree that your analysis provides some evidence that preparation did not help with COVID. I am cautious of the usefulness of this evidence because of the problems with the GHS – e.g. the UK came near top but basically had no plan to deal with any non-influenza pandemic that I have identified.
A confusing factor that might make it hard to tell if preparation helped is that, based on the UK experience (eg discussed here) it appears that having bad plans in place may actually be worse than no plans.
Evidence from COVID does suggest to me that specific preparation does help. Notably countries (E Asia, Australasia) that had SARs and prepared for future SARs type outbreaks managed COIVD better.
So maybe we can say something like: Prevention definitely helps. Both event specific preparation and generally building robust anti-fragile decision systems are useful approaches but the latter of those is more underinvested in. However good leadership is necessary as well as preparation and without good leadership (which maybe rare) preparation can turn out to be useless. Furthermore bad preparation, such as poor planning, can potentially hinder a response more than no preparation.
Does that seem like a good summary and sufficiently explain your findings.
I am thinking about doing more work to promote preparation so useful to hear if you disagree.
Prevention definitely helps. (It is a semantic question if you want to count prevention as a type of preparation or not)
I don’t think most people would consider prevention a type of preparation. EA-funded biorisk efforts presumably did not consider it that way. And more to the point, I do not want to lump prevention together with preparation because I am making an argument about preparation that is separate from prevention. So it’s not about just semantics, but precision on which efforts did well or poorly.
The idea that preparation (henceforth excluding prevention) helps is conventional wisdom and I think I would want to see good evidence against this to stop believing in this.
Conventional wisdom is worth little when it is the product of armchair speculation rather than experience. If people live through half a dozen pandemics and still have that conventional wisdom then we can have a different conversation.
On pandemics specifically the quick containment of SARs seems to be a success story (although I have not looked at how much preparation played a role it does seem to be a part of the story)
Wouldn’t preparation seem to be a part of the story of COVID-19 outcomes given a similarly superficial level of inquiry?
I agree but I think that designing systems that can make good decisions in a risk scenario is a form of preparation
Forget semantics. Did EA funding efforts and recipients design systems that made good decisions about COVID-19? Did anyone who talked about “pandemic preparation” pre-2020 use the term to encompass the design of systems like that?
A confusing factor that might make it hard to tell if preparation helped is that, based on the UK experience (eg discussed here) it appears that having bad plans in place may actually be worse than no plans.
Well you can’t just define preparation as “good plans”, that’s a no-true-Scotsman argument. If you have some way of ensuring that your preparation will be good preparation then it’s a different story.
Evidence from COVID does suggest to me that specific preparation does help. Notably countries (E Asia, Australasia) that had SARs and prepared for future SARs type outbreaks managed COIVD better.
That isn’t necessarily due to physical preparation, it could easily be intangible changes in the culture and political system, granting that there is in fact a causal connection as opposed to East Asia and Australasia just being better at this stuff.
iirc there was a study which found that American cities that lived through the Spanish Flu (1919) suffered less death early in the COVID19 outbreak. Cannot find the study now but if it’s really true then that would be hard to explain through preparation.
Does that seem like a good summary and sufficiently explain your findings.
I’m not sure exactly what anti-fragile means but that doesn’t sound right, decision systems in the US/UK for instance didn’t fall apart, they were just apathetic and unresponsive to good ideas just like they are for mundane problems that aren’t big crises. In other words they calmly kept operating the way they always do.
I don’t have reason to believe that there is a positive interaction between good leadership and good preparation. Maybe good preparation and good leadership act more as substitutes for each other rather than compliments.
Not sure it is useful to say ‘prevention helps’ since we cannot wish away viruses, we can only take measures to attempt to prevent viruses from emerging, and while those measures may be cost-effective it is a different conversation to which I have nothing to contribute.
I would summarize my view by saying that smart actions by government and civil society in the moment make the most difference, and if plans and preparation are to be helpful they will have to be done in careful ways to avoid the failures documented during COVID-19.
I don’t think most people would consider prevention a type of preparation. EA-funded biorisk efforts presumably did not consider it that way. And more to the point, I do not want to lump prevention together with preparation because I am making an argument about preparation that is separate from prevention. So it’s not about just semantics, but precision on which efforts did well or poorly.
I think it actually is common to include prevention under the umbrella of pandemic preparedness. for example, here’s the Council on Foreign Relation’s independent committee on Improving Pandemic Preparedness: “Based on the painful lessons of the current pandemic, the Task Force makes recommendations for improving U.S. and global capacities to deliver each of the three fundamentals of pandemic preparedness: prevention, detection, and response. ” Another example: https://www.path.org/articles/building-epidemic-preparedness-worldwide/
So it might be helpful to specify what you’re referring to by preparation.
It seems fair to call avoiding travel restrictions a dubious measure in hindsight, but circa 2019 it strikes me as a reasonable metric to put under “compliance with international norms”. There was an expert consensus that travel norms weren’t a good pandemic response tool (see my other comment) and not implementing them is indeed part of complying with the WHO IHRs.
I am not totally sure that compliance with international norms a good measure of national health security! However, the according to the Think Global Health article you linked on Twitter, even the WHO Joint External Evaluations weren’t well-correlated with COVID-19 deaths. (Those evaluations are how the prevention / detection / response capacity are measured in the Global Health Security Index, which then adds measures on health system / compliance with norms / risk landscape.)
The Global Health Security Index looks like a misfire. This isn’t directly about performance during the pandemic, but Nuclear Threat Initiative, funded by Open Phil for this purpose (h/t HowieL for pointing this out) and collaborating with the Johns Hopkins Center for Health Security, made the 2019 Global Health Security Index which seems invalidated by COVID-19 outcomes and may have encouraged actors to take the wrong moves. This ThinkGlobalHealth article describes how its ratings did not predict good performance against the virus. The article relies on official death counts rather than excess mortality, but I made that correction and reached similar results.
Looking through the index, there are some indicators which don’t make sense, like praising countries for avoiding travel restrictions (which is perverse), praising them for having more ethical regulations against surveillance and clinical trials (which may be ethically justified but is more likely to make it harder to fight a pandemic), and praising them for gender equality (a noble sentiment but not directly relevant to pandemics).
Even cutting some of those dubious measures out, I found the index was not predictive of excess mortality. In general it appears that effective pandemic response is not about preparation and this may have been systematically overlooked by EA efforts and funding recipients in the realm of biorisk.
Some people have also criticized the index for rating China moderately highly on prevention of pathogen release, considering that COVID-19 came from China, but considering that COVID-19 is just one data point of virus emergence or lab leak and that China is a very large country I don’t think this is right.
“effective pandemic response is not about preparation”
FYI—my impression is that pandemic preparedness is often defined broadly enough to include things like research into defensive technology (e.g. mRNA vaccines). It does seem like those investments were important for the response.
Hm, certainly the vaccine rollout was in hindsight the second most important thing after success or failure at initial lockdown and containment.
It does seem to have been neglected by preparation efforts and EA funding before the pandemic, but that’s understandable considering how much of a surprise this mRNA stuff was.
I think research into novel vaccine platforms like mRNA is a top priority. It’s neglected in the sense that way more resources should be going into it but also my impression[1] is that the USG does make up a decent proportion of funding for early stage research into that kind of thing. So that’s a sense in which the U.S.’s preparedness was prob good relative to other countries though not in an absolute sense.
Here’s an article I skimmed about the importance of govt (mostly NIH) funding for the development of mRNA vaccines. https://www.scientificamerican.com/article/for-billion-dollar-covid-vaccines-basic-government-funded-science-laid-the-groundwork/
Fwiw, I think it’s prob not the case that the mRNA stuff was that much of a surprise. This 2018 CHS report had self-amplifying mRNA vaccines as one of ~15 technologies to address GCBRs. https://jhsphcenterforhealthsecurity.s3.amazonaws.com/181009-gcbr-tech-report.pdf
[1] Though I’m rusty since I haven’t worked directly on biorisk for five years and was never an expert.
I second the impression that it’s not that much of a surprise. For example, CEPI was founded with a goal of accelerating vaccine development against the WHO R&D Blueprint priority diseases and according to their R&D webpage:
I think it was a surprise that non-self-amplifying mRNA vaccines work as well as they do (mRNA is more immunogenic than predicted, I guess, at least for COVID?). 18 months ago, I don’t think I would have bet on mRNA platform vaccines as the future over DNA or adenovirus vaccines.
Hello, Thank you for the interesting thoughts. The comments on the GHS index are useful and insightful.
Your analysis of COVID preparation on Twitter is really really interesting. Well done for doing that. I have not yet looked at your analysis spreadsheet but will try to do that soon.
To touch on a point you said about preparation, I think we can take a bit more of a nuanced approach to think about when preparation works rather than just saying “effective pandemic response is not about preparation”. Some thoughts from me on this (not just focused on pandemics).
Prevention definitely helps. (It is a semantic question if you want to count prevention as a type of preparation or not). The world is awash with very clear examples of disaster prevention whether it is engineering safe bridges, or flood prevention, or nuclear safety, or preventing pathogens escaping labs, etc.
The idea that preparation (henceforth excluding prevention) helps is conventional wisdom and I think I would want to see good evidence against this to stop believing in this.
Obviously preparation helps in the small cases, talk to a paramedic rushing to treat someone or a fireman. I have not looked into it but I get the impression that it helps in the medium cases, eg rapid response teams responding to terror attacks in the UK / France seem useful, although not an expert. On pandemics specifically the quick containment of SARs seems to be a success story (although I have not looked at how much preparation played a role it does seem to be a part of the story). There are not that many extreme COVID-level cases to look at, but it would be odd if it didn’t help in extreme cases too.
The specific wording of the claim in the linked article headline feels clickbait-y. When you actually read the article it actually says that competence matters more (I agree) and also that we should focus more on designing resilient anti-fragile systems rather than event specific preparation. I agree but I think that designing systems that can make good decisions in a risk scenario is a form of preparation.
I do agree that your analysis provides some evidence that preparation did not help with COVID. I am cautious of the usefulness of this evidence because of the problems with the GHS – e.g. the UK came near top but basically had no plan to deal with any non-influenza pandemic that I have identified.
A confusing factor that might make it hard to tell if preparation helped is that, based on the UK experience (eg discussed here) it appears that having bad plans in place may actually be worse than no plans.
Evidence from COVID does suggest to me that specific preparation does help. Notably countries (E Asia, Australasia) that had SARs and prepared for future SARs type outbreaks managed COIVD better.
So maybe we can say something like:
Prevention definitely helps. Both event specific preparation and generally building robust anti-fragile decision systems are useful approaches but the latter of those is more underinvested in. However good leadership is necessary as well as preparation and without good leadership (which maybe rare) preparation can turn out to be useless. Furthermore bad preparation, such as poor planning, can potentially hinder a response more than no preparation.
Does that seem like a good summary and sufficiently explain your findings.
I am thinking about doing more work to promote preparation so useful to hear if you disagree.
I don’t think most people would consider prevention a type of preparation. EA-funded biorisk efforts presumably did not consider it that way. And more to the point, I do not want to lump prevention together with preparation because I am making an argument about preparation that is separate from prevention. So it’s not about just semantics, but precision on which efforts did well or poorly.
Conventional wisdom is worth little when it is the product of armchair speculation rather than experience. If people live through half a dozen pandemics and still have that conventional wisdom then we can have a different conversation.
Wouldn’t preparation seem to be a part of the story of COVID-19 outcomes given a similarly superficial level of inquiry?
Forget semantics. Did EA funding efforts and recipients design systems that made good decisions about COVID-19? Did anyone who talked about “pandemic preparation” pre-2020 use the term to encompass the design of systems like that?
Well you can’t just define preparation as “good plans”, that’s a no-true-Scotsman argument. If you have some way of ensuring that your preparation will be good preparation then it’s a different story.
That isn’t necessarily due to physical preparation, it could easily be intangible changes in the culture and political system, granting that there is in fact a causal connection as opposed to East Asia and Australasia just being better at this stuff.
iirc there was a study which found that American cities that lived through the Spanish Flu (1919) suffered less death early in the COVID19 outbreak. Cannot find the study now but if it’s really true then that would be hard to explain through preparation.
I’m not sure exactly what anti-fragile means but that doesn’t sound right, decision systems in the US/UK for instance didn’t fall apart, they were just apathetic and unresponsive to good ideas just like they are for mundane problems that aren’t big crises. In other words they calmly kept operating the way they always do.
I don’t have reason to believe that there is a positive interaction between good leadership and good preparation. Maybe good preparation and good leadership act more as substitutes for each other rather than compliments.
Not sure it is useful to say ‘prevention helps’ since we cannot wish away viruses, we can only take measures to attempt to prevent viruses from emerging, and while those measures may be cost-effective it is a different conversation to which I have nothing to contribute.
I would summarize my view by saying that smart actions by government and civil society in the moment make the most difference, and if plans and preparation are to be helpful they will have to be done in careful ways to avoid the failures documented during COVID-19.
I think it actually is common to include prevention under the umbrella of pandemic preparedness. for example, here’s the Council on Foreign Relation’s independent committee on Improving Pandemic Preparedness: “Based on the painful lessons of the current pandemic, the Task Force makes recommendations for improving U.S. and global capacities to deliver each of the three fundamentals of pandemic preparedness: prevention, detection, and response. ” Another example: https://www.path.org/articles/building-epidemic-preparedness-worldwide/
So it might be helpful to specify what you’re referring to by preparation.
It seems fair to call avoiding travel restrictions a dubious measure in hindsight, but circa 2019 it strikes me as a reasonable metric to put under “compliance with international norms”. There was an expert consensus that travel norms weren’t a good pandemic response tool (see my other comment) and not implementing them is indeed part of complying with the WHO IHRs.
I am not totally sure that compliance with international norms a good measure of national health security! However, the according to the Think Global Health article you linked on Twitter, even the WHO Joint External Evaluations weren’t well-correlated with COVID-19 deaths. (Those evaluations are how the prevention / detection / response capacity are measured in the Global Health Security Index, which then adds measures on health system / compliance with norms / risk landscape.)