I think John Hopkins’ advice has generally been well respected but that is just rumour on the grapevine, I cannot say exactly why I think that.
Nuclear Threat Initiative’s Global Health Security Index said “international preparedness for epidemics and pandemics remains very weak” which seems correct. But the index also put the UK and the USA as the most prepared which seems incorrect (or at least gives a reason not to trust very shallow global indices). Eg yes the UK had a plan, tick box – but turns out it was not very good. Yes the UK did emergency exercises, tick box – but turns out it did not update plans based on the exercises. Etc
Not on your list but in this interview from Feb 2020, 80,000 Hours and the Future of Humanity Institute get a lot correct (eg need for social distancing) but they do both seem to disagree with the case for most travel bans, which seems incorrect in hindsight. See the intro where this is discussed.
Also not a bio org but EA Funded Our World in Data has done a good job on COVID data gathering and presentation.
One feature of the things that me and William have picked up on is that early on (say in Feb 2020 or earlier) advice coming from very respectable organisations was relatively poor.
I don’t think this should be seen as evidence that these organisations did badly (maybe a bit that they were over-confident) but that this was a very difficult situation to do things well in.
I don’t think this should be seen as evidence that these organisations did badly (maybe a bit that they were over-confident) but that this was a very difficult situation to do things well in.
I somewhat agree, but I think this point becomes much weaker if it was the case that at the same time when these organizations were giving poor advice some amateurs in the EA and rationality communities had already arrived at better conclusions, would have given better advice, etc.
I didn’t follow the relevant conversations closely enough to have much of an inside view on how strongly the latter is true, but my impression is that many people in the EA/rationality communities (including ones who did follow the conversations more closely) think it’s true. Even I am aware of some data points that seem to suggest such a conclusion (e.g., some conversations I remember).
FWIW, at least given what I know I think I find this less compelling as a ‘vindication of EA/rationalist epistemics’ or whatever than some other people seem to—I think the lessons we should learn depend on a number of additional things which I’m currently uncertain about:
Are we comparing like with like rather than cherrypicking? (I.e., comparing anecdotes of flawed advice from ‘expert organizations’ to anecdotes of ex-post correct advice from EAs/rationalists?)
Ideally I’d want to know about the full distribution of views among ‘expert organizations’ and the full distribution of views among ‘amateur EAs/rationalists who spent time looking into COVID things’, and then compare those.
What about some other relevant groups? E.g., how “well” did pharma/vaccine companies do? What about academics developing tests? It does seem like at least some of these groups started taking the novel coronavirus pretty seriously in January already.
What should we think about the ‘external validity’ of EA/rationalist successes? Some people did well in predicting how COVID would play out, what recommendations are good, etc., in the environmental condition of “having significant spare time and being able to post and discuss views with like-minded people without facing a lot of other constraints”. Would they still have done well if they had been in the environmental condition “expert making a public statement that needs to make sense to the general public” plus whatever other incentives these experts & organizations were subject to?
(The above wording might sound like I’m thinking it was “easier” for EAs/rationalists to arrive at correct conclusions or give good advice when making statements we would hear about. However, in fact, I’m unsure about the “net effect” of incentives and other environmental conditions. Similarly, I don’t mean to suggest that ‘external validity’ is necessarily poor, just that it seems worth thinking about before drawing strong conclusions.)
However, no matter the answers to these questions, your claim to me still sounds too generous to these organizations.
Hi, Yes good point, maybe I am being too generous.
FWIW I don’t remember anyone in the EA / rationalist community calling for the strategy that post-hoc seems to have worked best of a long lock-down to get to zero cases followed by border closures etc to keep cases at zero. (I remember a lot of people for example sharing this note which gets much right but stuff wrong: eg short lock-dock and comparatively easy to keep R below 1 with social distancing)
long lock-down to get to zero cases followed by border closures etc to keep cases at zero.
Can you say more about the strategy you have in mind and give examples of the countries that implemented it? It’s hard to judge whether EAs or rationalists advocated it from your characterization, which seems to build into it the relevant success conditions. Lots of countries had long lockdowns and strict border closures, but failed miserably at bringing cases down to zero or anything remotely approaching zero. Furthermore, whether this approach was best all-things-considered should obviously also take into account the enormous costs associated with these measures (in all of the countries that implemented long lockdowns and strict border closures).
weeatquince’s is sharing a widely held view, i.e. that eradication is superior to containment in health and economic outcomes, see e.g. this analysis. The idea itself is plausible, since a successful lockdown allows complete reopening of the internal economy afterwards.
Sample size is however small, especially when it comes to non-island countries. I only know of two non-island countries that seriously went for eradication coupled with border closures, namely Vietnam and Israel. Israel gave up at one point when cases started to rise (which is why it is listed among the containment countries in the analysis above) , but Vietnam succeeded (although it had to restrict travel heavily domestically as well). Personally, I believe it is a suboptimal strategy for non- authoritarian, non-island countries.
I think their original point stands though, that EA/rationalists did not seem to entertain the idea of eradication enough, but probably neither did biorisk organizations last year.
I feel that this discussion is not framed correctly. Yes, successful eradication is superior to containment in health and economic outcomes. This is a pretty weak claim that lots of people can agree with who otherwise differ considerably in their policy proposals. But the original claim was that EAs and rationalists hadn’t advocated for long lockdowns and border closures, and that this was relevant for retroactively assessing their performance. The plausibility of the latter claim must be evaluated by considering all the countries that implemented long lockdowns and border closures, and not just the tiny minority that were successful in attaining (near-)eradication by adopting those measures.
I took a quick look at the study you shared. Their analysis compared covid deaths, GDP growth and lockdown stringency in two groups of OECD countries during the first twelve months of the pandemic, and offered this as their original contribution to the study’s main thesis that countries which favored elimination had better health and economic outcomes than countries which favored mitigation (the rest of the study is a brief and unsystematic summary of some of the relevant literature). It turns out that the group which supposedly favored elimination consists of just five nations, four of which are islands and the fifth of which (South Korea) shares borders with a single county which has been completely isolated from the rest of the world for decades.
Let’s pause for a moment and consider how quickly this kind of evidence would have been dismissed if it had been presented in support of a politically inconvenient conclusion. Yet here it is offered, in the world’s most prestigious medical journal, to establish that “elimination, not mitigation, creates best outcomes for health, the economy, and civil liberties”.
For what it’s worth, I personally have no strong views on how the pandemic should have been handled. (My only strong meta-view is that decisions should have been based on explicit cost-effectiveness analyses, which were surprisingly absent from most policy discussions.) My impression is that EAs and rationalists—to which I would add the Metaculus community—did somewhat better than most experts, but this assessment is based mostly on comparing their performance on simple factual questions or issues involving basic sanity. Here, instead, we are told to downgrade our estimation of how well EAs and rationalists performed because, apparently, they did not advocate for the entire world a policy that was successful in only a handful of geographically highly isolated countries. I don’t find that plausible.
I don’t think I can help much with answering these questions.
I was thinking of counties like Australia and New Zealand and Taiwan. But whether or not the strategies adopted in these places was actually optimal or best with the available information or applicable to most countries that are not islands or had a high chance of failure – I cannot say!
All I can say is that there is at least one plausible strategy that seems to have worked well in at least some countries and I personally don’t really remember it being discussed within the EA space a year ago.
Feel free to draw what conclusions or analysis you will from that.
the enormous costs
Just to add, I expect (but I might be wrong) that these countries have had lower welfare and economic costs than most other places.
they do both seem to disagree with the case for most travel bans, which seems incorrect in hindsight
Can you say more about this, or point me to good resources off the top of your head? I didn’t follow the conversation on travel bans, and currently don’t have a good sense of what people think about them.
In particular, is the rough basis for “incorrect in hindsight” the impression that travel bans were a potentially essential part of the successful containment strategy of some countries such as New Zealand? Or are there other major considerations as well?
My understanding is that travel bans were widely believed to have greater costs than benefits before COVID. There are various quotes along those lines described in the (rather cynical) Lessons From the Crisis post on the topic of border closures.
In February 2020, I believed border closures weren’t worth it. I thought they disincentivised countries from being transparent about emerging outbreaks (because said countries would face economic punishment via closed borders) and could only slow down the spread of a disease, not stop it. While I’m still not entirely sure about the relative benefits of open reporting vs. slowed spread, I was definitely underestimating the benefits of the latter. Evidence from Vietnam and New Zealand shows that early and strong international border controls can indeed slow the spread to the point where local outbreaks don’t spiral beyond easy containment.
There was insufficient evidence to draw firm conclusions about the effectiveness of travel‐related quarantine on its own. Some of the included studies suggest that effects are likely to depend on factors such as the stage of the epidemic, the interconnectedness of countries, local measures undertaken to contain community transmission, and the extent of implementation and adherence.
The study finds that the domestic travel measures implemented in Wuhan were effective at reducing the importation of cases internationally and within China and that additional travel restrictions were also likely important. The study also finds that travel measures are more effective when implemented earlier in the outbreak.
A few opinions:
I think John Hopkins’ advice has generally been well respected but that is just rumour on the grapevine, I cannot say exactly why I think that.
Nuclear Threat Initiative’s Global Health Security Index said “international preparedness for epidemics and pandemics remains very weak” which seems correct. But the index also put the UK and the USA as the most prepared which seems incorrect (or at least gives a reason not to trust very shallow global indices). Eg yes the UK had a plan, tick box – but turns out it was not very good. Yes the UK did emergency exercises, tick box – but turns out it did not update plans based on the exercises. Etc
Not on your list but in this interview from Feb 2020, 80,000 Hours and the Future of Humanity Institute get a lot correct (eg need for social distancing) but they do both seem to disagree with the case for most travel bans, which seems incorrect in hindsight. See the intro where this is discussed.
Also not a bio org but EA Funded Our World in Data has done a good job on COVID data gathering and presentation.
One feature of the things that me and William have picked up on is that early on (say in Feb 2020 or earlier) advice coming from very respectable organisations was relatively poor.
I don’t think this should be seen as evidence that these organisations did badly (maybe a bit that they were over-confident) but that this was a very difficult situation to do things well in.
I somewhat agree, but I think this point becomes much weaker if it was the case that at the same time when these organizations were giving poor advice some amateurs in the EA and rationality communities had already arrived at better conclusions, would have given better advice, etc.
I didn’t follow the relevant conversations closely enough to have much of an inside view on how strongly the latter is true, but my impression is that many people in the EA/rationality communities (including ones who did follow the conversations more closely) think it’s true. Even I am aware of some data points that seem to suggest such a conclusion (e.g., some conversations I remember).
FWIW, at least given what I know I think I find this less compelling as a ‘vindication of EA/rationalist epistemics’ or whatever than some other people seem to—I think the lessons we should learn depend on a number of additional things which I’m currently uncertain about:
Are we comparing like with like rather than cherrypicking? (I.e., comparing anecdotes of flawed advice from ‘expert organizations’ to anecdotes of ex-post correct advice from EAs/rationalists?)
Ideally I’d want to know about the full distribution of views among ‘expert organizations’ and the full distribution of views among ‘amateur EAs/rationalists who spent time looking into COVID things’, and then compare those.
What about some other relevant groups? E.g., how “well” did pharma/vaccine companies do? What about academics developing tests? It does seem like at least some of these groups started taking the novel coronavirus pretty seriously in January already.
What should we think about the ‘external validity’ of EA/rationalist successes? Some people did well in predicting how COVID would play out, what recommendations are good, etc., in the environmental condition of “having significant spare time and being able to post and discuss views with like-minded people without facing a lot of other constraints”. Would they still have done well if they had been in the environmental condition “expert making a public statement that needs to make sense to the general public” plus whatever other incentives these experts & organizations were subject to?
(The above wording might sound like I’m thinking it was “easier” for EAs/rationalists to arrive at correct conclusions or give good advice when making statements we would hear about. However, in fact, I’m unsure about the “net effect” of incentives and other environmental conditions. Similarly, I don’t mean to suggest that ‘external validity’ is necessarily poor, just that it seems worth thinking about before drawing strong conclusions.)
However, no matter the answers to these questions, your claim to me still sounds too generous to these organizations.
Hi, Yes good point, maybe I am being too generous.
FWIW I don’t remember anyone in the EA / rationalist community calling for the strategy that post-hoc seems to have worked best of a long lock-down to get to zero cases followed by border closures etc to keep cases at zero. (I remember a lot of people for example sharing this note which gets much right but stuff wrong: eg short lock-dock and comparatively easy to keep R below 1 with social distancing)
Can you say more about the strategy you have in mind and give examples of the countries that implemented it? It’s hard to judge whether EAs or rationalists advocated it from your characterization, which seems to build into it the relevant success conditions. Lots of countries had long lockdowns and strict border closures, but failed miserably at bringing cases down to zero or anything remotely approaching zero. Furthermore, whether this approach was best all-things-considered should obviously also take into account the enormous costs associated with these measures (in all of the countries that implemented long lockdowns and strict border closures).
weeatquince’s is sharing a widely held view, i.e. that eradication is superior to containment in health and economic outcomes, see e.g. this analysis. The idea itself is plausible, since a successful lockdown allows complete reopening of the internal economy afterwards.
Sample size is however small, especially when it comes to non-island countries. I only know of two non-island countries that seriously went for eradication coupled with border closures, namely Vietnam and Israel. Israel gave up at one point when cases started to rise (which is why it is listed among the containment countries in the analysis above) , but Vietnam succeeded (although it had to restrict travel heavily domestically as well). Personally, I believe it is a suboptimal strategy for non- authoritarian, non-island countries.
I think their original point stands though, that EA/rationalists did not seem to entertain the idea of eradication enough, but probably neither did biorisk organizations last year.
Thanks for the clarification.
I feel that this discussion is not framed correctly. Yes, successful eradication is superior to containment in health and economic outcomes. This is a pretty weak claim that lots of people can agree with who otherwise differ considerably in their policy proposals. But the original claim was that EAs and rationalists hadn’t advocated for long lockdowns and border closures, and that this was relevant for retroactively assessing their performance. The plausibility of the latter claim must be evaluated by considering all the countries that implemented long lockdowns and border closures, and not just the tiny minority that were successful in attaining (near-)eradication by adopting those measures.
I took a quick look at the study you shared. Their analysis compared covid deaths, GDP growth and lockdown stringency in two groups of OECD countries during the first twelve months of the pandemic, and offered this as their original contribution to the study’s main thesis that countries which favored elimination had better health and economic outcomes than countries which favored mitigation (the rest of the study is a brief and unsystematic summary of some of the relevant literature). It turns out that the group which supposedly favored elimination consists of just five nations, four of which are islands and the fifth of which (South Korea) shares borders with a single county which has been completely isolated from the rest of the world for decades.
Let’s pause for a moment and consider how quickly this kind of evidence would have been dismissed if it had been presented in support of a politically inconvenient conclusion. Yet here it is offered, in the world’s most prestigious medical journal, to establish that “elimination, not mitigation, creates best outcomes for health, the economy, and civil liberties”.
For what it’s worth, I personally have no strong views on how the pandemic should have been handled. (My only strong meta-view is that decisions should have been based on explicit cost-effectiveness analyses, which were surprisingly absent from most policy discussions.) My impression is that EAs and rationalists—to which I would add the Metaculus community—did somewhat better than most experts, but this assessment is based mostly on comparing their performance on simple factual questions or issues involving basic sanity. Here, instead, we are told to downgrade our estimation of how well EAs and rationalists performed because, apparently, they did not advocate for the entire world a policy that was successful in only a handful of geographically highly isolated countries. I don’t find that plausible.
I don’t think I can help much with answering these questions.
I was thinking of counties like Australia and New Zealand and Taiwan. But whether or not the strategies adopted in these places was actually optimal or best with the available information or applicable to most countries that are not islands or had a high chance of failure – I cannot say!
All I can say is that there is at least one plausible strategy that seems to have worked well in at least some countries and I personally don’t really remember it being discussed within the EA space a year ago.
Feel free to draw what conclusions or analysis you will from that.
Just to add, I expect (but I might be wrong) that these countries have had lower welfare and economic costs than most other places.
I discuss the GHS index at greater length in my answer.
Thank you :-)
Can you say more about this, or point me to good resources off the top of your head? I didn’t follow the conversation on travel bans, and currently don’t have a good sense of what people think about them.
In particular, is the rough basis for “incorrect in hindsight” the impression that travel bans were a potentially essential part of the successful containment strategy of some countries such as New Zealand? Or are there other major considerations as well?
My understanding is that travel bans were widely believed to have greater costs than benefits before COVID. There are various quotes along those lines described in the (rather cynical) Lessons From the Crisis post on the topic of border closures.
In February 2020, I believed border closures weren’t worth it. I thought they disincentivised countries from being transparent about emerging outbreaks (because said countries would face economic punishment via closed borders) and could only slow down the spread of a disease, not stop it. While I’m still not entirely sure about the relative benefits of open reporting vs. slowed spread, I was definitely underestimating the benefits of the latter. Evidence from Vietnam and New Zealand shows that early and strong international border controls can indeed slow the spread to the point where local outbreaks don’t spiral beyond easy containment.
To link you to some resources, a September 2020 Cochrane meta-analysis, Travel‐related control measures to contain the COVID‐19 pandemic: a rapid review, concluded that:
The February 2021 paper Evidence of the effectiveness of travel-related measures during the early phase of the COVID-19 pandemic: a rapid systematic review (PDF link) (which I found in the recent Vox article Vietnam defied the experts and sealed its border to keep Covid-19 out. It worked.) concluded that:
Thank you, very helpful!