I don’t think this should be seen as evidence that these organisations did badly (maybe a bit that they were over-confident) but that this was a very difficult situation to do things well in.
I somewhat agree, but I think this point becomes much weaker if it was the case that at the same time when these organizations were giving poor advice some amateurs in the EA and rationality communities had already arrived at better conclusions, would have given better advice, etc.
I didn’t follow the relevant conversations closely enough to have much of an inside view on how strongly the latter is true, but my impression is that many people in the EA/rationality communities (including ones who did follow the conversations more closely) think it’s true. Even I am aware of some data points that seem to suggest such a conclusion (e.g., some conversations I remember).
FWIW, at least given what I know I think I find this less compelling as a ‘vindication of EA/rationalist epistemics’ or whatever than some other people seem to—I think the lessons we should learn depend on a number of additional things which I’m currently uncertain about:
Are we comparing like with like rather than cherrypicking? (I.e., comparing anecdotes of flawed advice from ‘expert organizations’ to anecdotes of ex-post correct advice from EAs/rationalists?)
Ideally I’d want to know about the full distribution of views among ‘expert organizations’ and the full distribution of views among ‘amateur EAs/rationalists who spent time looking into COVID things’, and then compare those.
What about some other relevant groups? E.g., how “well” did pharma/vaccine companies do? What about academics developing tests? It does seem like at least some of these groups started taking the novel coronavirus pretty seriously in January already.
What should we think about the ‘external validity’ of EA/rationalist successes? Some people did well in predicting how COVID would play out, what recommendations are good, etc., in the environmental condition of “having significant spare time and being able to post and discuss views with like-minded people without facing a lot of other constraints”. Would they still have done well if they had been in the environmental condition “expert making a public statement that needs to make sense to the general public” plus whatever other incentives these experts & organizations were subject to?
(The above wording might sound like I’m thinking it was “easier” for EAs/rationalists to arrive at correct conclusions or give good advice when making statements we would hear about. However, in fact, I’m unsure about the “net effect” of incentives and other environmental conditions. Similarly, I don’t mean to suggest that ‘external validity’ is necessarily poor, just that it seems worth thinking about before drawing strong conclusions.)
However, no matter the answers to these questions, your claim to me still sounds too generous to these organizations.
Hi, Yes good point, maybe I am being too generous.
FWIW I don’t remember anyone in the EA / rationalist community calling for the strategy that post-hoc seems to have worked best of a long lock-down to get to zero cases followed by border closures etc to keep cases at zero. (I remember a lot of people for example sharing this note which gets much right but stuff wrong: eg short lock-dock and comparatively easy to keep R below 1 with social distancing)
long lock-down to get to zero cases followed by border closures etc to keep cases at zero.
Can you say more about the strategy you have in mind and give examples of the countries that implemented it? It’s hard to judge whether EAs or rationalists advocated it from your characterization, which seems to build into it the relevant success conditions. Lots of countries had long lockdowns and strict border closures, but failed miserably at bringing cases down to zero or anything remotely approaching zero. Furthermore, whether this approach was best all-things-considered should obviously also take into account the enormous costs associated with these measures (in all of the countries that implemented long lockdowns and strict border closures).
weeatquince’s is sharing a widely held view, i.e. that eradication is superior to containment in health and economic outcomes, see e.g. this analysis. The idea itself is plausible, since a successful lockdown allows complete reopening of the internal economy afterwards.
Sample size is however small, especially when it comes to non-island countries. I only know of two non-island countries that seriously went for eradication coupled with border closures, namely Vietnam and Israel. Israel gave up at one point when cases started to rise (which is why it is listed among the containment countries in the analysis above) , but Vietnam succeeded (although it had to restrict travel heavily domestically as well). Personally, I believe it is a suboptimal strategy for non- authoritarian, non-island countries.
I think their original point stands though, that EA/rationalists did not seem to entertain the idea of eradication enough, but probably neither did biorisk organizations last year.
I feel that this discussion is not framed correctly. Yes, successful eradication is superior to containment in health and economic outcomes. This is a pretty weak claim that lots of people can agree with who otherwise differ considerably in their policy proposals. But the original claim was that EAs and rationalists hadn’t advocated for long lockdowns and border closures, and that this was relevant for retroactively assessing their performance. The plausibility of the latter claim must be evaluated by considering all the countries that implemented long lockdowns and border closures, and not just the tiny minority that were successful in attaining (near-)eradication by adopting those measures.
I took a quick look at the study you shared. Their analysis compared covid deaths, GDP growth and lockdown stringency in two groups of OECD countries during the first twelve months of the pandemic, and offered this as their original contribution to the study’s main thesis that countries which favored elimination had better health and economic outcomes than countries which favored mitigation (the rest of the study is a brief and unsystematic summary of some of the relevant literature). It turns out that the group which supposedly favored elimination consists of just five nations, four of which are islands and the fifth of which (South Korea) shares borders with a single county which has been completely isolated from the rest of the world for decades.
Let’s pause for a moment and consider how quickly this kind of evidence would have been dismissed if it had been presented in support of a politically inconvenient conclusion. Yet here it is offered, in the world’s most prestigious medical journal, to establish that “elimination, not mitigation, creates best outcomes for health, the economy, and civil liberties”.
For what it’s worth, I personally have no strong views on how the pandemic should have been handled. (My only strong meta-view is that decisions should have been based on explicit cost-effectiveness analyses, which were surprisingly absent from most policy discussions.) My impression is that EAs and rationalists—to which I would add the Metaculus community—did somewhat better than most experts, but this assessment is based mostly on comparing their performance on simple factual questions or issues involving basic sanity. Here, instead, we are told to downgrade our estimation of how well EAs and rationalists performed because, apparently, they did not advocate for the entire world a policy that was successful in only a handful of geographically highly isolated countries. I don’t find that plausible.
I don’t think I can help much with answering these questions.
I was thinking of counties like Australia and New Zealand and Taiwan. But whether or not the strategies adopted in these places was actually optimal or best with the available information or applicable to most countries that are not islands or had a high chance of failure – I cannot say!
All I can say is that there is at least one plausible strategy that seems to have worked well in at least some countries and I personally don’t really remember it being discussed within the EA space a year ago.
Feel free to draw what conclusions or analysis you will from that.
the enormous costs
Just to add, I expect (but I might be wrong) that these countries have had lower welfare and economic costs than most other places.
I somewhat agree, but I think this point becomes much weaker if it was the case that at the same time when these organizations were giving poor advice some amateurs in the EA and rationality communities had already arrived at better conclusions, would have given better advice, etc.
I didn’t follow the relevant conversations closely enough to have much of an inside view on how strongly the latter is true, but my impression is that many people in the EA/rationality communities (including ones who did follow the conversations more closely) think it’s true. Even I am aware of some data points that seem to suggest such a conclusion (e.g., some conversations I remember).
FWIW, at least given what I know I think I find this less compelling as a ‘vindication of EA/rationalist epistemics’ or whatever than some other people seem to—I think the lessons we should learn depend on a number of additional things which I’m currently uncertain about:
Are we comparing like with like rather than cherrypicking? (I.e., comparing anecdotes of flawed advice from ‘expert organizations’ to anecdotes of ex-post correct advice from EAs/rationalists?)
Ideally I’d want to know about the full distribution of views among ‘expert organizations’ and the full distribution of views among ‘amateur EAs/rationalists who spent time looking into COVID things’, and then compare those.
What about some other relevant groups? E.g., how “well” did pharma/vaccine companies do? What about academics developing tests? It does seem like at least some of these groups started taking the novel coronavirus pretty seriously in January already.
What should we think about the ‘external validity’ of EA/rationalist successes? Some people did well in predicting how COVID would play out, what recommendations are good, etc., in the environmental condition of “having significant spare time and being able to post and discuss views with like-minded people without facing a lot of other constraints”. Would they still have done well if they had been in the environmental condition “expert making a public statement that needs to make sense to the general public” plus whatever other incentives these experts & organizations were subject to?
(The above wording might sound like I’m thinking it was “easier” for EAs/rationalists to arrive at correct conclusions or give good advice when making statements we would hear about. However, in fact, I’m unsure about the “net effect” of incentives and other environmental conditions. Similarly, I don’t mean to suggest that ‘external validity’ is necessarily poor, just that it seems worth thinking about before drawing strong conclusions.)
However, no matter the answers to these questions, your claim to me still sounds too generous to these organizations.
Hi, Yes good point, maybe I am being too generous.
FWIW I don’t remember anyone in the EA / rationalist community calling for the strategy that post-hoc seems to have worked best of a long lock-down to get to zero cases followed by border closures etc to keep cases at zero. (I remember a lot of people for example sharing this note which gets much right but stuff wrong: eg short lock-dock and comparatively easy to keep R below 1 with social distancing)
Can you say more about the strategy you have in mind and give examples of the countries that implemented it? It’s hard to judge whether EAs or rationalists advocated it from your characterization, which seems to build into it the relevant success conditions. Lots of countries had long lockdowns and strict border closures, but failed miserably at bringing cases down to zero or anything remotely approaching zero. Furthermore, whether this approach was best all-things-considered should obviously also take into account the enormous costs associated with these measures (in all of the countries that implemented long lockdowns and strict border closures).
weeatquince’s is sharing a widely held view, i.e. that eradication is superior to containment in health and economic outcomes, see e.g. this analysis. The idea itself is plausible, since a successful lockdown allows complete reopening of the internal economy afterwards.
Sample size is however small, especially when it comes to non-island countries. I only know of two non-island countries that seriously went for eradication coupled with border closures, namely Vietnam and Israel. Israel gave up at one point when cases started to rise (which is why it is listed among the containment countries in the analysis above) , but Vietnam succeeded (although it had to restrict travel heavily domestically as well). Personally, I believe it is a suboptimal strategy for non- authoritarian, non-island countries.
I think their original point stands though, that EA/rationalists did not seem to entertain the idea of eradication enough, but probably neither did biorisk organizations last year.
Thanks for the clarification.
I feel that this discussion is not framed correctly. Yes, successful eradication is superior to containment in health and economic outcomes. This is a pretty weak claim that lots of people can agree with who otherwise differ considerably in their policy proposals. But the original claim was that EAs and rationalists hadn’t advocated for long lockdowns and border closures, and that this was relevant for retroactively assessing their performance. The plausibility of the latter claim must be evaluated by considering all the countries that implemented long lockdowns and border closures, and not just the tiny minority that were successful in attaining (near-)eradication by adopting those measures.
I took a quick look at the study you shared. Their analysis compared covid deaths, GDP growth and lockdown stringency in two groups of OECD countries during the first twelve months of the pandemic, and offered this as their original contribution to the study’s main thesis that countries which favored elimination had better health and economic outcomes than countries which favored mitigation (the rest of the study is a brief and unsystematic summary of some of the relevant literature). It turns out that the group which supposedly favored elimination consists of just five nations, four of which are islands and the fifth of which (South Korea) shares borders with a single county which has been completely isolated from the rest of the world for decades.
Let’s pause for a moment and consider how quickly this kind of evidence would have been dismissed if it had been presented in support of a politically inconvenient conclusion. Yet here it is offered, in the world’s most prestigious medical journal, to establish that “elimination, not mitigation, creates best outcomes for health, the economy, and civil liberties”.
For what it’s worth, I personally have no strong views on how the pandemic should have been handled. (My only strong meta-view is that decisions should have been based on explicit cost-effectiveness analyses, which were surprisingly absent from most policy discussions.) My impression is that EAs and rationalists—to which I would add the Metaculus community—did somewhat better than most experts, but this assessment is based mostly on comparing their performance on simple factual questions or issues involving basic sanity. Here, instead, we are told to downgrade our estimation of how well EAs and rationalists performed because, apparently, they did not advocate for the entire world a policy that was successful in only a handful of geographically highly isolated countries. I don’t find that plausible.
I don’t think I can help much with answering these questions.
I was thinking of counties like Australia and New Zealand and Taiwan. But whether or not the strategies adopted in these places was actually optimal or best with the available information or applicable to most countries that are not islands or had a high chance of failure – I cannot say!
All I can say is that there is at least one plausible strategy that seems to have worked well in at least some countries and I personally don’t really remember it being discussed within the EA space a year ago.
Feel free to draw what conclusions or analysis you will from that.
Just to add, I expect (but I might be wrong) that these countries have had lower welfare and economic costs than most other places.