Hi Vasco. Thanks for all of your help giving feedback on the report and the modeling underpinning the CEA.
I am going to focus on the main points that you make. I hope to explain why I chose not to adopt the changes you mention in your comment and also to highlight some key weaknesses and limitations of my model.
Points I address (paraphrasing what you said):
By asking domain experts you probably got an overestimate for “probability that advocacy succeeds”. You should have also asked people in other fields.
Although you mention fungibility, you don’t account for it in cost-effectiveness estimates. You should be more explicit that fungibility undermines cost-effectiveness of grants that perform other, less effective interventions.
You overestimate the mortality effects of mild cooling events: if we apply your model to the 1815 eruption, we get higher mortality rates than actually occurred.
Poverty is a strong predictor of famine mortality. If your persistence estimate relied only on poverty & malnutrition burden indicators, the full-term benefits of policy advocacy would be significantly lower
By asking domain experts you probably got an overestimate for “probability that advocacy succeeds”. You should have also asked people in other fields.
I agree that domain experts are likely to overestimate the probability of successful policy advocacy in their space.
In my defence, only two of the seven experts I consulted for estimates worked in food resilience specifically. The geometric mean of their estimates was 30%; only slightly higher than the group average of 24%. The other experts would be best classified as GCR experts (so still likely to be overly optimistic)
The difficulty is that people who are not domain experts are (by definition) not well-informed. I don’t think people at GiveWell will have an accurate understanding of prospects for ASRS resilience policy advocacy. Especially because this is a very small field.
Although you mention fungibility, you don’t account for it in cost-effectiveness estimates. You should be more explicit that fungibility undermines cost-effectiveness of grants that perform other, less effective interventions.
I think that in ideal circumstances, fungibility should be accounted for in cost-effectiveness analysis. But since it depends on the organization receiving the funding, I decided not to do quantitative estimates of fungibility effects in this report. Maybe we will do so when we evaluate specific grants in this area.
I agree that funding to orgs who only do one highly cost-effective thing is generally less fungible.
You overestimate the mortality effects of mild cooling events: if we apply your model to the 1815 eruption, we get higher mortality rates than actually occurred.
I love this analysis, thanks for doing it.
First, let me say that yes, my model is very sensitive to mortality estimates in mild cooling scenarios. My estimate may be too high, but I believe there are compelling reasons not to be confident of this.
To illustrate my model for mortality in a mild cooling event (1-2.65 degrees cooling):
2% probability of 6% mortality (no adaptation, no food trade)
18% probability of 3% mortality (no food trade)
80% probability of no mortality
This gives an average of approximately 0.6% mortality
My counterarguments are as follows:
On a broad level, I think that a ‘panic’ scenario could include countries banning food exports to secure domestic supplies. This would be catastrophic for some food-importing countries even in normal climate conditions. The same cannot be said for the world of 1815, where almost all food was consumed locally and very few people lived far from agricultural areas.
I think the comparison with 1815 is well worth doing. However, there are a number of reasons why the validity of the comparison is limited:
A global 1% mortality event in 1815 may not have even been noticed. We have to patch together estimates of famine mortality in 1815 because there was almost no systematic documentation at the time. People were not aware of any global phenomenon, so nobody was trying to “join up the dots” and tally the full famine impact that year. Especially if much of the effects were felt in South Asia, East Asia or Africa, it is possible that a major-yet-distributed famine could have gone unnoticed
Relatively few people in the world of 1815 relied on food imports, as mentioned above[1]. Breakdown in international food trade is the main famine mechanism in modern-day agricultural catastrophes, but it barely applies to the agrarian economies of 1815.
Local famine effects may not have been worse in Indonesia. Undoubtedly, the effects of ash blanketing crops would have been worse near the eruption. But stratospheric soot quickly circulates around the world. Furthermore, Indonesia has a warm climate and would not have been at risk from completely failed harvests through unseasonal frost. Multiple annual harvests are common in the tropics; in higher latitudes, a failed crop leaves farmers without food for almost a year.
Cooling damage is highly superlinear. The Pinatubo eruption of 1991 caused 0.5 degrees of cooling and is not associated with important declines in agricultural productivity. Thus we might expect the expected burden of an 1815-level cooling event (0.8 to 1.3 degrees cooling) to be far lower than a 1-2.65 degree cooling event[2].
Poverty is a strong predictor of famine mortality. If your persistence estimate relied only on poverty & malnutrition burden indicators, the full-term benefits of policy advocacy would be significantly lower.
To push back:
A world with no extreme poverty and no nutritional diseases would still be vulnerable to global agricultural catastrophes. The advantage of my “grain production per capita” metric is that it has no such edge-case problems: more grain is always good for food security.
The famine/poverty relationship is strong in normal times. There are a number of reasons that it may become less strong in an agricultural catastrophe
Many of the global poor are subsistence farmers in warm countries. This demographic will be close to food supplies and away from the risk of frosts etc.. They may be better-positioned to survive than their middle-class compatriots in cities.
People at the bottom of the pile are the most at-risk of post-catastrophe famine, regardless of whether they are in absolute poverty. A wealthier world would simply have higher food prices, leaving the poorest without enough to eat.
As described above, the main cause of famine in my model is the breakdown of food trade. It is not clear that progress against poverty and nutritional diseases is an indication that populations are less dependent on food trade. If anything, development is probably associated with increased reliance on trade.
A counterargument could be that Western Europe appears to have had particularly bad summer cooling in 1816 - as well as better record-keeping than much of the world—and their famines were not so bad. On the other hand, spring cooling may be more important, as late frosts can ruin harvests of wheat, potatoes etc.
Thanks for the clarifying comment, Stan! I strongly upvoted it.
As a preliminary note, I think it makes a lot of sense to give feedback on analyses like yours privately, but I wonder whether it is worth for me to invest significant time in writing comments like mine above. It seems that they are often downvoted, and that I can sometimes tell before hand when this is going to be case. So, to the extent karma is a good proxy for what people value, I wonder whether I am just spending signicant time on doing something which has little value. In this particular case, I am still guessing it was worth it because, even if it had negligible value to the public, it was still relevant for my own cause prioritisation (and making it public had little cost).
For what is worth, I was already aware of the arguments you mentioned, and directionally agree with all the points you make. I just think their effect is not as strong as you do, so I maintain my adjustments are warranted.
In any case:
[...] even if not, I estimate it [ASRS policy advocacy] is only 1.61 % (= 0.242/15.0) as cost-effective as corporate campaigns for chicken welfare, such as the ones supported by The Humane League (THL). So I think donors who value 1 unit of welfare in humans as much as 1 unit of welfare in animals (i.e. who reject speciesism) had better donate to THL instead of an organisation doing ASRS policy advocacy.
I would be curious to know CEARCH’s position on animal welfare. I noted there are 0 animal welfare causes in your long list of 588. Their absence is especially surprising given the presence of causes like sporting excellence and freedom of hobby.
Could you comment on the 2 points above?
A global 1% mortality event in 1815 may not have even been noticed.
Agreed, so I adjusted for underreporting in my calculations. I considered the actual mortality to be 13.9 (= 1⁄0.0721) times as high as the reported one.
Cooling damage is highly superlinear.
Agreed, so I adjusted less strongly your mortality rates for more severe coolings:
I wonder whether it is worth for me to invest significant time in writing comments like mine above. It seems that they are often downvoted, and that I can sometimes tell before hand when this is going to be case. So, to the extent karma is a good proxy for what people value, I wonder whether I am just spending signicant time on doing something which has little value.
I am sad to see your comment getting downvotes as I do think it contributes a lot of value to the discussion.
I can guess why you might be getting them. You often respond to cause-prio posts with “what about corporate campaigns for chicken welfare?”, and many people now probably switch off and downvote when they see this. Maybe just keep the chicken comparison to one line and link to your original post/comment?
Also, you comment is 3200 words long—about 3x longer than the actual post. I think a 200-word summary-of-the-comment with bullet points would be really useful for readers who have only read this post and are unable to pick up the finer points of your modeling critique.
On animal welfare
I think that if you adopt RP’s moral weight estimates and reject speciesism, it is almost inevitable that the most cost-effective interventions to improve wellbeing will be animal welfare interventions.
My understanding is that CEARCH is not against evaluating animal welfare interventions in principle, but in practice we are not doing so while comparisons between human and animal welfare remain so shaky. Our research direction is also partly driven by the value of information, ie. how much resources we can plausibly redirect and the impact this will have. Maybe this is too deterministic of me, but I feel that banging the drum about corporate chicken campaigns will only open so many wallets.
Hi Vasco. Thanks for all of your help giving feedback on the report and the modeling underpinning the CEA.
I am going to focus on the main points that you make. I hope to explain why I chose not to adopt the changes you mention in your comment and also to highlight some key weaknesses and limitations of my model.
Points I address (paraphrasing what you said):
By asking domain experts you probably got an overestimate for “probability that advocacy succeeds”. You should have also asked people in other fields.
Although you mention fungibility, you don’t account for it in cost-effectiveness estimates. You should be more explicit that fungibility undermines cost-effectiveness of grants that perform other, less effective interventions.
You overestimate the mortality effects of mild cooling events: if we apply your model to the 1815 eruption, we get higher mortality rates than actually occurred.
Poverty is a strong predictor of famine mortality. If your persistence estimate relied only on poverty & malnutrition burden indicators, the full-term benefits of policy advocacy would be significantly lower
By asking domain experts you probably got an overestimate for “probability that advocacy succeeds”. You should have also asked people in other fields.
I agree that domain experts are likely to overestimate the probability of successful policy advocacy in their space.
In my defence, only two of the seven experts I consulted for estimates worked in food resilience specifically. The geometric mean of their estimates was 30%; only slightly higher than the group average of 24%. The other experts would be best classified as GCR experts (so still likely to be overly optimistic)
The difficulty is that people who are not domain experts are (by definition) not well-informed. I don’t think people at GiveWell will have an accurate understanding of prospects for ASRS resilience policy advocacy. Especially because this is a very small field.
Although you mention fungibility, you don’t account for it in cost-effectiveness estimates. You should be more explicit that fungibility undermines cost-effectiveness of grants that perform other, less effective interventions.
I think that in ideal circumstances, fungibility should be accounted for in cost-effectiveness analysis. But since it depends on the organization receiving the funding, I decided not to do quantitative estimates of fungibility effects in this report. Maybe we will do so when we evaluate specific grants in this area.
I agree that funding to orgs who only do one highly cost-effective thing is generally less fungible.
You overestimate the mortality effects of mild cooling events: if we apply your model to the 1815 eruption, we get higher mortality rates than actually occurred.
I love this analysis, thanks for doing it.
First, let me say that yes, my model is very sensitive to mortality estimates in mild cooling scenarios. My estimate may be too high, but I believe there are compelling reasons not to be confident of this.
To illustrate my model for mortality in a mild cooling event (1-2.65 degrees cooling):
2% probability of 6% mortality (no adaptation, no food trade)
18% probability of 3% mortality (no food trade)
80% probability of no mortality
This gives an average of approximately 0.6% mortality
My counterarguments are as follows:
On a broad level, I think that a ‘panic’ scenario could include countries banning food exports to secure domestic supplies. This would be catastrophic for some food-importing countries even in normal climate conditions. The same cannot be said for the world of 1815, where almost all food was consumed locally and very few people lived far from agricultural areas.
I think the comparison with 1815 is well worth doing. However, there are a number of reasons why the validity of the comparison is limited:
A global 1% mortality event in 1815 may not have even been noticed. We have to patch together estimates of famine mortality in 1815 because there was almost no systematic documentation at the time. People were not aware of any global phenomenon, so nobody was trying to “join up the dots” and tally the full famine impact that year. Especially if much of the effects were felt in South Asia, East Asia or Africa, it is possible that a major-yet-distributed famine could have gone unnoticed
Relatively few people in the world of 1815 relied on food imports, as mentioned above[1]. Breakdown in international food trade is the main famine mechanism in modern-day agricultural catastrophes, but it barely applies to the agrarian economies of 1815.
Local famine effects may not have been worse in Indonesia. Undoubtedly, the effects of ash blanketing crops would have been worse near the eruption. But stratospheric soot quickly circulates around the world. Furthermore, Indonesia has a warm climate and would not have been at risk from completely failed harvests through unseasonal frost. Multiple annual harvests are common in the tropics; in higher latitudes, a failed crop leaves farmers without food for almost a year.
Cooling damage is highly superlinear. The Pinatubo eruption of 1991 caused 0.5 degrees of cooling and is not associated with important declines in agricultural productivity. Thus we might expect the expected burden of an 1815-level cooling event (0.8 to 1.3 degrees cooling) to be far lower than a 1-2.65 degree cooling event[2].
Poverty is a strong predictor of famine mortality. If your persistence estimate relied only on poverty & malnutrition burden indicators, the full-term benefits of policy advocacy would be significantly lower.
To push back:
A world with no extreme poverty and no nutritional diseases would still be vulnerable to global agricultural catastrophes. The advantage of my “grain production per capita” metric is that it has no such edge-case problems: more grain is always good for food security.
The famine/poverty relationship is strong in normal times. There are a number of reasons that it may become less strong in an agricultural catastrophe
Many of the global poor are subsistence farmers in warm countries. This demographic will be close to food supplies and away from the risk of frosts etc.. They may be better-positioned to survive than their middle-class compatriots in cities.
People at the bottom of the pile are the most at-risk of post-catastrophe famine, regardless of whether they are in absolute poverty. A wealthier world would simply have higher food prices, leaving the poorest without enough to eat.
As described above, the main cause of famine in my model is the breakdown of food trade. It is not clear that progress against poverty and nutritional diseases is an indication that populations are less dependent on food trade. If anything, development is probably associated with increased reliance on trade.
Thanks again for the detailed feedback!!
Admittedly, this would have made some people more vulnerable as it was difficult to relieve famine-stricken areas.
A counterargument could be that Western Europe appears to have had particularly bad summer cooling in 1816 - as well as better record-keeping than much of the world—and their famines were not so bad. On the other hand, spring cooling may be more important, as late frosts can ruin harvests of wheat, potatoes etc.
Thanks for the clarifying comment, Stan! I strongly upvoted it.
As a preliminary note, I think it makes a lot of sense to give feedback on analyses like yours privately, but I wonder whether it is worth for me to invest significant time in writing comments like mine above. It seems that they are often downvoted, and that I can sometimes tell before hand when this is going to be case. So, to the extent karma is a good proxy for what people value, I wonder whether I am just spending signicant time on doing something which has little value. In this particular case, I am still guessing it was worth it because, even if it had negligible value to the public, it was still relevant for my own cause prioritisation (and making it public had little cost).
For what is worth, I was already aware of the arguments you mentioned, and directionally agree with all the points you make. I just think their effect is not as strong as you do, so I maintain my adjustments are warranted.
In any case:
Could you comment on the 2 points above?
Agreed, so I adjusted for underreporting in my calculations. I considered the actual mortality to be 13.9 (= 1⁄0.0721) times as high as the reported one.
Agreed, so I adjusted less strongly your mortality rates for more severe coolings:
I am sad to see your comment getting downvotes as I do think it contributes a lot of value to the discussion.
I can guess why you might be getting them. You often respond to cause-prio posts with “what about corporate campaigns for chicken welfare?”, and many people now probably switch off and downvote when they see this. Maybe just keep the chicken comparison to one line and link to your original post/comment?
Also, you comment is 3200 words long—about 3x longer than the actual post. I think a 200-word summary-of-the-comment with bullet points would be really useful for readers who have only read this post and are unable to pick up the finer points of your modeling critique.
On animal welfare
I think that if you adopt RP’s moral weight estimates and reject speciesism, it is almost inevitable that the most cost-effective interventions to improve wellbeing will be animal welfare interventions.
My understanding is that CEARCH is not against evaluating animal welfare interventions in principle, but in practice we are not doing so while comparisons between human and animal welfare remain so shaky. Our research direction is also partly driven by the value of information, ie. how much resources we can plausibly redirect and the impact this will have. Maybe this is too deterministic of me, but I feel that banging the drum about corporate chicken campaigns will only open so many wallets.
Thanks for the feedback on the votes and animal welfare comparison!