Thank you for this detailed and transparent response!
I applaud HLI for creating a chart (and now an R Shiny App) to show how philosophical views can affect the tradeoff between predominately life-saving and predominately life-enhancing interventions. However, one challenge with that approach is that almost any changes to your CEA model will be outcome-changing for donors in some areas of that chart. [1]
For example, the 53-> 38% correction alone switched the recommendation for donors with a deprivationist framework who think the neutral point is over ~ 0.65 but under 1.58. Given that GiveWell’s moral weights were significantly derived from donor preferences, and (0.5, deprivationism) is fairly implied by those donor weights, I think that correction shifted the recommendation from SM to AMF for a significant number of donors even though it was only material to one of three philosophical approaches and about 1 point of neutral-point assumptions.
GiveWell reduced the WELLBY estimate from about 62 (based on the 38% figure) to about 17, a difference of about 45. If I’m simplifying your position correctly, for about half of those WELLBYs you disagree with GiveWell that an adjustment is appropriate. For about half of them, you believe a discount is likely appropriate, but think it is likely less than GiveWell modelled.
If we used GiveWell’s numbers for that half but HLI’s numbers otherwise, that split suggests that we’d end up with about 39.5 WELLBYs. So one way to turn your response into a donor-actionable statement would be to say that there is a zone of uncertainty between 39.5 and 62 WELLBYs. One might also guess that the heartland of that zone is between about 45 and 56.5 WELLBYs, reasoning that it is less likely that your discounts will be less than 25% or more than 75% of GiveWell’s.
The bottom end of that zone of uncertainty (39.5) would pull the neutral point at which a deprivationist approach would conclude AMF = SM up to about 2.9. I suspect few people employing a deprivationist approach have the neutral point that high. AMF is also superior to SM on a decent number of TRIA-based approaches at 39.5 WELLBYs.
So it seems there are two reasonable approaches to donor advice under these kinds of circumstances:
One approach would encourage donors within a specified zone of uncertainty to hold their donations until HLI sufficiently updates its CEA for SM to identify a more appropriate WELLBY figure ; or
The other approach would encourage donors to make their decision based on HLI’s best estimate of what the WELLBY figure on the next update of the CEA will be. Even if the other approach is correct, there will be some donors who need to use this approach for various reasons (e.g., tax reasons).
I don’t think reaffirming advice on the current model in the interim without any adjustments is warranted, unless you believe the adjustments will be minor enough such that a reasonable donor would likely not find them of substantive importance no matter where they are on the philosophical chart.[2]
In the GiveWell model, the top recommendation is to give to a regranting fund, and there isn’t any explicit ranking of the four top charities. So the recommendation is actually to defer the choice of specific charity to someone who has the most up-to-date information when the monies are actually donated to the effective charity. Moreover, all four top charities are effective in very similar ways. Thus, GiveWell’s bottom-line messaging to donors is much less sensitive to changes in the CEA for any given charity.
I am not sure how to define “minor.” I think whether the change flips the recommendation to the donor is certainly relevant, but wouldn’t go so far as to say that any change that flips the recommendation for a given donor’s philosophical assumptions would be automatically non-minor. On the other hand, I think a large enough change can be non-minor even if it doesn’t flip the recommendation on paper. Some donors apply discounts and bonuses not reflected in HLI’s model. For instance, one could reasonably apply a discount to SM when compared to better-studied interventions, on the basis that CEAs usually decrease as they become more complete. Or one could reasonably apply a bonus to SM because funding a smaller organization is more likely to have a positive effect on its future cost-effectiveness. Thus, just because the change is not outcome-determinative on HLI’s base model doesn’t mean it isn’t so on the donor’s application of the model. The time-to-update and amount of funds involved are also relevant. All that being said, my gut thinks that the starting point for determining minor vs. non-minor is somewhere in the neighborhood of 10%.
You raise a fair point. One we’ve been discussing internally. Given the recent and expected adjustments to StrongMinds, it seems reasonable to update and clarify our position on AMF to say something like, “Under more views, AMF is better than or on par with StrongMinds. Note that currently, under our model, when AMF is better than StrongMinds, it isn’t wildly better.” Of course, while predicting how future research will pan out is tricky, we’d aim to be more specific.
Thank you for this detailed and transparent response!
I applaud HLI for creating a chart (and now an R Shiny App) to show how philosophical views can affect the tradeoff between predominately life-saving and predominately life-enhancing interventions. However, one challenge with that approach is that almost any changes to your CEA model will be outcome-changing for donors in some areas of that chart. [1]
For example, the 53-> 38% correction alone switched the recommendation for donors with a deprivationist framework who think the neutral point is over ~ 0.65 but under 1.58. Given that GiveWell’s moral weights were significantly derived from donor preferences, and (0.5, deprivationism) is fairly implied by those donor weights, I think that correction shifted the recommendation from SM to AMF for a significant number of donors even though it was only material to one of three philosophical approaches and about 1 point of neutral-point assumptions.
GiveWell reduced the WELLBY estimate from about 62 (based on the 38% figure) to about 17, a difference of about 45. If I’m simplifying your position correctly, for about half of those WELLBYs you disagree with GiveWell that an adjustment is appropriate. For about half of them, you believe a discount is likely appropriate, but think it is likely less than GiveWell modelled.
If we used GiveWell’s numbers for that half but HLI’s numbers otherwise, that split suggests that we’d end up with about 39.5 WELLBYs. So one way to turn your response into a donor-actionable statement would be to say that there is a zone of uncertainty between 39.5 and 62 WELLBYs. One might also guess that the heartland of that zone is between about 45 and 56.5 WELLBYs, reasoning that it is less likely that your discounts will be less than 25% or more than 75% of GiveWell’s.
The bottom end of that zone of uncertainty (39.5) would pull the neutral point at which a deprivationist approach would conclude AMF = SM up to about 2.9. I suspect few people employing a deprivationist approach have the neutral point that high. AMF is also superior to SM on a decent number of TRIA-based approaches at 39.5 WELLBYs.
So it seems there are two reasonable approaches to donor advice under these kinds of circumstances:
One approach would encourage donors within a specified zone of uncertainty to hold their donations until HLI sufficiently updates its CEA for SM to identify a more appropriate WELLBY figure ; or
The other approach would encourage donors to make their decision based on HLI’s best estimate of what the WELLBY figure on the next update of the CEA will be. Even if the other approach is correct, there will be some donors who need to use this approach for various reasons (e.g., tax reasons).
I don’t think reaffirming advice on the current model in the interim without any adjustments is warranted, unless you believe the adjustments will be minor enough such that a reasonable donor would likely not find them of substantive importance no matter where they are on the philosophical chart.[2]
In the GiveWell model, the top recommendation is to give to a regranting fund, and there isn’t any explicit ranking of the four top charities. So the recommendation is actually to defer the choice of specific charity to someone who has the most up-to-date information when the monies are actually donated to the effective charity. Moreover, all four top charities are effective in very similar ways. Thus, GiveWell’s bottom-line messaging to donors is much less sensitive to changes in the CEA for any given charity.
I am not sure how to define “minor.” I think whether the change flips the recommendation to the donor is certainly relevant, but wouldn’t go so far as to say that any change that flips the recommendation for a given donor’s philosophical assumptions would be automatically non-minor. On the other hand, I think a large enough change can be non-minor even if it doesn’t flip the recommendation on paper. Some donors apply discounts and bonuses not reflected in HLI’s model. For instance, one could reasonably apply a discount to SM when compared to better-studied interventions, on the basis that CEAs usually decrease as they become more complete. Or one could reasonably apply a bonus to SM because funding a smaller organization is more likely to have a positive effect on its future cost-effectiveness. Thus, just because the change is not outcome-determinative on HLI’s base model doesn’t mean it isn’t so on the donor’s application of the model. The time-to-update and amount of funds involved are also relevant. All that being said, my gut thinks that the starting point for determining minor vs. non-minor is somewhere in the neighborhood of 10%.
Jason,
You raise a fair point. One we’ve been discussing internally. Given the recent and expected adjustments to StrongMinds, it seems reasonable to update and clarify our position on AMF to say something like, “Under more views, AMF is better than or on par with StrongMinds. Note that currently, under our model, when AMF is better than StrongMinds, it isn’t wildly better.” Of course, while predicting how future research will pan out is tricky, we’d aim to be more specific.