GiveWell

Karma: 3,555

GiveWell 5 Aug 2022 21:05 UTC
135 points
0 ∶ 0
on: Deworming and decay: replicating GiveWell’s cost-effectiveness analysis
This is Alex Cohen, GiveWell senior researcher, responding from GiveWell’s EA Forum account.

Joel, Samuel and Michael — Thank you for the deep engagement on our deworming cost-effectiveness analysis.

We really appreciate you prodding us to think more about how to deal with any decay in benefits in our model, since it has the potential to meaningfully impact our funding recommendations.

We agree with HLI that there is some evidence for benefits of deworming declining over time and that this is an issue we haven’t given enough weight to in our analysis.

We’re extremely grateful to HLI for bringing this to our attention and think it will allow us to make better decisions on recommending funding to deworming going forward.

We would like to encourage more of this type of engagement with our research. We’re planning to announce prizes for criticism of our work in the future. When we do, we plan to give a retroactive prize to HLI.

We’re planning to do additional work to incorporate this feedback into an updated deworming cost-effectiveness estimate. In the meantime, we wanted to share our initial thoughts. At a high level:
- We agree with HLI that there is some evidence for benefits of deworming declining over time and that this is an issue we haven’t given enough weight to in our analysis. We don’t totally agree with HLI on how to incorporate decay in our cost-effectiveness model and think HLI is making a mistake that leads it to overstate the decline in cost-effectiveness from incorporating decay. However, we still guess incorporating decay more in our model could meaningfully change our estimated cost-effectiveness of deworming. We plan to conduct additional research and publish updated estimates soon.
- Once we do this work, our best guess is that we will reduce our estimate of the cost-effectiveness of deworming by 10%-30%. Had we made this change in 2019 when KLPS-4 was released, we would have recommended $2-$8m less in grants to deworming (out of $55m total) since 2019.
- We also agree that we should do more to improve the transparency of our cost-effectiveness estimates. We plan to make key assumptions and judgment calls underlying our deworming cost-effectiveness estimate clearer on our website.
Should we adjust the effect of deworming down to account for decay in benefits?

We agree with HLI that there is some evidence in the Kenya Life Panel Survey (KLPS) data for benefits declining over time. We haven’t explicitly made an adjustment in the cost-effectiveness analysis for the possibility that the effects decline over time.

Where I think we disagree is on how to incorporate that potential decay in benefits in our model. While incorporating decay into our model will most likely reduce cost-effectiveness overall, my guess is that HLI’s approach overstates the decline in cost-effectiveness from incorporating decay for three reasons. We plan to explore these further in our updated cost-effectiveness estimate for deworming, but I’ll summarize them quickly here.

First, if we were to model decay based on the KLPS data, we would likely use a higher starting point. We think HLI may have made an error in interpreting the data here.
- To estimate decay, HLI’s model begins with GiveWell’s current estimate of the benefits of deworming — which is based on the average of results from the 10-, 15-, and 20-year follow-ups (KLPS 2, KLPS 3, and KLPS 4, respectively) — then assumes effects decline from that value. However, if we believe that there are declining effects from KLPS 2 to KLPS 3 to KLPS 4, this should imply above average effects in initial years (as found in KLPS 2 and KLPS 3) and then below average effects in later years (as found in KLPS 4).
- Specifically, in its decay model, HLI uses a starting value of 0.006 units of ln(consumption). This is our current estimate of “Benefit of one year’s income (discounted back because of delay between deworming and working for income.” This is based on averaging across results from KLPS 2, KLPS 3, and KLPS 4. If we were to incorporate this decay model, we would most likely choose a value above 0.006 for initial years that declines below 0.006 in later years. For example, if we used the value effect size from KLPS 2 and applied the same adjustments we currently do, this value would be 0.012 in the first year of impacts. We expect this to substantially increase cost-effectiveness, compared to HLI’s model.
Second, we guess that we would not put full weight on the decay model, since it seems like there’s a decent chance the observed decline is due to chance or lack of robustness to different specifications of the effects over time.
- The three rounds of KLPS data provide three estimates for the effect of deworming at 10-, 15- and 20-year follow-up, which seem to show a decline in effect (based on percentage increase in income and consumption) over time. We could interpret this as either a decline in effect over time, as recommended by HLI, or as three noisy estimates of a constant effect over time, which is the interpretation in our current cost-effectiveness analysis. We’re unsure how much credence to put on each of these interpretations going forward, but it’s unlikely that we will decide to put all of our credence on “decline in effect over time.”
- When we look at evidence like this, we typically favor pooled results when there is no a priori reason to believe effects differ over time, across geography, etc. (e.g., a meta-analysis of RCTs for a malaria prevention program) because this increases the precision and robustness of the effect measurement. In cases where there’s more reason to believe the effects vary across time or geography, we’re more likely to focus on “sub-group” results, rather than pooled effects. We acknowledge this is often a subjective assessment.
- In the deworming case, there are some reasons to put weight on the decay story. First, the point estimates we have from KLPS 2, KLPS 3, and KLPS 4, in terms of impact on ln income and consumption, tend toward a decline over time. Second, there are plausible stories for why effects would decline. For example, it’s possible individuals in the control group are catching up to individuals who were dewormed due to broader trends in the economy. This is speculative, however, and we haven’t looked into drivers of changes over time.
- However, we also think there are reasons to put weight on the “noisy effects” story, which is why our current cost-effectiveness analysis uses a pooled estimate as our best guess of effects over time. First, the evidence for decline comes from three imprecise estimates of income and two imprecise estimates from consumption with overlapping confidence intervals. And comparing effect sizes across rounds and measures is not straightforward – for example, the small sample KLPS 3 consumption results implied at least a doubling of deworming’s cost-effectiveness relative to GiveWell’s historical model if taken literally (in part due to idiosyncrasies about effects being measured at the per-capita level in a household, rather than merely for a single individual who was dewormed), which is why we funded a larger consumption survey in KLPS-4 and expected to see a much smaller effect in a larger sample.^[1] Factors like this give us reason to believe that some of the observed decline in particular measures is due to chance or measurement error. In this case, we would expect the average pooled effect factoring in multiple types of measures to be the best predictor of what we’ll find in future KLPS rounds. Second, it seems plausible that effects would be constant over time (or could even compound over time). For example, adults who were dewormed as children and see greater cognitive or educational gains may be less likely to enter sectors like agriculture, which may have flatter earnings trajectories, or be more likely to move to cities, where opportunities for wage growth are higher. However, these stories are also speculative.
- As a result, even if we did incorporate the decay model, we would put less than 100% weight on it. We’d like to look further into studies of interventions where the mechanism (improving child development) is more plausibly similar to deworming to see if this provides additional information, as well as any evidence of mechanisms for deworming specifically that would point toward decline in effects. We’d also like to further explore how to interpret the round-by-round estimates, since many factors change between rounds (such as rates of labor force participation and methods of earnings measurement) and we would like to better understand how to predict future changes in control group standards of living when taking all of this into account.
Third, we would likely update our “replicability adjustment” for deworming, based on these results.
- This is noted in the blog post. HLI notes, though, “The more puzzling concept is the idea that, if you realise you should change one bit of your analysis, you would be justified to arbitrarily alter another, unrelated and non-specified, part of it to ensure you retain a ‘plausible result’.”
- Our approach to the replicability adjustment is to do an informal Bayesian update. We have a prior that is more skeptical of the impact of deworming than seen in the data from KLPS. We incorporate that prior into our estimate of the effect size through the replicability adjustment. The larger the effect size estimated by the KLPS data, the greater the gap between that result and our prior, and the larger the adjustment needed to incorporate our prior.^[2] As a result of this adjustment, we input a smaller effect of ln income in our cost-effectiveness analysis than we would if we took the data from KLPS at face value.
- In our current cost-effectiveness analysis, we adjust the pooled effect of 0.109 in ln income downward by 87%. This reflects our prior belief that we should expect a much lower effect of deworming on later-life income. If we thought the pooled effect was lower than 0.109, we would likely apply a less strict adjustment. We would plan to specify how we update our adjustment in our follow-up work on this.
Our current best guess is that incorporating decay into our cost-effectiveness estimates would reduce the cost-effectiveness of deworming charities by 10%-30%. This adjustment would have led to $2-$8 million less out of $55 million total to deworming since late 2019 (when the most recent deworming study results were released).

We plan to do some additional research to refine our estimates and share an updated cost-effectiveness analysis soon.

Where we’d like to improve on reasoning transparency

We also agree with HLI that we have room for improvement on explaining our cost-effectiveness models. The decision about how to model whether benefits decline is an example of that—the reasoning I outlined above isn’t on our website. We only wrote, “the KLPS 4 results are smaller in magnitude (on a percentage increase basis) and higher variance than earlier survey rounds.”

We plan to update our website to make it clearer what key judgment calls are driving our cost-effectiveness estimates, why we’ve chosen specific parameters or made key assumptions, and how we’ve prioritized research questions that could potentially change our bottom line.

Encouraging more feedback on our research

We’re extremely grateful to HLI for taking the time to dig into our work and provide feedback. We think this type of engagement improves the quality of our research and our grant recommendations, which helps us allocate resources more cost-effectively, and we’d like to encourage more of it.

In the near future, we plan to announce prizes to individuals and organizations who identify issues in our cost-effectiveness analyses that are likely to lead to meaningful changes in our decisions.

As part of that contest, we also plan to retroactively recommend a prize to HLI (details TBD). We believe HLI’s feedback is likely to change some of our funding recommendations, at least marginally, and perhaps more importantly improve our decision-making across multiple interventions.
1. ↩︎
  See the “internal forecasts” we published, which roughly predict a 25% chance of large consumption results (similar to KLPS-3) that would have doubled our estimate of deworming’s cost-effectiveness. (Put differently, we predicted a 75% chance of not updating to the KLPS-3-like magnitude of results after seeing KLPS-4′s larger survey.)
2. ↩︎
  To see how the math on this works, see this tool, which we used in generating our replicability adjustments. If you input a prior with mean = 0.5 and standard deviation = 0.25, and see evidence with mean = 10 and standard deviation = 1, the posterior effect estimate is ~1.1, for a “replicability adjustment” relative to the evidence of 1.1/10 = ~11%. However, if the evidence shows a smaller effect closer to the prior (mean = 5, sd = 1), the estimated posterior effect is ~0.8, with a replicability adjustment of 0.8/5 = ~16%. So, the overall estimated posterior effect falls when the evidence shows a lower effect estimate (from ~1.1 to ~0.8), but the skeptical Bayesian “replicability adjustment” is slightly less extreme in the second case (an 84% discount instead of an 89% discount). This is what we mean when we say that the replicability adjustment must be updated in conjunction with the estimated effect size, and this is what we have done historically.
What links here?

GiveWell 28 Jul 2022 15:59 UTC
75 points
0 ∶ 0
on: Deworming and decay: replicating GiveWell’s cost-effectiveness analysis
Hi, Joel, Sam, and Michael -

We really appreciate this type of thoughtful engagement. We find a lot of value in hearing well-reasoned critiques of our research and weaknesses in our communication: thank you for sharing this!

Facilitating feedback like this is a big part of why we hold transparency as one of our values. You’re right that it shouldn’t be as difficult as it is to understand why we made the decisions we did in our model and how our deworming estimates rely on priors and evidence. Our deworming cost-effectiveness analysis (and frankly other models as well) falls short of our transparency goals–this is a known shortfall in how we communicate about our research, and we are working on improving this.

We want to take the time to more deeply consider the points raised in this post, and do plan on sharing more thinking about our approach. Thanks again for your critical engagement with our research.

GiveWell 6 Jan 2023 23:43 UTC
65 points
10 ∶ 0
on: Misleading phrase in a GiveWell Youtube ad
Hi, Thomas,

We agree that the language you highlight is misleading, and we’re sorry that it was included in the ad. It wasn’t a part of the copy we provided to advertisers, and we didn’t notice it in our review of the ad before it went live.

We’re now speaking to the agency that coordinates these ads about the possibility of updating the video, but we think this is unlikely to happen, as the ad has already been live as is for some time and can’t easily be reuploaded.

Thank you for bringing this to our attention!

Best,

Miranda (GiveWell Communications Associate)

GiveWell 3 Apr 2024 18:43 UTC
54 points
8 ∶ 0
on: [Linkpost] Leif Wenar’s The Deaths of Effective Altruism
We want to share a few thoughts that might help clarify our approach.
Our research incorporates potential downsides or unintended consequences, such that our recommendations take into account factors like potential side effects. Most (if not all) of the information in Leif Wenar’s WIRED piece is something that anyone could find on our website. However, we believe some of what is published in the piece is misleading or inaccurate. Here is a lightly edited version of the material that GiveWell sent to WIRED in response to a request for comment on Leif Wenar’s piece.
What links here?
- [Linkpost] Leif Wenar’s The Deaths of Effective Altruism by Arden Wiese (27 Mar 2024 15:35 UTC; 38 points)

GiveWell 5 Mar 2024 20:53 UTC
49 points
0 ∶ 0
on: Bloomberg: Unacknowledged problems with LLINs are causing a rise in malaria.
Hi Ian, thanks for sharing this article – our team wrote up some notes on how this topic intersects with our work, and I (Isabel Arjmand writing on behalf of GiveWell) thought it might be useful to share here.
Like the Bloomberg editorial board, we’re concerned about stalling progress in the fight against malaria, but we’re skeptical that quality issues with PermaNet 2.0s have influenced this progress as much as the article suggests.
All things considered, we believe that malaria nets are, and have been, highly effective in reducing malaria burden. The Against Malaria Foundation had first shared the studies highlighted above with us in 2020, and the claims in the Bloomberg article have prompted us to do some additional research.
Based on the work we’ve done so far, we aren’t convinced that decreased net quality is primarily responsible for malaria resurging in Papua New Guinea. So far, we see this as a milder negative update on nets than the article would indicate, in part because we think these tests of net quality may not be a perfect proxy for effectiveness in reducing cases and in part because we no longer fund PermaNet 2.0s (for unrelated reasons). At the same time, renewed interest in the evidence around PermaNet 2.0 quality is a nudge for us to prioritize further work to understand net quality control in general.
More detail on the implications of this research for GiveWell’s work
While we no longer fund PermaNet 2.0s because we now fund newer types of nets instead, they make up roughly 20% of nets we’ve funded historically. The studies referenced in the Bloomberg article looked at nets distributed in Papua New Guinea and indicate that the post-2012 PermaNet 2.0s perform worse on certain efficacy tests. We aren’t sure how well those efficacy tests serve as a proxy for malaria transmission (e.g. mosquitoes in these tests could be impaired from the exposure to insecticides even if it isn’t sufficient to kill them). We’re also skeptical that changes to the formulation of PermaNet 2.0s were the key driver of increased malaria cases in Papua New Guinea. During this time, we think other factors like insecticide resistance and shifts in biting patterns likely played a meaningful role (as highlighted in this paper). That said, we see these studies as a negative update on the effectiveness of those nets.
We did a quick back-of-the-envelope calculation (so this is more illustrative than fully baked, at this point):
- Assuming the insecticide treatment on PermaNet 2.0s was 80% less effective after 2012 would make those nets look 30-50% less effective overall than we’d previously modeled. That’s because we model roughly 30% of the benefit of nets as coming from the physical barrier in the absence of insecticide resistance, and we already discount the effectiveness of nets like PermaNet 2.0 because of insecticide resistance. We would guess that with further work, we’d estimate that 80% is on the pessimistic side of things (which would put the overall impact on net efficacy at the low end of our 30-50% range, or lower).
- Then, assuming that similar issues don’t apply to other nets (which could be wrong – we plan to look into this more), our overall nets grantmaking would look roughly 5-10% less cost-effective than we’d previously estimated, since PermaNet 2.0s are around 20% of our historical nets distributed. That proportion has varied over time. In 2018, all of the nets we funded were PermaNet 2.0s; now, we fund newer types of nets instead.
While concerns specific to PermaNet 2.0s don’t directly affect our future allocation decisions, this issue does raise more general concerns about quality control for nets. Ideally, we would have prioritized more work in this area in the past. We’re planning to learn more about quality control processes and we also want to better understand how others in the malaria field are thinking about this.

GiveWell 18 Jan 2024 20:08 UTC
24 points
2 ∶ 0
on: Malaria vaccines: how confident are we?
Thanks SoGive for the post! We wanted to share some of GiveWell’s current thinking around malaria vaccines in case it’s helpful. We also wrote a report on RTS,S in 2022 here and have recommended a couple grants for vaccine rollout and research.
On a cost-per-person-reached basis, we agree ITNs and SMC are superior to either of the two WHO-approved malaria vaccines. However, we think there’s less of a differential in cost-effectiveness than this post implies, for a number of reasons:
- The difference in all-in delivery costs is probably less substantial: We think it roughly costs $6 to deliver an ITN to a household^[1], and roughly $7 to provide a child with a full course of SMC. (See also impact metrics here). We estimate the total costs for fully immunizing a child range from $23-$43, depending on the choice of vaccine.^[2]
- We would approach the comparison for the reduction in malaria incidence differently: Using the effect size from the Pryce et al meta-analysis for ITNs (which consists mostly of trials that lasted one year) and comparing it with the incidence reduction observed after 12 months in RTS,S/R21 trials is not as straightforward as is seems:
  - The reduction found in Pryce et al has to be considerably updated in light of recent developments, notably new types of nets (PBO, dual active ingredient) and increased insecticide resistance.
  - The RTS,S Clinical Partnership trial includes results from a follow-up four years after the start of the intervention and finds a reduction in clinical malaria from 28% (3 dose group) to 36% (4-dose group) at endpoint. Our malaria team recommends these results rather than the earlier snapshots, as they are less noisy.
  - Our best guess is that R21 and RTS,S are similarly effective at preventing malaria. Available evidence suggests that short-term effectiveness is broadly similar between RTS,S and R21. No data has been published for the impact of R21 on malaria incidence over the long run (>20 months after the first dose). Because the short-term outcomes are broadly similar, and both vaccines employ the same mechanism to induce an immune response in the vaccinated person^[3], our best guess is that four doses of R21 probably offer similar levels of protection as four doses of RTS,S over the long run.
  - We believe the apparent difference cited in the post is most likely due to the different setup between trials (Datoo 2022, unlike the RTS,S trial, was carried out in a seasonal setting). Note also that results from a phase III trial on R21 are now available (Datoo 2023 (preprint)).
- The duration of protection differs: Due to factors such as attrition and physical decay, we currently estimate an ITN to provide between 1.2 − 2.0 years of effective protection. As indicated above, we estimate that malaria vaccines offer around 30% protection over four years.
There are, of course, additional factors that need to be taken into account to get a full picture (for example, what coverage levels are achievable for each intervention?). However, our current best guess is that even with those included, nets (and SMC) will be more cost-effective than malaria vaccines—just not by an order of magnitude.
1. ^
  Costs per child reached are much higher, roughly $15-$26.
2. ^
  The price per dose of RTS,S was $9.30 in 2022, and estimates for R21 indicate a price per dose of $3.90. We expect that, on average, 70% of children who received three doses will also get a booster shot, which implies vaccine costs per child between $14-$37. The best costing estimates for the delivery of the doses suggest around $9 per child.
3. ^
  “The leading malaria vaccine in development is the circumsporozoite protein (CSP)-based particle vaccine, RTS,S, which targets the pre-erythrocytic stage of Plasmodium falciparum infection. It induces modest levels of protective efficacy, thought to be mediated primarily by CSP-specific antibodies. We aimed to enhance vaccine efficacy by generating a more immunogenic CSP-based particle vaccine and therefore developed a next-generation RTS,S-like vaccine, called R21. The major improvement is that in contrast to RTS,S, R21 particles are formed from a single CSP-hepatitis B surface antigen (HBsAg) fusion protein, and this leads to a vaccine composed of a much higher proportion of CSP than in RTS,S.” Collins et al. 2017, “Abstract”

GiveWell 31 Aug 2022 16:53 UTC
24 points
0 ∶ 0
on: Red-teaming contest: demographics and power structures in EA
This is Miranda Kaplan, a communications associate at GiveWell. Thanks so much for publishing this; we really value critiques of our work, especially of our approach to moral weights, which is a particularly challenging feature of our analysis. I’ve shared this internally so the relevant people can take a thorough read through it, and we may share a response here, as we are able.

GiveWell 12 Sep 2022 18:58 UTC
20 points
0 ∶ 0
in reply to: MichaelPlant’s comment on: Announcing the Change Our Mind Contest for critiques of our cost-effectiveness analyses
Hi Michael—this is Isabel Arjmand, Special Projects Officer at GiveWell. Thank you for the feedback and for HLI’s critique of deworming, which played a role in inspiring this contest!
We designed this contest to incentivize critiques that are relatively straightforward for us to evaluate and particularly likely to change our mind about upcoming allocation decisions. This is our first time running a contest like this, so we wanted to keep the scope manageable. We may run future contests with different or broader prompts.
A bit more color on why we’re keeping the contest’s scope to our existing cost-effectiveness analyses:
1. We believe critiques of our existing cost-effectiveness analyses will be relatively straightforward for us to review, as opposed to broader critiques, such as those that suggest we take an entirely different approach to recommending giving opportunities. We anticipate that having this well-defined scope will make it easier for us to compare and give due consideration to all entries with our current research capacity (which we are hoping to expand!).
2. We’re getting ready to make some large decisions about how to allocate funding across the programs we currently support at the end of the year. This contest is designed to solicit the feedback that we think has the greatest potential to improve those upcoming decisions; excellent entries could meaningfully change how we allocate funds, leading to more lives saved or improved. Broader critiques or proposals for wholly new approaches would be unlikely to influence this year’s decisions, given how much vetting we put into our allocations and how little time remains before they are finalized.
Outside of this contest, we welcome feedback on all aspects of our work, and we’re glad to receive those at any time via email, as blog comments on our open threads, or here on the EA Forum.
We appreciate your continued engagement on subjective well-being, particularly the useful feedback you provided on our draft reports explaining why we’re not as optimistic about subjective well-being measures and Interpersonal Psychotherapy Groups as you are. We’re still planning to publish those reports, but we’re behind the timeline we originally laid out. Thanks for your patience with this!
What links here?
- Lorenzo Buonanno's comment on The winners of the Change Our Mind Contest—and some reflections by GiveWell (15 Dec 2022 22:28 UTC; 9 points)

GiveWell 6 Dec 2021 19:17 UTC
18 points
0 ∶ 0
in reply to: JoelMcGuire’s comment on: Donating money, buying happiness: new meta-analyses comparing the cost-effectiveness of cash transfers and psychotherapy in terms of subjective well-being
Thanks for the flag, Joel.
Brian, our team is working on our own reports on how we view interpersonal group therapy interventions and subjective well-being measures more generally. We expect to publish our reports within the next 3-6 months.
We have spoken to HLI about their work, and HLI has given us feedback on our reports. It’s been really helpful to discuss this topic with Michael, Joel, and the team at HLI. Their work has provided some updates to how we view this topic, even if we do not ultimately end up reaching the same conclusions.
We’re still looking into this area and some of the important questions HLI has raised. While we plan to provide a more detailed view once our reports are published, a few areas where we differ from HLI are below:
- The most concrete methodological difference between our approach and HLI’s is that we have a stronger view that there are larger intra-household spillovers for cash transfers than for therapy. HLI assumes any spillover effects to other household members are proportional across interventions—i.e., if a cash transfer benefits other household members’ subjective well-being x% as much as it benefits the recipient, the same is true for therapy. We have a strong prior that, since income is shared within households, other household members would benefit much more from a cash transfer than from therapy delivered to one of its members. If there are 4 to 5 members per household (roughly what we estimate for participants in GiveDirectly’s program) and there were no household multipliers from therapy, psychotherapy would be 2x-3x as cost-effective as cash transfers, taking HLI’s other assumptions as given. HLI has flagged this as an uncertainty and something they may look further into, and we would be interested to see what they find. It seems possible there are significant spillovers from therapy, but our current best guess is these would be much smaller than for cash.
- Another concrete difference is that we’re not sure comparing standard deviation effects across interventions is appropriate. HLI compares the effect on well-being in standard deviations (SDs) across cash transfer and therapy interventions. Our impression is that therapy interventions like StrongMinds target individuals with depression, who might be concentrated at the lower end of subjective well-being scales, to a greater extent than cash transfer interventions do. If this is the case, then the measure of well-being per SD may be lower for therapy than for cash transfer interventions. For example, a SD for psychotherapy may be a 0.5 on a 10-point scale of well-being, while the SD for cash transfers may be 1. HLI notes this as an uncertainty, and we’d be interested in more evidence on this, too.
- We put less weight on subjective well-being as a measure to compare different interventions. As HLI notes, subjective well-being measures could provide a common currency for comparing interventions like cash and therapy. We agree this is a useful perspective. However, we think there are some limitations to these measures, and give weight to other factors. I expect that GiveWell would find interventions that have benefits beyond subjective well-being more cost-effective than HLI would. As a result, even if a subjective well-being approach showed a program was more cost-effective than programs we’d expect to recommend marginal funding to, we wouldn’t necessarily make a recommendation based on that alone (though we would give it some weight).
- When we’ve looked at group therapy from perspectives other than comparing different interventions’ effects on subjective well-being measures, we find much lower cost-effectiveness than HLI is finding. We’ve considered a few different angles. First, we’ve looked at the effect of therapy under our current moral weights, which we use to trade off outcomes like increasing consumption, averting deaths, and averting morbidity. Under this approach and using the effect of depression on DALYs, we find similar cost-effectiveness between therapy and cash transfers. Second, we’ve taken a very shallow look at a recent trial that compares cash transfers with therapy head-to-head. This trial finds a much smaller effect of therapy vs. cash. Third, it seems intuitively surprising that a $1,000 cash transfer for a household in a low-income country would have a substantially smaller effect than 3-4 months of therapy (0.5 SD for cash transfers vs. 1.6 SD for therapy in HLI’s study). If the average GiveDirectly household has consumption of $2,000-$3,000, a $1,000 transfer amounts to a 30%-50% increase in baseline consumption for an entire household. We would guess that if we, for example, gave households the choice between 3-4 months of therapy and a $1,000 transfer, many more would choose the transfer. While the above approaches definitely have drawbacks and we don’t think we should put 100% weight on them either, the implications seem substantially different enough from HLI’s estimates that they give us some additional skepticism.
We still have a lot of uncertainty about how to compare different interventions like cash transfers and therapy, and making these comparisons is crucial to our decisions on what funding opportunities to recommend to our donors. As a result, we hope to continue to discuss this topic with individuals who have a differing view than us on our moral weights so that we can continue to refine our approach.
We look forward to engaging once we publish a fully vettable report. Until then, I hope this answers the immediate questions you have about where the views of GiveWell and HLI differ.
What links here?

GiveWell 17 Oct 2023 23:50 UTC
16 points
2 ∶ 0
on: A critical review of GiveWell’s 2022 cost-effectiveness model
Hi Froolow,

Thank you so much for this thoughtful review!

We’ve looked through the issues you identify as errors, and we believe that each of those is either not an error or is something we’ve deprioritized because addressing it would not change our grantmaking decisions. You can see our analysis here.

This only applies to the potential errors you pointed out, not your higher-level feedback or judgment calls. We’re grateful for your engagement here, and we are working on improving the architecture of our CEAs, in line with your higher-level feedback. We also recognize that the interpretability of our cost-effectiveness analysis is an area we need to improve, and we’re working on that as well.

Since the linked document was written for internal use, it hasn’t been fully vetted in the same way that what we publish on our website is. However, we thought it may be useful to share in its current form.

Best,

Andrew (GiveWell)

GiveWell 5 Jun 2023 15:55 UTC
16 points
2 ∶ 0
on: Beyond Cost-Effectiveness: Insights for Effective Altruism from Health Economics
Thanks for this thoughtful post, Tom. You’ve definitely raised some thoughts that have been on our mind recently, such as how GiveWell could systematically incorporate more external input into our grant-making process. We’ve taken some steps towards this, such as with a beneficiary survey in 2019, and seeking out more external experts, but we’d like to do more – it was really great to read through your recommendations.
We did want to clarify, however, that GiveWell’s approach to modeling cost-effectiveness and making grant recommendations is heavily context-specific. You’re right that we start with intervention-level analysis in order to get a rough sense of the cost-effectiveness of any given program. But, our next step is to modify our models with many charity- and context-specific data and only fund grant opportunities that are above our 10x bar. For example, when we’re considering making grants to Malaria Consortium’s SMC program, we assess funding opportunities at the country level, taking into consideration the differences in malaria prevalence, demographics (age distribution), mortality rates, program costs, and the spending we might expect from other actors in each setting. The result is that the same program may clear our cost-effectiveness bar in some locations and not in others. For a recent example, see this page about a grant we recommended in January for Malaria Consortium; we decided to extend funding for its SMC program in Nigeria, Burkina Faso, and Togo (where estimated cost-effectiveness was near or above our bar), but provide only exit funding for Chad (which was below our bar).
If there’s reason to expect variations within countries, we also build out our model at the subnational level. For example, in 2022 we updated our model of Malaria Consortium’s SMC program in Nigeria with state-level malaria prevalence and mortality data. While we don’t capture every variance one could expect, we’re trying to adjust for the major variances that could exist in different contexts (and which therefore could affect our bottom-line grantmaking decisions).
We realize that this nuance isn’t captured in our external-facing marketing, and we’re working on updates to our website to address this. We really appreciate your engagement with our work; we aspire to account for context-specific variables in our grantmaking and appreciate the push to consider this further and to make this aspect of our work clearer.

GiveWell 5 Jun 2023 19:06 UTC
15 points
0 ∶ 0
on: Taxing Tobacco: the intervention that got away (happy World No Tobacco Day)
Hi, J.T.,
As shown by the research conversation notes you link to, GiveWell lightly explored funding tobacco control policy advocacy in the past. Ultimately we decided to focus on other policy areas (alcohol policy, lead exposure, and self-harm from pesticide ingestion) that appeared more neglected. You can read more about our public health regulation research on this page—see “Cause areas we investigated at a shallow level and deprioritized,” and this spreadsheet, linked from footnote 24, which gives our estimates of how much is spent on tobacco policy (and other causes we looked into) per unit of burden.
Thank you for the thought-provoking post!
Best,
Miranda Kaplan
GiveWell Communications Associate

GiveWell 19 May 2023 20:00 UTC
15 points
2 ∶ 0
on: On Child Wasting, Mega-Charities, and Measurability Bias
Hi, Jesper,
Thank you for this post, and apologies that it took a while for us to respond!
We agree that more public information clarifying the value of donating to these large charities would be helpful. One thing that has changed about GiveWell since the 2011 blog post is that we now have a much larger staff and have gained more research experience, so we have more capacity to investigate the complicated questions that working with very large charities can bring up. We’re now more open to investigating opportunities within “mega-charities” than we were previously.
One factor that we consider whenever we make a grant is funging, or the possibility that a grant from GiveWell will cause other actors to allocate their funding differently. If a program gets money from GiveWell, another funder that would have supported that program might then decide to fund a different program that’s less cost-effective, reducing the impact of our funding. Or, the organization that runs the program could decide to move some of its unrestricted funding to another of its programs that’s less impactful. We would want to probe the possibility of the latter scenario as part of any investigation into a large organization that runs many programs.
We’ve spent a significant amount of time researching malnutrition treatment programs in the last few years, and made multiple grants, including to the large charity International Rescue Committee (IRC) and the smaller Alliance for International Medical Action (ALIMA). In late 2021, we published a blog post about why malnutrition treatment programs seemed extremely promising. But, although we did recommend grants for these programs, we have found it challenging to model their cost-effectiveness. In particular, we don’t have a clear sense from studies of how many deaths they prevent, due to ethical considerations limiting the research that can be done—it’s (justifiably) unethical to withhold malnutrition treatment, so it’s not possible to conduct a true randomized controlled trial of treatment vs. no treatment.
After conducting extensive internal research, plus hiring a couple of external experts to do their own analysis, we believe some malnutrition treatment programs are in the range of cost-effectiveness of programs we would consider directing funding to—i.e., similar to that of our top charities, which we estimate can save a life for roughly $3,500-$5,500. We still have major uncertainties about parts of our cost-effectiveness analyses, which we’re unlikely to resolve. But we think we may be able to reduce our uncertainty in other areas, and we’re moving forward with work on those aspects of our model. Simultaneously, we’re still investigating specific charities’ programs (in specific locations) as potential giving opportunities.
All that said, like you, we would be very surprised if the true cost to save a life (for any program, not just malnutrition treatment) were on the order of $210. Our cost per life saved estimates include all costs of running the program, including non-philanthropic costs (such as those covered by the government), and attempt to account for the other factors you mention, such as the fact that not all treated children would otherwise die from malnutrition and the likelihood that another funder would support this program if we didn’t.
If you’re curious to learn more, you can read a page about one of the grants we made to ALIMA here, and our most recent malnutrition treatment intervention report here.
I hope that’s helpful!
Best,
Miranda Kaplan
GiveWell Communications Associate

GiveWell 31 Aug 2022 15:05 UTC
15 points
0 ∶ 0
on: Adding Quantified Uncertainty to GiveWell’s Cost Effectiveness Analysis of the Against Malaria Foundation
This is Miranda Kaplan, a communications associate with GiveWell. Thanks so much for engaging with our cost-effectiveness model, we really appreciate it! This is on our research team’s radar, and they might reach out to you to seek clarification for anything they don’t understand or want to dig into more deeply.

GiveWell 1 Nov 2022 14:51 UTC
14 points
2 ∶ 0
on: A dozen doubts about GiveWell’s numbers
Thanks for your entry!

GiveWell 30 Aug 2022 14:14 UTC
14 points
1 ∶ 0
in reply to: Andrew Gimber’s comment on: Changes to our top charity criteria, and a new giving option
Hi, Andrew,
Apologies for the delay in responding!
It is a bit tricky to compare expected impact across the various funds. The tl;dr answer, without putting any real calculations into it, is that in practice, we don’t expect there to be large differences. But theoretically there could be, and if you can give unrestricted or less restricted (assuming you trust GiveWell), that’s probably better, as it allows us to deploy funding where it will be most impactful. Here are a few points to consider.
- Because a large portion of our funding is either technically unrestricted or flexible, we think that in practice, it’s unlikely that our grantmaking to top charities or non–top charity programs will be constrained by the proportion of funding we receive for the Maximum Impact vs. All Grants fund. A lot of our funding comes from Open Philanthropy as flexible funding intended for grantmaking (so, very similar to funding from the All Grants Fund); this typically has gone to a mix of top charities and other programs. We also receive enough unrestricted funding nowadays that some of it ends up getting granted out, due to the excess assets policy you mention, as well as our single-donor cap, which prevents one donor from providing too much of our operating support (more here). In 2021, the vast majority of our grants from unrestricted funding went to top charities, and indeed the majority of our grant funding in general goes to top charities—as mentioned in the above post, we think that the ratio this year will be about 3:1 (based on the pipeline of opportunities we’re looking at right now, not on any rules or proclivities).
- Though we will use the All Grants Fund to support some opportunities that are higher-expected-value than our top charities, we wouldn’t predict that the All Grants Fund will be systematically higher in expected value than the Maximum Impact Fund. We are using the same cost-effectiveness bar for grants from both funds, and many of our grants to non–top charity programs are similar in cost-effectiveness to—not greatly more cost-effective than—our top charities.
- Three, with all of the above said, we agree with your suggestion that the most impactful way to give to GiveWell, assuming you trust our decision-making, is unrestricted. There could be a world in which we get way, way more Maximum Impact Fund donations than All Grants Fund donations, and because we’re compelled to spend the former on top charities, we end up funding still-excellent-but-less-cost-effective opportunities from those charities (say, 8x cash rather than 10x) and have to raise our bar for granting to other programs to, say, 12x because we’re flexible funding–constrained. We don’t think that’s going to happen because in reality, as noted above, a lot of our funding is flexible. But giving to us unrestricted (or restricted to grantmaking only, through the All Grants Fund) means that we can shift funding around as needed such that we’re maximizing the overall impact of our portfolio. The Maximum Impact Fund, however, remains our top recommendation for donors who want to be assured that their donation goes toward high-impact/high-confidence opportunities, versus the riskier options that might be funded via the All Grants Fund.
I hope that’s somewhat helpful!
Best,
Miranda

GiveWell 19 May 2022 16:31 UTC
13 points
0 ∶ 0
in reply to: WilliamKiely’s comment on: IPTi for malaria: a promising intervention with likely room to scale
Hi, William,
Thanks for your patience and flexibility! Though you said you weren’t necessarily expecting a response, we do try to answer comments promptly, and I apologize that it took a while to get back to you on this.
This answer may not cover every question you raised, but I think it will get at most of it.
- On the timing of the two IPTi scoping grants:
  - We believe the concurrence of these grants is related to a general growing interest in IPTi as a promising tool with which to combat malaria, especially as progress against the disease has plateaued in recent years (see WHO’s 2021 World Malaria Report for more on this) and infants remain vulnerable. Other global actors are also making investments in IPTi; for example, Unitaid has funded the IPTi+ implementation research project, which aims to test out delivery models for the intervention while administering IPTi at scale. WHO is also set to release updated, less restrictive guidelines for administering IPTi, which should make it easier for some countries to adopt it. (We describe the current WHO guidelines in more detail, and discuss other factors that may have inhibited more widespread adoption, here.)
  - We did speak with Gates to coordinate before we recommended our own scoping grant to Malaria Consortium and PATH, to make sure that we weren’t duplicating efforts. The goals of the two grants are a bit different. The Gates-funded study is aimed at learning more about the feasibility and effectiveness of IPTi in real-world settings (as opposed to trial settings), and generating evidence to support IPTi’s adoption among policymakers in Nigeria. Our grant was less focused on evidence generation—we funded PATH and Malaria Consortium to gather information that will inform our cost-effectiveness analysis, and to put together a proposal for scaled-up IPTi implementation in Nigeria and DRC, including costs, timeline, and delivery platforms (more information here).
- On the possibility of Gates funding a larger-scale IPTi program:
  - We’ve talked to Gates and other large funders about whether they’re likely to fund IPTi at scale soon, and it seems most of them are waiting for the results of the Gates and Unitaid research before deciding whether to do so. That would mean waiting on funding implementation until at least 2024 or 2025. One of the goals of our scoping grant is to learn whether there are opportunities to accelerate implementation of IPTi even before those other studies are concluded (see our thinking under “Waiting on more evidence generation vs. moving ahead now” in the “risks and reservations” section of the grant page).
  - We also get the impression that some of the big funders in the malaria space have institutional mandates that wouldn’t allow them to support technical assistance to help scale up IPTi, which is what we’re interested in potentially funding. (These funders might instead support, for example, ongoing drug procurement for IPTi once the program is already being delivered at scale.)
- On coordinating with Gates or WHO to get more funding: One of the factors we consider before recommending any grant is the possibility of leverage (i.e., our grant causes other funders to invest more money in the same intervention) or funging (our grant means other funders decide not to invest in the same intervention). Leverage is a good thing for the world, since we believe most other funders would, counterfactually, fund something less cost-effective than the opportunities we’re interested in. One big question for us as we consider future support of IPTi is whether our funding would actually be a deciding factor in other funders supporting IPTi, since we know that interest in this intervention is already growing—it might be that they would have funded IPTi anyway, without our grant. We have been speaking with Gates, WHO, Unitaid, President’s Malaria Initiative, and other big actors to learn what barriers they have to funding IPTi implementation, and what we could do, beyond the ongoing evidence generation efforts, to help crowd in more funds. We’ll continue to have conversations with them to get a sense of how our funding affects their decision-making.
- On other interventions that are recommended by WHO but not yet scaled: We do think this, or something like this, could be a promising avenue. We’re aware of a few other interventions for which the evidence seems strong, but there are no NGOs currently implementing the program, and we’re looking into how we might find or create funding opportunities for those interventions.
I hope that helps!
Best,
Miranda

GiveWell 31 Dec 2022 3:55 UTC
12 points
1 ∶ 2
on: Givewell, and its hiring process, needs serious reform
Hi, Bob,
We’re very sorry to hear that you had a bad experience! We take feedback like this seriously and have passed it on to the senior staff in charge of research hiring.
Responding to all applicants in as timely a manner as we’d like has been challenging, in large part due to understaffing on the research team. But, we are working to improve on this. We invite others who may be considering applying for one of our research positions to review our FAQs for more information about the hiring process, including the typical timeline.
Thank you for sharing your concerns!

Best,
Miranda Kaplan, GiveWell Communications Associate

GiveWell 5 Oct 2022 15:35 UTC
11 points
0 ∶ 0
on: Winners of the EA Criticism and Red Teaming Contest
This is Miranda Kaplan, a communications associate at GiveWell. We want to thank everyone who submitted reviews of GiveWell’s analyses as contest entries! It’s extremely valuable to us when people outside our organization engage deeply and critically with our work. We will always do our best to consider and respond to such critiques, though, given other pressures on staff time, we may not be able to publish a full response.

Although CEA’s done it in the post above, we’ll take this opportunity to again plug our Change Our Mind Contest, which closes on October 31. Entries submitted to the Criticism and Red Teaming Contest are eligible, as long as they meet all other Change Our Mind Contest requirements (described here). We look forward to reading your work and thank you in advance for your participation!

GiveWell 13 Apr 2023 19:38 UTC
10 points
3 ∶ 0
in reply to: Robi Rahman’s comment on: How much funding does GiveWell expect to raise through 2025?
Hi Robi, thanks for the question! This is Isabel Arjmand from GiveWell – I wrote the above post, and I run the model we use to forecast funds raised.
The uncertainty primarily stems from the fact that Open Philanthropy is in the process of reconsidering how it allocates funding across cause areas.
We’ve spoken with Open Philanthropy about their plans, and our understanding is that the amount of funding Open Philanthropy might give to GiveWell in the future is highly uncertain at this point. Our impression is that it could be hundreds of millions of dollars annually or could be very little. While the funding we may receive from Open Philanthropy in the future is uncertain, Open Philanthropy confirmed in conversation with us that its confidence in GiveWell’s research hasn’t changed.
Based on our qualitative impressions of the uncertainty and of the range of plausible outcomes, we created a quantitative model for what Open Philanthropy might give in the future, which is what feeds into our forecasts.
We expect to know more later this year once Open Philanthropy has wrapped up this project.
Please let us know if you have any more questions!