Hi Michael,
Thank you for raising these points! We’ve shared your thoughts with our research team for future consideration.
If you have any other questions or concerns about our work, please feel free to reach out to us!
Roman Guglielmo
Donor Relations Associate
GiveWell
Hi!
Thanks for this comment. We are interested in potentially funding family planning programs, including Family Empowerment Media. Our research and grant decisions are independent of our outreach and fundraising plans. To clarify and add detail on some of the points above:
GiveWell recommended a $500,000 grant to Family Empowerment Media in March 2023. This funding would support a planned randomized controlled trial (RCT) of its program. We haven’t yet published about this grant, but we plan to soon. (Our grant page publication often lags our funding decisions considerably, though this is something we’re working to improve!)
We believe that FEM or other types of family planning–related programs may be very cost-effective—our current rough estimate for FEM’s program (not the RCT of the program) puts it at 18-28 times as cost-effective as unconditional cash transfers—but we have a high degree of uncertainty about some of our inputs and expect that the results of FEM’s RCT would help us address some of our uncertainties. We are also continuing to assess how to prioritize family planning programs among the many other promising programs we’re investigating.
I hope that’s helpful!
Best,
Miranda Kaplan
GiveWell Communications Associate
Hi, Hakon,
We were very excited to learn of this trial, a much-needed step toward reducing deaths from tuberculosis. We applaud Gates and Wellcome for this sizable commitment of resources, which is, as Nick points out, necessary to gain adequate information about the vaccine’s efficacy, but beyond the reach of most funders.
We have investigated TB-related funding opportunities, and we remain very open to funding either programs or research. However, we have several significant uncertainties about the programs we’ve explored so far and have not yet funded any at scale.
We are working on a report summarizing the evidence for mass TB screen-and-treat programs, which will provide more detail on our views specific to that type of program.
Best, Miranda
Hi, Nick,
Yes, to clarify, MiracleFeet is selecting the on-the-ground evaluator that will conduct the monitoring in each location, although GiveWell will lightly vet MiracleFeet’s choices and meet with each evaluator.
The primary purpose of the monitoring grant is to understand how many children are treated for clubfoot both with and without MiracleFeet’s support. So, although MiracleFeet has records of children treated through facilities it’s supported, we also want an assessment of baseline treatment coverage before MiracleFeet launches its program (or expands it, in the case of the Philippines). We do plan to incorporate some form of data audit as part of endline activities; we’ll work out the details of that at a later date.
Thanks again for your interest in this and for taking the time to ask questions!
Miranda
Hi, Nick,
Thanks for your comments, your insight into this grant, and your support!
We do expect to get input from local hospital staff on existing treatment coverage through the baseline surveys. The monitoring grant will fund the creation of a sampling frame that includes both public and private health facilities, which we think will yield more complete data than contacting hospitals through our partners.
We agree that potential bias from external evaluators is a risk for the reasons you’ve mentioned. While we won’t be involved in the selection of evaluators, we plan to do the following to mitigate that risk:
Meet with representatives of all the firms and vet them at a high level so we can identify red (and green) flags.
Work closely with both MiracleFeet and the external evaluators throughout the process, so that we can ask questions about and provide input on their research strategies along the way.
We don’t think this will completely eliminate uncertainty about the quality of monitoring results, but we expect it will help. We also think there is some value to be gained from working with evaluators who have a strong familiarity with the local context.
I hope that’s helpful!
Best,
Miranda
Hi, J.T.,
Sorry for the delay in getting back to you!
In an ideal world, we would have included financial burden for tobacco in these rough calculations (which were mostly intended to narrow down what we’d focus on in our initial foray into public health regulation grants). But essentially, the last line of your response is the answer: because we value children’s lives so highly, we expected that incorporating the financial burden of tobacco use would not be enough for us to prioritize it over lead. Lead specifically affects young children, and the gap in estimated spend per unit of burden ($0.07 for lead versus $0.24 for tobacco) is large. If we were doing a deeper investigation of tobacco policy as a possible funding area, we would aim to more thoroughly account for its harms.
Best,
Miranda
Hi, J.T.,
As shown by the research conversation notes you link to, GiveWell lightly explored funding tobacco control policy advocacy in the past. Ultimately we decided to focus on other policy areas (alcohol policy, lead exposure, and self-harm from pesticide ingestion) that appeared more neglected. You can read more about our public health regulation research on this page—see “Cause areas we investigated at a shallow level and deprioritized,” and this spreadsheet, linked from footnote 24, which gives our estimates of how much is spent on tobacco policy (and other causes we looked into) per unit of burden.
Thank you for the thought-provoking post!
Best,
Miranda Kaplan
GiveWell Communications Associate
Thanks for this thoughtful post, Tom. You’ve definitely raised some thoughts that have been on our mind recently, such as how GiveWell could systematically incorporate more external input into our grant-making process. We’ve taken some steps towards this, such as with a beneficiary survey in 2019, and seeking out more external experts, but we’d like to do more – it was really great to read through your recommendations.
We did want to clarify, however, that GiveWell’s approach to modeling cost-effectiveness and making grant recommendations is heavily context-specific. You’re right that we start with intervention-level analysis in order to get a rough sense of the cost-effectiveness of any given program. But, our next step is to modify our models with many charity- and context-specific data and only fund grant opportunities that are above our 10x bar. For example, when we’re considering making grants to Malaria Consortium’s SMC program, we assess funding opportunities at the country level, taking into consideration the differences in malaria prevalence, demographics (age distribution), mortality rates, program costs, and the spending we might expect from other actors in each setting. The result is that the same program may clear our cost-effectiveness bar in some locations and not in others. For a recent example, see this page about a grant we recommended in January for Malaria Consortium; we decided to extend funding for its SMC program in Nigeria, Burkina Faso, and Togo (where estimated cost-effectiveness was near or above our bar), but provide only exit funding for Chad (which was below our bar).
If there’s reason to expect variations within countries, we also build out our model at the subnational level. For example, in 2022 we updated our model of Malaria Consortium’s SMC program in Nigeria with state-level malaria prevalence and mortality data. While we don’t capture every variance one could expect, we’re trying to adjust for the major variances that could exist in different contexts (and which therefore could affect our bottom-line grantmaking decisions).
We realize that this nuance isn’t captured in our external-facing marketing, and we’re working on updates to our website to address this. We really appreciate your engagement with our work; we aspire to account for context-specific variables in our grantmaking and appreciate the push to consider this further and to make this aspect of our work clearer.
Hi, Peter! So sorry I missed this question earlier and have been delayed in responding.
We’ve described in the above post what we know about R21 so far (see the second and third paragraphs from the end). To summarize, R21 has been shown to have high efficacy in protecting against malaria, but it’s unclear to us so far how generalizable those results will be. R21 is also reportedly less complicated to manufacture, which could be helpful as demand for malaria vaccine is expected to outstrip supply—but we can’t independently verify this. We’ll keep monitoring the literature on R21, and we’ll consider any funding opportunities as they come up.
Best,
Miranda Kaplan
GiveWell Communications Associate
Hi, Nick,
Thanks for your comment! Apologies that it took a while to respond to this.
Re: how much funding is needed to successfully roll out the vaccine, we’ve provided a budget breakdown on the grant page. The majority of this funding is going toward training and other activities needed to distribute the vaccine, vaccine-related supplies, and shipping and handling for the doses donated by GSK. Only about $1.8 million of the total, or less than half, is going toward the costs of having PATH and WHO provide technical support for this project.
For every grant opportunity we evaluate, we do consider the likelihood that another funder will step in to cover the costs absent our support. In a conversation with PATH and WHO, we learned that there were no other suitable candidates for funding this rollout of RTS,S to comparator areas, though we didn’t independently verify this.
As for whether the governments of Kenya, Ghana, and Malawi could successfully speed up the implementation of RTS,S without NGO/WHO technical support, this is a subjective assessment. We frequently hear from NGOs that the governments where programs we fund operate tend to have many competing priorities, so progress on projects like this can be slow. The theory is that providing dedicated funding (and with it, human capacity) for a single project can accelerate the timeline of results. We try to confirm whether this is right by talking to other relevant actors, including government officials themselves.
It would be interesting to try out what you suggest—giving the funding directly to a country government to see if they could achieve the same results without technical assistance—but because there are so many country-specific factors that inform the success of an intervention, we think it’d be hard to tell if a slower vaccine rollout in a given country was due to lack of technical assistance or some other factor.
I hope that’s helpful!
Best,
Miranda Kaplan
GiveWell Communications Associate
Hi, Jesper,
Thank you for this post, and apologies that it took a while for us to respond!
We agree that more public information clarifying the value of donating to these large charities would be helpful. One thing that has changed about GiveWell since the 2011 blog post is that we now have a much larger staff and have gained more research experience, so we have more capacity to investigate the complicated questions that working with very large charities can bring up. We’re now more open to investigating opportunities within “mega-charities” than we were previously.
One factor that we consider whenever we make a grant is funging, or the possibility that a grant from GiveWell will cause other actors to allocate their funding differently. If a program gets money from GiveWell, another funder that would have supported that program might then decide to fund a different program that’s less cost-effective, reducing the impact of our funding. Or, the organization that runs the program could decide to move some of its unrestricted funding to another of its programs that’s less impactful. We would want to probe the possibility of the latter scenario as part of any investigation into a large organization that runs many programs.
We’ve spent a significant amount of time researching malnutrition treatment programs in the last few years, and made multiple grants, including to the large charity International Rescue Committee (IRC) and the smaller Alliance for International Medical Action (ALIMA). In late 2021, we published a blog post about why malnutrition treatment programs seemed extremely promising. But, although we did recommend grants for these programs, we have found it challenging to model their cost-effectiveness. In particular, we don’t have a clear sense from studies of how many deaths they prevent, due to ethical considerations limiting the research that can be done—it’s (justifiably) unethical to withhold malnutrition treatment, so it’s not possible to conduct a true randomized controlled trial of treatment vs. no treatment.
After conducting extensive internal research, plus hiring a couple of external experts to do their own analysis, we believe some malnutrition treatment programs are in the range of cost-effectiveness of programs we would consider directing funding to—i.e., similar to that of our top charities, which we estimate can save a life for roughly $3,500-$5,500. We still have major uncertainties about parts of our cost-effectiveness analyses, which we’re unlikely to resolve. But we think we may be able to reduce our uncertainty in other areas, and we’re moving forward with work on those aspects of our model. Simultaneously, we’re still investigating specific charities’ programs (in specific locations) as potential giving opportunities.
All that said, like you, we would be very surprised if the true cost to save a life (for any program, not just malnutrition treatment) were on the order of $210. Our cost per life saved estimates include all costs of running the program, including non-philanthropic costs (such as those covered by the government), and attempt to account for the other factors you mention, such as the fact that not all treated children would otherwise die from malnutrition and the likelihood that another funder would support this program if we didn’t.
If you’re curious to learn more, you can read a page about one of the grants we made to ALIMA here, and our most recent malnutrition treatment intervention report here.
I hope that’s helpful!
Best,
Miranda Kaplan
GiveWell Communications Associate
GiveWell is hiring for a number of different roles: https://www.givewell.org/about/jobs. We offer flexible hours, benefits, and the ability to work remotely. You may want to check out the content editor position in particular, if you’re looking for flexibility. Good luck!
Hi, Joseph! I sent you a PM with a contact who can advise you/your parents according to the specifics of their situation.
Best, Miranda Kaplan GiveWell Communications Associate
Hi Robi, thanks for the question! This is Isabel Arjmand from GiveWell – I wrote the above post, and I run the model we use to forecast funds raised.
The uncertainty primarily stems from the fact that Open Philanthropy is in the process of reconsidering how it allocates funding across cause areas.
We’ve spoken with Open Philanthropy about their plans, and our understanding is that the amount of funding Open Philanthropy might give to GiveWell in the future is highly uncertain at this point. Our impression is that it could be hundreds of millions of dollars annually or could be very little. While the funding we may receive from Open Philanthropy in the future is uncertain, Open Philanthropy confirmed in conversation with us that its confidence in GiveWell’s research hasn’t changed.
Based on our qualitative impressions of the uncertainty and of the range of plausible outcomes, we created a quantitative model for what Open Philanthropy might give in the future, which is what feeds into our forecasts.
We expect to know more later this year once Open Philanthropy has wrapped up this project.
Please let us know if you have any more questions!
Hi, Joel,
Alex here, responding to your comment. Thank you for taking the time to give us this feedback!
In response to some of your specific points:
You’re right that we should have characterized the results from Lång and Nystedt (2018) as mixed rather than positive. Thanks for pointing out that mistake. We will update the spreadsheet so that study is correctly color-coded, and update the relevant part of the post. With this adjustment, among the studies we looked at, 3 suggest decreasing effects over time, 2 suggest increasing effects over time, and 5 show mixed effects. This still doesn’t seem like it adds up to strong evidence for either increasing or decreasing effects, so my prior of a flat effect over time remains the same.
We excluded Duflo et al. 2021 because it didn’t appear to include much about life cycle impacts on income from the intervention. It does report some increases in income for women in the treatment group between 2019 and 2020. However, I’d be reluctant to interpret that as evidence for increases over adulthood, because it represents only one year and because it compares pre-COVID results with results during COVID, which means other factors are probably at play.
That said, I agree that a more in-depth analysis might lead to a different prior for how we should expect early-life health interventions to affect income over the life cycle. We didn’t prioritize an in-depth analysis for this adjustment, but we would be open to more work to create a better-informed prior of deworming’s income effects over time. This would require deeper engagement with the studies we looked at to better understand their methodologies, relevance to deworming, and other factors. At the moment, it’s not a high-priority project for GiveWell staff, but we’re considering an external partnership to explore this further. We imagine that having a better grasp on how income effects change over time could inform our analysis not just of deworming but also of other programs we support, including vitamin A supplementation and seasonal malaria chemoprevention.
We’ll continue to share here if more work on this leads us to further updates.
Best,
Alex
I’m glad you found this response helpful!
We expect to wait until the results of FEM’s RCT are available before deciding whether to recommend funding for the program itself; we do have some remaining uncertainties, and it’s possible that additional work could lead to significant changes in our cost-effectiveness estimate. We also plan to continue our own research on family planning programs in general, which may affect our funding decisions in this area.
As our investigations progress, we’ll share updates on our website. Please feel free to reach out directly if you have questions in the meantime!
Best,
Roman Guglielmo
Donor Relations Associate
GiveWell