Hi Tony, This is Mark again. Thanks for the nudge on this monitoring point!
We agree that real-time monitoring of critical activities is likely to be a cost-effective investment. For example, New Incentives (a program that provides conditional cash transfers for immunizations working with Northern Nigeria health clinics) tracks and shares with us data on cold chain failures, stockouts, rollout delays, security incidents, fraud, double treatment of children, and retention rates (see more information in this intervention report). We think this type of monitoring is a good investment for New Incentives and want others to do something similar, because it:
Helps grantees catch and quickly address issues that would have affected impact (e.g., replacing skunked vaccines that wouldn’t have worked) and identify program improvements.
Helps GiveWell triangulate coverage data of uncertain quality (e.g., if New Incentives clinics are usually stocked out of vaccines, that makes it unlikely that New Incentives is increasing vaccination coverage no matter what coverage surveys say)
Informs GiveWell’s understanding of why some programs work, while others do not. For example, if stockouts caused New Incentives to not work, then we might want to fund programs to increase vaccine stock or improve allocation of vaccines in the future.
We’re still trying to understand the trade-offs between grantees doing this internally vs. externally, and we expect to learn as we go. It makes sense to me that it is usually most efficient and effective for grantees to do the core of this monitoring internally since we want them to be able to address these issues in real-time (point 1) rather than waiting for a report from an external firm. In cases where GiveWell is relying on the accuracy of this data for our decision-making (points 2 and 3), I think it makes sense to double-check data accuracy through external data quality audits and/or interviews with local stakeholders.
The same goes with standardizing. We think it’s good to have a minimum standard, but we guess there’s also variation in what’s feasible or makes sense across grantees and expect to figure that out with them. We’ve standardized monitoring and reporting within a few program areas, such as across all our recent water chlorination grants, in collaboration with our grantees (e.g., agreeing on consistent standards on what grantees are expected to track, at what frequency, and how findings should be acted on). However, we haven’t developed monitoring standards that apply across all program areas. I think there could be value in having broader principles, but, so far, we’ve prioritized addressing concrete issues we’ve identified rather than developing these broader principles. We are planning to publish more on the details of these program area-specific monitoring and reporting standards in the near-future.
GiveWell Updates
Learn MoreGiveWell is launching a new RFI to expand and strengthen our malaria grantmaking in Africa and help our donors make a greater impact. Expressions of interest can be submitted through one of two tracks, the first for malaria chemoprevention and vector control pilot programs and the second for research and evaluation. Submissions are due June 24.
Listen Here: Evaluating and Funding a New Kind of GrantMost of GiveWell’s grantmaking focuses on programs that reduce child mortality, but our growing research capacity has expanded what we’re able to evaluate and fund, including highly cost-effective programs that meaningfully improve quality of life. In a recent podcast episode, GiveWell CEO and co-founder Elie Hassenfeld speaks with Program Officer Meika Ball about our grant to MiracleFeet—an organization that expands access to clubfoot treatment—and her recent site visit to see the program in action in Côte d’Ivoire.
Register HereSave the date for our upcoming webinar, Looking Back to Give Better: How GiveWell Evaluates Its Grantmaking, on Tuesday, June 9. GiveWell co-founder and CEO Elie Hassenfeld will moderate a conversation with Program Directors Alex Cohen and Julie Faller about how we evaluate whether a grant achieved the impact we initially estimated—and how we use those findings to make better impact estimates over time.
Listen Here: Scrutinizing One of Our Longest-Funded ProgramsVitamin A supplementation has one of the strongest evidence bases of any program we’ve evaluated. But when you dig in, the evidence is more complicated than it looks. GiveWell Senior Researcher Stephan Guyenet joins CEO and co-founder Elie Hassenfeld on the GiveWell Conversations podcast to talk about the evidence for vitamin A supplementation, the hard questions that remain, and how our expanded research capacity is helping us go deeper so we can direct funding more effectively.