Hi Tony, This is Mark again. Thanks for the nudge on this monitoring point!
We agree that real-time monitoring of critical activities is likely to be a cost-effective investment. For example, New Incentives (a program that provides conditional cash transfers for immunizations working with Northern Nigeria health clinics) tracks and shares with us data on cold chain failures, stockouts, rollout delays, security incidents, fraud, double treatment of children, and retention rates (see more information in this intervention report). We think this type of monitoring is a good investment for New Incentives and want others to do something similar, because it:
Helps grantees catch and quickly address issues that would have affected impact (e.g., replacing skunked vaccines that wouldn’t have worked) and identify program improvements.
Helps GiveWell triangulate coverage data of uncertain quality (e.g., if New Incentives clinics are usually stocked out of vaccines, that makes it unlikely that New Incentives is increasing vaccination coverage no matter what coverage surveys say)
Informs GiveWell’s understanding of why some programs work, while others do not. For example, if stockouts caused New Incentives to not work, then we might want to fund programs to increase vaccine stock or improve allocation of vaccines in the future.
We’re still trying to understand the trade-offs between grantees doing this internally vs. externally, and we expect to learn as we go. It makes sense to me that it is usually most efficient and effective for grantees to do the core of this monitoring internally since we want them to be able to address these issues in real-time (point 1) rather than waiting for a report from an external firm. In cases where GiveWell is relying on the accuracy of this data for our decision-making (points 2 and 3), I think it makes sense to double-check data accuracy through external data quality audits and/or interviews with local stakeholders.
The same goes with standardizing. We think it’s good to have a minimum standard, but we guess there’s also variation in what’s feasible or makes sense across grantees and expect to figure that out with them. We’ve standardized monitoring and reporting within a few program areas, such as across all our recent water chlorination grants, in collaboration with our grantees (e.g., agreeing on consistent standards on what grantees are expected to track, at what frequency, and how findings should be acted on). However, we haven’t developed monitoring standards that apply across all program areas. I think there could be value in having broader principles, but, so far, we’ve prioritized addressing concrete issues we’ve identified rather than developing these broader principles. We are planning to publish more on the details of these program area-specific monitoring and reporting standards in the near-future.
Hi Tony, This is Mark again. Thanks for the nudge on this monitoring point!
We agree that real-time monitoring of critical activities is likely to be a cost-effective investment. For example, New Incentives (a program that provides conditional cash transfers for immunizations working with Northern Nigeria health clinics) tracks and shares with us data on cold chain failures, stockouts, rollout delays, security incidents, fraud, double treatment of children, and retention rates (see more information in this intervention report). We think this type of monitoring is a good investment for New Incentives and want others to do something similar, because it:
Helps grantees catch and quickly address issues that would have affected impact (e.g., replacing skunked vaccines that wouldn’t have worked) and identify program improvements.
Helps GiveWell triangulate coverage data of uncertain quality (e.g., if New Incentives clinics are usually stocked out of vaccines, that makes it unlikely that New Incentives is increasing vaccination coverage no matter what coverage surveys say)
Informs GiveWell’s understanding of why some programs work, while others do not. For example, if stockouts caused New Incentives to not work, then we might want to fund programs to increase vaccine stock or improve allocation of vaccines in the future.
We’re still trying to understand the trade-offs between grantees doing this internally vs. externally, and we expect to learn as we go. It makes sense to me that it is usually most efficient and effective for grantees to do the core of this monitoring internally since we want them to be able to address these issues in real-time (point 1) rather than waiting for a report from an external firm. In cases where GiveWell is relying on the accuracy of this data for our decision-making (points 2 and 3), I think it makes sense to double-check data accuracy through external data quality audits and/or interviews with local stakeholders.
The same goes with standardizing. We think it’s good to have a minimum standard, but we guess there’s also variation in what’s feasible or makes sense across grantees and expect to figure that out with them. We’ve standardized monitoring and reporting within a few program areas, such as across all our recent water chlorination grants, in collaboration with our grantees (e.g., agreeing on consistent standards on what grantees are expected to track, at what frequency, and how findings should be acted on). However, we haven’t developed monitoring standards that apply across all program areas. I think there could be value in having broader principles, but, so far, we’ve prioritized addressing concrete issues we’ve identified rather than developing these broader principles. We are planning to publish more on the details of these program area-specific monitoring and reporting standards in the near-future.