Hi Mark — thank you for the detailed response and for sharing the steps GiveWell is already taking on monitoring implementation fidelity. All of this is very encouraging, particularly the moves toward real-time identification and resolution of issues at the grantee level.
One area I’d love to hear more about is the following:
“Ensuring grantees are adequately monitoring critical activities. We’ve made it standard practice to identify the critical activities required for the desired impact and to ensure grantees have plans for monitoring these in real-time and addressing issues quickly.”
I think this is one of the most important levers available. Internal grantee monitoring, when done well, is likely to be the fastest and most cost-effective feedback mechanism. External evaluation processes generate valuable information, but they are almost always slower and more resource-intensive. Beyond improving grantmaking decisions directly, I’d expect this to have an indirect benefit: making M&E expectations explicit and visible tends to strengthen internal M&E culture within organisations over time.
What I’d be curious about is whether GiveWell has considered moving toward some degree of transparent standardisation of what “monitoring critical activities” actually looks like in practice — for instance, a shared framework defining what grantees are expected to track, at what frequency, and how findings should be acted on.
Is this something GiveWell is actively developing, and if so, is there scope for that framework to be shared more publicly?
Hi Tony, This is Mark again. Thanks for the nudge on this monitoring point!
We agree that real-time monitoring of critical activities is likely to be a cost-effective investment. For example, New Incentives (a program that provides conditional cash transfers for immunizations working with Northern Nigeria health clinics) tracks and shares with us data on cold chain failures, stockouts, rollout delays, security incidents, fraud, double treatment of children, and retention rates (see more information in this intervention report). We think this type of monitoring is a good investment for New Incentives and want others to do something similar, because it:
Helps grantees catch and quickly address issues that would have affected impact (e.g., replacing skunked vaccines that wouldn’t have worked) and identify program improvements.
Helps GiveWell triangulate coverage data of uncertain quality (e.g., if New Incentives clinics are usually stocked out of vaccines, that makes it unlikely that New Incentives is increasing vaccination coverage no matter what coverage surveys say)
Informs GiveWell’s understanding of why some programs work, while others do not. For example, if stockouts caused New Incentives to not work, then we might want to fund programs to increase vaccine stock or improve allocation of vaccines in the future.
We’re still trying to understand the trade-offs between grantees doing this internally vs. externally, and we expect to learn as we go. It makes sense to me that it is usually most efficient and effective for grantees to do the core of this monitoring internally since we want them to be able to address these issues in real-time (point 1) rather than waiting for a report from an external firm. In cases where GiveWell is relying on the accuracy of this data for our decision-making (points 2 and 3), I think it makes sense to double-check data accuracy through external data quality audits and/or interviews with local stakeholders.
The same goes with standardizing. We think it’s good to have a minimum standard, but we guess there’s also variation in what’s feasible or makes sense across grantees and expect to figure that out with them. We’ve standardized monitoring and reporting within a few program areas, such as across all our recent water chlorination grants, in collaboration with our grantees (e.g., agreeing on consistent standards on what grantees are expected to track, at what frequency, and how findings should be acted on). However, we haven’t developed monitoring standards that apply across all program areas. I think there could be value in having broader principles, but, so far, we’ve prioritized addressing concrete issues we’ve identified rather than developing these broader principles. We are planning to publish more on the details of these program area-specific monitoring and reporting standards in the near-future.
Hi Mark — thank you for the detailed response and for sharing the steps GiveWell is already taking on monitoring implementation fidelity. All of this is very encouraging, particularly the moves toward real-time identification and resolution of issues at the grantee level.
One area I’d love to hear more about is the following:
I think this is one of the most important levers available. Internal grantee monitoring, when done well, is likely to be the fastest and most cost-effective feedback mechanism. External evaluation processes generate valuable information, but they are almost always slower and more resource-intensive. Beyond improving grantmaking decisions directly, I’d expect this to have an indirect benefit: making M&E expectations explicit and visible tends to strengthen internal M&E culture within organisations over time.
What I’d be curious about is whether GiveWell has considered moving toward some degree of transparent standardisation of what “monitoring critical activities” actually looks like in practice — for instance, a shared framework defining what grantees are expected to track, at what frequency, and how findings should be acted on.
Is this something GiveWell is actively developing, and if so, is there scope for that framework to be shared more publicly?
Hi Tony, This is Mark again. Thanks for the nudge on this monitoring point!
We agree that real-time monitoring of critical activities is likely to be a cost-effective investment. For example, New Incentives (a program that provides conditional cash transfers for immunizations working with Northern Nigeria health clinics) tracks and shares with us data on cold chain failures, stockouts, rollout delays, security incidents, fraud, double treatment of children, and retention rates (see more information in this intervention report). We think this type of monitoring is a good investment for New Incentives and want others to do something similar, because it:
Helps grantees catch and quickly address issues that would have affected impact (e.g., replacing skunked vaccines that wouldn’t have worked) and identify program improvements.
Helps GiveWell triangulate coverage data of uncertain quality (e.g., if New Incentives clinics are usually stocked out of vaccines, that makes it unlikely that New Incentives is increasing vaccination coverage no matter what coverage surveys say)
Informs GiveWell’s understanding of why some programs work, while others do not. For example, if stockouts caused New Incentives to not work, then we might want to fund programs to increase vaccine stock or improve allocation of vaccines in the future.
We’re still trying to understand the trade-offs between grantees doing this internally vs. externally, and we expect to learn as we go. It makes sense to me that it is usually most efficient and effective for grantees to do the core of this monitoring internally since we want them to be able to address these issues in real-time (point 1) rather than waiting for a report from an external firm. In cases where GiveWell is relying on the accuracy of this data for our decision-making (points 2 and 3), I think it makes sense to double-check data accuracy through external data quality audits and/or interviews with local stakeholders.
The same goes with standardizing. We think it’s good to have a minimum standard, but we guess there’s also variation in what’s feasible or makes sense across grantees and expect to figure that out with them. We’ve standardized monitoring and reporting within a few program areas, such as across all our recent water chlorination grants, in collaboration with our grantees (e.g., agreeing on consistent standards on what grantees are expected to track, at what frequency, and how findings should be acted on). However, we haven’t developed monitoring standards that apply across all program areas. I think there could be value in having broader principles, but, so far, we’ve prioritized addressing concrete issues we’ve identified rather than developing these broader principles. We are planning to publish more on the details of these program area-specific monitoring and reporting standards in the near-future.