Thanks for your comments, your insight into this grant, and your support!
We do expect to get input from local hospital staff on existing treatment coverage through the baseline surveys. The monitoring grant will fund the creation of a sampling frame that includes both public and private health facilities, which we think will yield more complete data than contacting hospitals through our partners.
We agree that potential bias from external evaluators is a risk for the reasons you’ve mentioned. While we won’t be involved in the selection of evaluators, we plan to do the following to mitigate that risk:
Meet with representatives of all the firms and vet them at a high level so we can identify red (and green) flags.
Work closely with both MiracleFeet and the external evaluators throughout the process, so that we can ask questions about and provide input on their research strategies along the way.
We don’t think this will completely eliminate uncertainty about the quality of monitoring results, but we expect it will help. We also think there is some value to be gained from working with evaluators who have a strong familiarity with the local context.
Thanks for the reply, and most of this makes sense to me.
I’m not sure I understand how you won’t be involved in the selection of evaluators, who will do that exactly? Or maybe you mean you won’t select the on-the-ground evaluators as in that will be done by the company, which makes sense.
“The monitoring grant will fund the creation of a sampling frame that includes both public and private health facilities, which we think will yield more complete data than contacting hospitals through our partners.” This could work (high risk), but seems like a roundabout and inefficient way to do things. Following up on that data from multiple hospitals in West Africa for example could be a nightmare.
I would have thought with this kind of massive funding and the relatively small number of people who get procedures (in the thousands), MiracleFeet could maintain a database the contact details of every kid who gets help—this wouldn’t be hard and would make M&E so much easier for everyone. Hospitals might well collect substandard information which makes proper follow up impossible, spoiling your M&E efforts.
If I was going to give one piece of advice on M&E, it would be that your evaluators should follow up personally a completely random sample of individuals who had been treated—both to check that both the interventions actually happened, and that the claimed improvement is real. There should be a list of names, home locations and phone numbers of every single patient who received treatment—I think if that’s not there and individuals can’t be followed for this kind of intervention, than meaningful M&E becomes close to impossible.
Yes, to clarify, MiracleFeet is selecting the on-the-ground evaluator that will conduct the monitoring in each location, although GiveWell will lightly vet MiracleFeet’s choices and meet with each evaluator.
The primary purpose of the monitoring grant is to understand how many children are treated for clubfoot both with and without MiracleFeet’s support. So, although MiracleFeet has records of children treated through facilities it’s supported, we also want an assessment of baseline treatment coverage before MiracleFeet launches its program (or expands it, in the case of the Philippines). We do plan to incorporate some form of data audit as part of endline activities; we’ll work out the details of that at a later date.
Thanks again for your interest in this and for taking the time to ask questions!
Hi, Nick,
Thanks for your comments, your insight into this grant, and your support!
We do expect to get input from local hospital staff on existing treatment coverage through the baseline surveys. The monitoring grant will fund the creation of a sampling frame that includes both public and private health facilities, which we think will yield more complete data than contacting hospitals through our partners.
We agree that potential bias from external evaluators is a risk for the reasons you’ve mentioned. While we won’t be involved in the selection of evaluators, we plan to do the following to mitigate that risk:
Meet with representatives of all the firms and vet them at a high level so we can identify red (and green) flags.
Work closely with both MiracleFeet and the external evaluators throughout the process, so that we can ask questions about and provide input on their research strategies along the way.
We don’t think this will completely eliminate uncertainty about the quality of monitoring results, but we expect it will help. We also think there is some value to be gained from working with evaluators who have a strong familiarity with the local context.
I hope that’s helpful!
Best,
Miranda
Thanks for the reply, and most of this makes sense to me.
I’m not sure I understand how you won’t be involved in the selection of evaluators, who will do that exactly? Or maybe you mean you won’t select the on-the-ground evaluators as in that will be done by the company, which makes sense.
“The monitoring grant will fund the creation of a sampling frame that includes both public and private health facilities, which we think will yield more complete data than contacting hospitals through our partners.” This could work (high risk), but seems like a roundabout and inefficient way to do things. Following up on that data from multiple hospitals in West Africa for example could be a nightmare.
I would have thought with this kind of massive funding and the relatively small number of people who get procedures (in the thousands), MiracleFeet could maintain a database the contact details of every kid who gets help—this wouldn’t be hard and would make M&E so much easier for everyone. Hospitals might well collect substandard information which makes proper follow up impossible, spoiling your M&E efforts.
If I was going to give one piece of advice on M&E, it would be that your evaluators should follow up personally a completely random sample of individuals who had been treated—both to check that both the interventions actually happened, and that the claimed improvement is real. There should be a list of names, home locations and phone numbers of every single patient who received treatment—I think if that’s not there and individuals can’t be followed for this kind of intervention, than meaningful M&E becomes close to impossible.
Hi, Nick,
Yes, to clarify, MiracleFeet is selecting the on-the-ground evaluator that will conduct the monitoring in each location, although GiveWell will lightly vet MiracleFeet’s choices and meet with each evaluator.
The primary purpose of the monitoring grant is to understand how many children are treated for clubfoot both with and without MiracleFeet’s support. So, although MiracleFeet has records of children treated through facilities it’s supported, we also want an assessment of baseline treatment coverage before MiracleFeet launches its program (or expands it, in the case of the Philippines). We do plan to incorporate some form of data audit as part of endline activities; we’ll work out the details of that at a later date.
Thanks again for your interest in this and for taking the time to ask questions!
Miranda