Wow thanks so much again for the great insights. So good to have experienced development practitioners here!
To give some background I came to EA partly because I saw how useless most NGOs are here where I live, and the EA framework answers many of the questions as to why, and some of the questions as to how to fix the problem. If I was the one doing M&E and had a magic wand, I would probably decide to shut down over 80% of NGOs and programs that I assessed.
Also we have had a bunch of USAID and other funded M&E pass through many of our health centers, and they have almost never either found our biggest problems nor suggested good solutions—with one exception of a specifically focused financial management assessment which was actually really helpful.
I won’t respond to everything but just make a few comments :)..
Your M&E might just be better First, the level of M&E you do might be so much better than I have seen, that some of the issues I talk about might not apply so much.
”For example if you find an education project is cost-effectively increasing test scores, but only for female students and not struggling male students what do you say?”
That you have even done the kind of analysis that allows you to ask this kind of great question would put you above nearly any M&E that I have ever seen here in Northern Uganda. Even the concept of calculated “cost effectiveness” as we know it rarely (if ever) considered here. I can’t think of anyone who has assessed either the bigger health centers we operate or OneDay Health who has included this in an assessment.
I’m not sure how you would answer that question, but the fact that you have even reached that point means that in my eyes you are already winning to some degree. Also this analysis is so fantastic thanks for sharing I haven’t seen that before! My only comment is that I don’t think the analysis generated “mixed’ results -they seem very clear to me :D!
External assessors for data collection, local assessors for analysis and change? For an assessment like this one of Miraclefeet, I favour external assessors to gather the basic data then perhaps local assessors could take over on the analysis? Data collection needs to be squeaky clean otherwise everything else falls down. This particular assessment should be fairly straightforward to assess by first gathering these data
1. Have the clubfoot procedures actually been done as stated? This needs a random selection of a sample of all patients allegedly worked on (say 100 randomly selected from a list of 5000 patients provided by Miricalefeet) then each one of those should be physically followed up in their home and checked. This isn’t difficult, and anything else is open to bias.
2. What has the “average intervention” achieved? Then those same 100 patients should be assessed for impact—what is their objective level of functionality and subjective improvement in wellbeing/quality of life after the procedure.
Once these 2 pieces of data are gathered, the organisational analysis and discussion you speak of can start and that might be more productive on a local-to-local level, providing the local expertise is available.
Integrity there but comes second? I know integrity is an important driver like you say, and I love your comment that it is easy to compute and hard to action. In my experience integrity is usually there, but often falls behind the other “positive skew” motivating factors. Also I agree that M&E shouldn’t always be done by external firms partly because of the reasons you state. An added reason is that external firms often hire lots of local people to do much of the work anyway, so the same issues I outlined remain.
A small disagreement? ”I have seen orgs who have been unhappy with an external evaluation because they feel the evaluators made judgements when they didn’t fully understand the operating context (and how can they with often only a fly by visit to project locations) or did not properly take into account the values of the organisation or the community but rather only listened to the funder.”
In my experience this response might be a red flag. A sign that the org might dodging and weaving after failing to perform. I believe almost all organisation should do do pre specified actions A,B and C which provides impact X, Y, and Z. If these actions aren’t happening and impact isn’t produced then that needs to be fixed or maybe the work needs to stop. External evaluators’ job isn’t to understand the context (how could they possibly do that? Its not realistic. I’ve been in Uganda for 10 years and in many ways I still don’t understand the local context) -that is our job, the practitioners. Their job is to see what the org is doing and whether the impact is happening.
As a side note I’m a little disappointed that we don’t have more engagement on this discussion. - the “M&E question” is so important, but perhaps its not sexy and probably isn’t accessible to many.
Wow thanks so much again for the great insights. So good to have experienced development practitioners here!
To give some background I came to EA partly because I saw how useless most NGOs are here where I live, and the EA framework answers many of the questions as to why, and some of the questions as to how to fix the problem. If I was the one doing M&E and had a magic wand, I would probably decide to shut down over 80% of NGOs and programs that I assessed.
Also we have had a bunch of USAID and other funded M&E pass through many of our health centers, and they have almost never either found our biggest problems nor suggested good solutions—with one exception of a specifically focused financial management assessment which was actually really helpful.
I won’t respond to everything but just make a few comments :)..
Your M&E might just be better
First, the level of M&E you do might be so much better than I have seen, that some of the issues I talk about might not apply so much.
”For example if you find an education project is cost-effectively increasing test scores, but only for female students and not struggling male students what do you say?”
That you have even done the kind of analysis that allows you to ask this kind of great question would put you above nearly any M&E that I have ever seen here in Northern Uganda. Even the concept of calculated “cost effectiveness” as we know it rarely (if ever) considered here. I can’t think of anyone who has assessed either the bigger health centers we operate or OneDay Health who has included this in an assessment.
I’m not sure how you would answer that question, but the fact that you have even reached that point means that in my eyes you are already winning to some degree. Also this analysis is so fantastic thanks for sharing I haven’t seen that before! My only comment is that I don’t think the analysis generated “mixed’ results -they seem very clear to me :D!
External assessors for data collection, local assessors for analysis and change?
For an assessment like this one of Miraclefeet, I favour external assessors to gather the basic data then perhaps local assessors could take over on the analysis? Data collection needs to be squeaky clean otherwise everything else falls down. This particular assessment should be fairly straightforward to assess by first gathering these data
1. Have the clubfoot procedures actually been done as stated? This needs a random selection of a sample of all patients allegedly worked on (say 100 randomly selected from a list of 5000 patients provided by Miricalefeet) then each one of those should be physically followed up in their home and checked. This isn’t difficult, and anything else is open to bias.
2. What has the “average intervention” achieved? Then those same 100 patients should be assessed for impact—what is their objective level of functionality and subjective improvement in wellbeing/quality of life after the procedure.
Once these 2 pieces of data are gathered, the organisational analysis and discussion you speak of can start and that might be more productive on a local-to-local level, providing the local expertise is available.
Integrity there but comes second?
I know integrity is an important driver like you say, and I love your comment that it is easy to compute and hard to action. In my experience integrity is usually there, but often falls behind the other “positive skew” motivating factors. Also I agree that M&E shouldn’t always be done by external firms partly because of the reasons you state. An added reason is that external firms often hire lots of local people to do much of the work anyway, so the same issues I outlined remain.
A small disagreement?
”I have seen orgs who have been unhappy with an external evaluation because they feel the evaluators made judgements when they didn’t fully understand the operating context (and how can they with often only a fly by visit to project locations) or did not properly take into account the values of the organisation or the community but rather only listened to the funder.”
In my experience this response might be a red flag. A sign that the org might dodging and weaving after failing to perform. I believe almost all organisation should do do pre specified actions A,B and C which provides impact X, Y, and Z. If these actions aren’t happening and impact isn’t produced then that needs to be fixed or maybe the work needs to stop. External evaluators’ job isn’t to understand the context (how could they possibly do that? Its not realistic. I’ve been in Uganda for 10 years and in many ways I still don’t understand the local context) - that is our job, the practitioners. Their job is to see what the org is doing and whether the impact is happening.
As a side note I’m a little disappointed that we don’t have more engagement on this discussion. - the “M&E question” is so important, but perhaps its not sexy and probably isn’t accessible to many.