For me, the value of independent impact evaluation seems particularly clear- though I would agree that orgs doing it in-house is still usually better than nothing.
You mention difficulty, orgs being busy, and orgs having strong priors as possible reasons for the lack of impact evaluation. I’d speculate that financial cost is perhaps the largest factor. Orgs that want to have an impact evaluation, but can’t afford the time cost, could readily commission external evaluations (were finance no issue).[1] RP’s Surveys and Data Analysis Team has provided impact assessment for a couple of core meta orgs, and provided pro bono consultation on impact assessment to a larger number of orgs, and I know our other departments have done cost-effectiveness models in other areas, and I see several other people mentioning they do this in the comments. My impression is that people often dramatically overestimate the extent to which large EA orgs have sufficiently unlimited funding that they can just pay for anything they’d find valuable, without cost being an issue. But, for small-medium size EA projects it seems particularly clear that they often could not afford to pay, even though many of them (in my experience) value impact assessment.
Related to both the points above, it seems to me that interest from funders is one of the biggest potential drivers of whether orgs do impact assessment. If funders desired orgs to have external impact assessments, this would serve as a strong incentive for orgs. Funders could even consider a heuristic, that projects receiving >$X a year should dedicate $Y to external impact assessment. Of course, for that to work, funders would need to provide commensurate additional funding.[2]
Granted, commissioning an external impact evaluation still entails non-trivial time cost, since they need to engage with and provide information to the external assessor for the evaluation to be useful.
Anecdotally, I encounter lots of examples where orgs, of various sizes, are interested in receiving surveys or other private analyses which would help assess their impact, but can’t afford to pay for them.
Just chiming in to say that for EA Netherlands, financial cost is definitely a big factor. Another factor is that for a long time, we didn’t have a sufficiently established programme to evaluate. A third is that, until recently, we didn’t know anything about M&E other than ‘we should do an impact evaluation!’.
Fortunately, the second and third factors are beginning to change, so hopefully we’ll actually be able to commission something soon.
However, realistically, we’d only have a few thousand to spend, and I don’t know how much expertise that would get us. So, if there’s anyone in this thread who thinks they can help us given our low budget, please do reach out!
I agree independence has advantages. OTOH I think there are also important advantages to internal impact evaluation: the results are more likely to be bought into internally, and important context or nuance is less likely to be missed. For making a theory of change specifically, I think it’s quite important this is done internally, usually. Overall I think the ideal setup would quite often be for organisations to have their own internal impact evaluation function.
And that’s interesting on funder interest. In a few cases, organisations I’ve spoken to have been able to get specific grants for impact evaluation. But also org’s might choose to reallocate their existing budget, without needing additional funding, if they consider impact evaluation an essential function. E.g. for a fixed budget, they might decide that they should be allocating at least e.g. 5 / 10% to impact evaluation. (But I guess this might be harder if it required pulling back on existing activities)
I kind’ve see this as more at the org-level than funder-level tbh, similarly to any other spending decision facing an organisation. Perhaps because I’m thinking most about the benefits to org’s themselves. But I definitely still agree that funder interest is a big driver.
I think there are also important advantages to internal impact evaluation: the results are more likely to be bought into internally, and important context or nuance is less likely to be missed.
I agree there are some advantages to internal evaluation. But I think “the results are more likely to be bought into internally” is, in many cases, only an advantage insofar that orgs are erroneously more likely to trust their own work than external independent work.
That said, I agree that the importance of orgs bringing important context and nuance (and just basic information) to the evaluation can hardly be over-stated. My general take here is that the ideal arrangement is for the org and external evaluators to work very closely on an evaluation, so they can combine the benefits of insider knowledge and external expertise.
I would even say that in those kinds of cases, it’s not extremely important whether the evaluation is primarily lead by the org or primarily lead by the external evaluator (so long as there’s still scope for the external evaluator to offer an independent, and maybe even dissenting, take on the org’s conclusions). I think people can reasonably disagree about how important it is that, in addition, the external evaluator is truly independent (i.e. ideally funded by an external funder, not selected and contracted by the org in question, which obviously potentially risks biasing the evaluator).
I actually think it could be quite reasonable for an org to trust or place more weight on an internal evaluation more than an external one, but apart from that fully agree with all you say!
Thanks for writing this!
For me, the value of independent impact evaluation seems particularly clear- though I would agree that orgs doing it in-house is still usually better than nothing.
You mention difficulty, orgs being busy, and orgs having strong priors as possible reasons for the lack of impact evaluation. I’d speculate that financial cost is perhaps the largest factor. Orgs that want to have an impact evaluation, but can’t afford the time cost, could readily commission external evaluations (were finance no issue).[1] RP’s Surveys and Data Analysis Team has provided impact assessment for a couple of core meta orgs, and provided pro bono consultation on impact assessment to a larger number of orgs, and I know our other departments have done cost-effectiveness models in other areas, and I see several other people mentioning they do this in the comments. My impression is that people often dramatically overestimate the extent to which large EA orgs have sufficiently unlimited funding that they can just pay for anything they’d find valuable, without cost being an issue. But, for small-medium size EA projects it seems particularly clear that they often could not afford to pay, even though many of them (in my experience) value impact assessment.
Related to both the points above, it seems to me that interest from funders is one of the biggest potential drivers of whether orgs do impact assessment. If funders desired orgs to have external impact assessments, this would serve as a strong incentive for orgs. Funders could even consider a heuristic, that projects receiving >$X a year should dedicate $Y to external impact assessment. Of course, for that to work, funders would need to provide commensurate additional funding.[2]
Granted, commissioning an external impact evaluation still entails non-trivial time cost, since they need to engage with and provide information to the external assessor for the evaluation to be useful.
Anecdotally, I encounter lots of examples where orgs, of various sizes, are interested in receiving surveys or other private analyses which would help assess their impact, but can’t afford to pay for them.
Just chiming in to say that for EA Netherlands, financial cost is definitely a big factor. Another factor is that for a long time, we didn’t have a sufficiently established programme to evaluate. A third is that, until recently, we didn’t know anything about M&E other than ‘we should do an impact evaluation!’.
Fortunately, the second and third factors are beginning to change, so hopefully we’ll actually be able to commission something soon.
However, realistically, we’d only have a few thousand to spend, and I don’t know how much expertise that would get us. So, if there’s anyone in this thread who thinks they can help us given our low budget, please do reach out!
Thanks David!
I agree independence has advantages. OTOH I think there are also important advantages to internal impact evaluation: the results are more likely to be bought into internally, and important context or nuance is less likely to be missed. For making a theory of change specifically, I think it’s quite important this is done internally, usually. Overall I think the ideal setup would quite often be for organisations to have their own internal impact evaluation function.
And that’s interesting on funder interest. In a few cases, organisations I’ve spoken to have been able to get specific grants for impact evaluation. But also org’s might choose to reallocate their existing budget, without needing additional funding, if they consider impact evaluation an essential function. E.g. for a fixed budget, they might decide that they should be allocating at least e.g. 5 / 10% to impact evaluation. (But I guess this might be harder if it required pulling back on existing activities)
I kind’ve see this as more at the org-level than funder-level tbh, similarly to any other spending decision facing an organisation. Perhaps because I’m thinking most about the benefits to org’s themselves. But I definitely still agree that funder interest is a big driver.
Thanks for the reply!
I agree there are some advantages to internal evaluation. But I think “the results are more likely to be bought into internally” is, in many cases, only an advantage insofar that orgs are erroneously more likely to trust their own work than external independent work.
That said, I agree that the importance of orgs bringing important context and nuance (and just basic information) to the evaluation can hardly be over-stated. My general take here is that the ideal arrangement is for the org and external evaluators to work very closely on an evaluation, so they can combine the benefits of insider knowledge and external expertise.
I would even say that in those kinds of cases, it’s not extremely important whether the evaluation is primarily lead by the org or primarily lead by the external evaluator (so long as there’s still scope for the external evaluator to offer an independent, and maybe even dissenting, take on the org’s conclusions). I think people can reasonably disagree about how important it is that, in addition, the external evaluator is truly independent (i.e. ideally funded by an external funder, not selected and contracted by the org in question, which obviously potentially risks biasing the evaluator).
I actually think it could be quite reasonable for an org to trust or place more weight on an internal evaluation more than an external one, but apart from that fully agree with all you say!