Agreed Emre, I think that’s a good point and probably something I didn’t consider much for the purpose of this piece. I guess one factor that makes it less valuable for organisations conducting this research internally is that they’ve got pretty strong incentives to find good results, otherwise it means their work isn’t effective. I think this can be a reason for people to measure the wrong thing, or otherwise not conduct research that really gets at the heart of whether they’re being effective or not.
Also agreed about not being able to extrapolate much when there is small sample sizes. Something I’ve found quite hard for campaigning-related research is the fact that it’s so context dependent that it’s unclear how much you can extrapolate from even 1000 campaigns if they’ve been conducted in 20 different countries working on 3-4 different issues. Obviously somethings will generalise, but some won’t! And it’s hard to tell which one is in which camp a priori.
And your last point is another tough thing about research. It’s well and good conducting great research—but often it can slip under the radar or people don’t actually implement the recommendations. I think this is a time when external bodies (e.g. charity evaluators or funders) can be good at holding organisations to account on doing the most impactful things.
[doxxing myself—I wrote the submission above!]
Agreed Emre, I think that’s a good point and probably something I didn’t consider much for the purpose of this piece. I guess one factor that makes it less valuable for organisations conducting this research internally is that they’ve got pretty strong incentives to find good results, otherwise it means their work isn’t effective. I think this can be a reason for people to measure the wrong thing, or otherwise not conduct research that really gets at the heart of whether they’re being effective or not.
Also agreed about not being able to extrapolate much when there is small sample sizes. Something I’ve found quite hard for campaigning-related research is the fact that it’s so context dependent that it’s unclear how much you can extrapolate from even 1000 campaigns if they’ve been conducted in 20 different countries working on 3-4 different issues. Obviously somethings will generalise, but some won’t! And it’s hard to tell which one is in which camp a priori.
And your last point is another tough thing about research. It’s well and good conducting great research—but often it can slip under the radar or people don’t actually implement the recommendations. I think this is a time when external bodies (e.g. charity evaluators or funders) can be good at holding organisations to account on doing the most impactful things.