I sometimes worry that the feedback loop from do useful research all the way down to see a significant change in the world is very long and opaque. If people have general thoughts on this feedback loop or on evaluating research organisations do share.
Here are some posts I’ve found useful in relation to that or similar matters:
Do research organisations make theory of change diagrams? Should they? (and the answers and comments there)
Rethink Priorities Impact Survey
Should surveys about the quality/impact of research outputs be more common?
(this is one I wrote, rather than one I “found useful”, though I found the process of thinking and writing about this topic useful)
Can we intentionally improve the world? Planners vs. Hayekians
AI Governance: Opportunity and Theory of Impact
Also relevant, but a bit more tangentially:
Rethink Priorities 2019 Impact and Strategy
Identifying Plausible Paths to Impact and their Strategic Implications
Effective Altruism Foundation: Plans for 2020
Thank you all super interesting reading.FWIW as a donor I would be very wary of giving to a research organisation without a theory of change and/or strategic plan and an idea of how to measure impact (surveys or otherwise). Someone saying such work was not needed would be a massive red flag to me. Like if a global health charity says we don’t need to measure impact we know we are doing good – maybe that global health charity is the most effective global health charity in the world but it is not going to be able to convince me of that fact.
I’d also be wary of that, and I tentatively think that many research orgs should probably move towards doing more explicit thinking about their theory of change / strategic plan and impact/progress assessment, and providing somewhat more public info about this. (The fact and way that Rethink Priorities does this stuff was one of the things that made me excited about getting a job with them [though this comment is just my personal opinion, as with most/all of my comments].)
That said, I get the impression that your version of these stances might be a bit stronger than mine, or a bit different.
One thing that feels worth stating explicitly is that me not having seen an org’s theory of change / strategic plan / approach to impact assessment doesn’t necessarily mean they don’t have these things. They might have fair reasons for not making these things public.
And I’m also potentially ok with an org not having explicitly written these things down, if they’ve had thorough discussions, have a shared understanding, regularly check in about these things, etc. (This might apply especially to smaller and newer orgs.)
(I’m not sure whether you’d disagree with these things—they just felt worth stating explicitly. I’m also uncertain about my views on these matters, and want to think more about them over the coming year.)