I sometimes worry that the feedback loop from do useful research all the way down to see a significant change in the world is very long and opaque. If people have general thoughts on this feedback loop or on evaluating research organisations do share.
Here are some posts Iāve found useful in relation to that or similar matters:
FWIW as a donor I would be very wary of giving to a research organisation without a theory of change and/āor strategic plan and an idea of how to measure impact (surveys or otherwise). Someone saying such work was not needed would be a massive red flag to me. Like if a global health charity says we donāt need to measure impact we know we are doing good ā maybe that global health charity is the most effective global health charity in the world but it is not going to be able to convince me of that fact.
Iād also be wary of that, and I tentatively think that many research orgs should probably move towards doing more explicit thinking about their theory of change /ā strategic plan and impact/āprogress assessment, and providing somewhat more public info about this. (The fact and way that Rethink Priorities does this stuff was one of the things that made me excited about getting a job with them [though this comment is just my personal opinion, as with most/āall of my comments].)
That said, I get the impression that your version of these stances might be a bit stronger than mine, or a bit different.
One thing that feels worth stating explicitly is that me not having seen an orgās theory of change /ā strategic plan /ā approach to impact assessment doesnāt necessarily mean they donāt have these things. They might have fair reasons for not making these things public.
And Iām also potentially ok with an org not having explicitly written these things down, if theyāve had thorough discussions, have a shared understanding, regularly check in about these things, etc. (This might apply especially to smaller and newer orgs.)
(Iām not sure whether youād disagree with these thingsāthey just felt worth stating explicitly. Iām also uncertain about my views on these matters, and want to think more about them over the coming year.)
Here are some posts Iāve found useful in relation to that or similar matters:
Do research organisations make theory of change diagrams? Should they? (and the answers and comments there)
Rethink Priorities Impact Survey
Should surveys about the quality/āimpact of research outputs be more common?
(this is one I wrote, rather than one I āfound usefulā, though I found the process of thinking and writing about this topic useful)
Can we intentionally improve the world? Planners vs. Hayekians
AI Governance: Opportunity and Theory of Impact
Also relevant, but a bit more tangentially:
Rethink Priorities 2019 Impact and Strategy
Identifying Plausible Paths to Impact and their Strategic Implications
Effective Altruism Foundation: Plans for 2020
Thank you all super interesting reading.
FWIW as a donor I would be very wary of giving to a research organisation without a theory of change and/āor strategic plan and an idea of how to measure impact (surveys or otherwise). Someone saying such work was not needed would be a massive red flag to me. Like if a global health charity says we donāt need to measure impact we know we are doing good ā maybe that global health charity is the most effective global health charity in the world but it is not going to be able to convince me of that fact.
Iād also be wary of that, and I tentatively think that many research orgs should probably move towards doing more explicit thinking about their theory of change /ā strategic plan and impact/āprogress assessment, and providing somewhat more public info about this. (The fact and way that Rethink Priorities does this stuff was one of the things that made me excited about getting a job with them [though this comment is just my personal opinion, as with most/āall of my comments].)
That said, I get the impression that your version of these stances might be a bit stronger than mine, or a bit different.
One thing that feels worth stating explicitly is that me not having seen an orgās theory of change /ā strategic plan /ā approach to impact assessment doesnāt necessarily mean they donāt have these things. They might have fair reasons for not making these things public.
And Iām also potentially ok with an org not having explicitly written these things down, if theyāve had thorough discussions, have a shared understanding, regularly check in about these things, etc. (This might apply especially to smaller and newer orgs.)
(Iām not sure whether youād disagree with these thingsāthey just felt worth stating explicitly. Iām also uncertain about my views on these matters, and want to think more about them over the coming year.)