I think the basic inference we can draw from those anecdotes (and lived experience in DC) is this line from the post:
Whereas the potential for impact is widely accepted, the average level of think tank impact is far more uncertain.
The best-known studies on the question of impact (to my knowledge) are included in Appendix A, but I don’t know that they’d fully satisfy you (they don’t fully satisfy me either).
Questions about average impact (or “systemic estimates”) are tough to answer, in part because all the concepts involved are rather underspecified. “Think tank work” involves many things, and varies widely in quality. Some types of work and some think tanks could be quite effective, whereas others may not be. “Impact” can also mean many things, most of which are by definition hard to observe and measure (I myself work at a think tank and probably >50% of the impact I’ve had is talking policymakers or government staff out of bad ideas, which means 0 observed action in the world).
Even if you had months to collect data, or had the ability to run an RCT, I’m not sure (a) what variables you would include in a regression model or (b) even if you did manage to come up with something, what useful things we could learn from the coefficients/average relationships. Due to the wide variety of think tank work that exists, the relevant reference class for any given think tank-related decision (e.g. if you are a funder assessing a specific proposal or an applicant considering whether to take a job) is going to be rather narrow (much narrower than “think tank projects” or “think tank jobs” writ large).
By the way, I relied on your report and some other literature to write a brief EA Wiki entry on think tanks. If you decide to read it and have any critical feedback, feel free to leave a comment or get in touch.
I think the basic inference we can draw from those anecdotes (and lived experience in DC) is this line from the post:
The best-known studies on the question of impact (to my knowledge) are included in Appendix A, but I don’t know that they’d fully satisfy you (they don’t fully satisfy me either).
Questions about average impact (or “systemic estimates”) are tough to answer, in part because all the concepts involved are rather underspecified. “Think tank work” involves many things, and varies widely in quality. Some types of work and some think tanks could be quite effective, whereas others may not be. “Impact” can also mean many things, most of which are by definition hard to observe and measure (I myself work at a think tank and probably >50% of the impact I’ve had is talking policymakers or government staff out of bad ideas, which means 0 observed action in the world).
Even if you had months to collect data, or had the ability to run an RCT, I’m not sure (a) what variables you would include in a regression model or (b) even if you did manage to come up with something, what useful things we could learn from the coefficients/average relationships. Due to the wide variety of think tank work that exists, the relevant reference class for any given think tank-related decision (e.g. if you are a funder assessing a specific proposal or an applicant considering whether to take a job) is going to be rather narrow (much narrower than “think tank projects” or “think tank jobs” writ large).
Thanks for the answer.
By the way, I relied on your report and some other literature to write a brief EA Wiki entry on think tanks. If you decide to read it and have any critical feedback, feel free to leave a comment or get in touch.
Cool, great to see, thanks!