Thank you for taking the time to write this valuable report.
Do you happen to be familiar with systematic attempts to estimate the impact of think tanks on policy, or to identify the main paths to impact? I am aware of various anecdotes and case studies—you discuss the Center for Budget and Policy Priorities, Rob mentions the Center for Global Development and others, and I would add the Institute of Economic Affairs—, but it’s unclear to me what can be inferred from this type of impressionistic evidence.
I think the basic inference we can draw from those anecdotes (and lived experience in DC) is this line from the post:
Whereas the potential for impact is widely accepted, the average level of think tank impact is far more uncertain.
The best-known studies on the question of impact (to my knowledge) are included in Appendix A, but I don’t know that they’d fully satisfy you (they don’t fully satisfy me either).
Questions about average impact (or “systemic estimates”) are tough to answer, in part because all the concepts involved are rather underspecified. “Think tank work” involves many things, and varies widely in quality. Some types of work and some think tanks could be quite effective, whereas others may not be. “Impact” can also mean many things, most of which are by definition hard to observe and measure (I myself work at a think tank and probably >50% of the impact I’ve had is talking policymakers or government staff out of bad ideas, which means 0 observed action in the world).
Even if you had months to collect data, or had the ability to run an RCT, I’m not sure (a) what variables you would include in a regression model or (b) even if you did manage to come up with something, what useful things we could learn from the coefficients/average relationships. Due to the wide variety of think tank work that exists, the relevant reference class for any given think tank-related decision (e.g. if you are a funder assessing a specific proposal or an applicant considering whether to take a job) is going to be rather narrow (much narrower than “think tank projects” or “think tank jobs” writ large).
By the way, I relied on your report and some other literature to write a brief EA Wiki entry on think tanks. If you decide to read it and have any critical feedback, feel free to leave a comment or get in touch.
Thank you for taking the time to write this valuable report.
Do you happen to be familiar with systematic attempts to estimate the impact of think tanks on policy, or to identify the main paths to impact? I am aware of various anecdotes and case studies—you discuss the Center for Budget and Policy Priorities, Rob mentions the Center for Global Development and others, and I would add the Institute of Economic Affairs—, but it’s unclear to me what can be inferred from this type of impressionistic evidence.
I think the basic inference we can draw from those anecdotes (and lived experience in DC) is this line from the post:
The best-known studies on the question of impact (to my knowledge) are included in Appendix A, but I don’t know that they’d fully satisfy you (they don’t fully satisfy me either).
Questions about average impact (or “systemic estimates”) are tough to answer, in part because all the concepts involved are rather underspecified. “Think tank work” involves many things, and varies widely in quality. Some types of work and some think tanks could be quite effective, whereas others may not be. “Impact” can also mean many things, most of which are by definition hard to observe and measure (I myself work at a think tank and probably >50% of the impact I’ve had is talking policymakers or government staff out of bad ideas, which means 0 observed action in the world).
Even if you had months to collect data, or had the ability to run an RCT, I’m not sure (a) what variables you would include in a regression model or (b) even if you did manage to come up with something, what useful things we could learn from the coefficients/average relationships. Due to the wide variety of think tank work that exists, the relevant reference class for any given think tank-related decision (e.g. if you are a funder assessing a specific proposal or an applicant considering whether to take a job) is going to be rather narrow (much narrower than “think tank projects” or “think tank jobs” writ large).
Thanks for the answer.
By the way, I relied on your report and some other literature to write a brief EA Wiki entry on think tanks. If you decide to read it and have any critical feedback, feel free to leave a comment or get in touch.
Cool, great to see, thanks!