Constructive critique is always welcome! I’m sorry the previous response didn’t sufficiently engage with your points. I guess the main thing I didn’t address directly was your question of “were concerns like these taken into account,” and the answer is “basically yes,” although not to the level of detail or precision that would be ideal going forward. Some of the prompts we asked our volunteer researchers to consider included:
How are decisions made in this institution? Who has ultimate authority? Who has practical authority?
How and under what circumstances does this institution make changes to a) its overall priorities and b) the way that it operates? Is it possible to influence relevant subdivisions of this institution even if its overall leadership or culture is resistant to change?
Is this institution likely to become more or less important on the world stage in the next 10 years? What about the next 100? Please note any relevant contingencies, e.g. potential mergers or splits.
FYI, the full model is now posted in my response above to MathiasKB; it sounds like it might be helpful for you to take a look if you have further questions.
Continuing on:
the ‘capacity for change’ / ‘neglectedness, tractability’ turns out to be a significantly lower leverage point within those institutions, which potentially reinforces the point we might have made a reasonable guess at: that impact / scale can be inversely correlated with flexibility / capacity for change / tractability / etc
As I mentioned in my response to weeatquince, this inverse relationship is already baked into the analysis in the sense that absent institution-specific evidence to the contrary, we assumed that a larger and more bureaucratic or socially-distant-from-EA organization would be harder to influence. I really want to emphasize that the list in the article is not just a list of the most important institutions, it is the list we came up with after we took considerations about tractability into account. Now, it is entirely possible that we underrated those concerns overall. Still, I suspect you may be overrating them—for example, just checking my LinkedIn I find that I have seven connections in common with the current Chief of Staff to the President of the United States...and not because I have been consciously cultivating those connections, but simply because our social and professional circles are not that distant from each other. And I’m just one person: when you combine all of the networks of everyone involved in EA and everyone connected to EIP and the improving institutional decision-making community more broadly, that’s a lot of potential network reach.
I get a sense from having had a brief look at the methodology that insider knowledge from making change in these organisations could have been woven in earlier; either by talking to EAs / EA aligned types working within government or big tech companies or whatever else. This would have been useful for deciding what unit of analysis should be, or just sense-checking ‘will what we produce be useful?’
I’m not sure what you mean by “unit of analysis” in this context, could you give an example? In an ideal world, I think you’re right that the project would have benefited from more engagement with the types of folks you’re talking about. However, members of the project team did include a person working at one of the big tech companies on our list, another person working at a top consulting firm, another person who is at the World Bank, a couple of people who work for the UK government, etc. And one of our advisors chairs an external working group at the WHO. So we did have some of the kinds of access you’re talking about, which I think is a decent start given that we were putting all of this together essentially on a $25k grant.
I’m a bit concerned by choosing to build a model for this, given as you say this work is highly contextual and we don’t have most of this context.
Yeah, this is a totally fair observation, and I’ll confess that at one point I considered ditching the model entirely and just publishing the survey results. In the end, however, I think it proved really useful to us. I’m a big believer in the principle that high uncertainty need not preclude a quantitative approach (Doug Hubbard’s book How to Measure Anything is a really useful resource on this topic—see Luke Muehlhauser’s summary/review here). I personally got a lot out of fiddling with the numbers and learning how robust they were to different assumptions, and that helped give me the confidence to include it in our analysis. It’s not to say that I don’t expect changes to the topline takeaways once we get better information—I do expect them—but I’d be moderately surprised if they were so drastic that it makes this earlier analysis look completely silly in retrospect.
Constructive critique is always welcome! I’m sorry the previous response didn’t sufficiently engage with your points. I guess the main thing I didn’t address directly was your question of “were concerns like these taken into account,” and the answer is “basically yes,” although not to the level of detail or precision that would be ideal going forward. Some of the prompts we asked our volunteer researchers to consider included:
How are decisions made in this institution? Who has ultimate authority? Who has practical authority?
How and under what circumstances does this institution make changes to a) its overall priorities and b) the way that it operates? Is it possible to influence relevant subdivisions of this institution even if its overall leadership or culture is resistant to change?
Is this institution likely to become more or less important on the world stage in the next 10 years? What about the next 100? Please note any relevant contingencies, e.g. potential mergers or splits.
FYI, the full model is now posted in my response above to MathiasKB; it sounds like it might be helpful for you to take a look if you have further questions.
Continuing on:
As I mentioned in my response to weeatquince, this inverse relationship is already baked into the analysis in the sense that absent institution-specific evidence to the contrary, we assumed that a larger and more bureaucratic or socially-distant-from-EA organization would be harder to influence. I really want to emphasize that the list in the article is not just a list of the most important institutions, it is the list we came up with after we took considerations about tractability into account. Now, it is entirely possible that we underrated those concerns overall. Still, I suspect you may be overrating them—for example, just checking my LinkedIn I find that I have seven connections in common with the current Chief of Staff to the President of the United States...and not because I have been consciously cultivating those connections, but simply because our social and professional circles are not that distant from each other. And I’m just one person: when you combine all of the networks of everyone involved in EA and everyone connected to EIP and the improving institutional decision-making community more broadly, that’s a lot of potential network reach.
I’m not sure what you mean by “unit of analysis” in this context, could you give an example? In an ideal world, I think you’re right that the project would have benefited from more engagement with the types of folks you’re talking about. However, members of the project team did include a person working at one of the big tech companies on our list, another person working at a top consulting firm, another person who is at the World Bank, a couple of people who work for the UK government, etc. And one of our advisors chairs an external working group at the WHO. So we did have some of the kinds of access you’re talking about, which I think is a decent start given that we were putting all of this together essentially on a $25k grant.
Yeah, this is a totally fair observation, and I’ll confess that at one point I considered ditching the model entirely and just publishing the survey results. In the end, however, I think it proved really useful to us. I’m a big believer in the principle that high uncertainty need not preclude a quantitative approach (Doug Hubbard’s book How to Measure Anything is a really useful resource on this topic—see Luke Muehlhauser’s summary/review here). I personally got a lot out of fiddling with the numbers and learning how robust they were to different assumptions, and that helped give me the confidence to include it in our analysis. It’s not to say that I don’t expect changes to the topline takeaways once we get better information—I do expect them—but I’d be moderately surprised if they were so drastic that it makes this earlier analysis look completely silly in retrospect.