For what it’s worth, I agree with the theoretical argument that people might be biased to overweight interestingness over importance.
But I think I disagree fairly strong with the following (implied?) points:
The incentive gradient within EA (always, or usually) points people towards work with longer feedback loops than shorter feedback loops.
The incentive gradient within EA rewards you for academia than for more direct work in EA.
The incentive gradient within EA rewards you for more theoretical work than more applied work
I think I agree with a limited version of:
People seem drawn to “bigger picture”, more abstract work over narrow, concrete, specific problems.
People may naturally be overly inclined to do theoretical work over applied work
But I think this is in spite of EA incentive gradients, not because of them.
Perhaps in some specific circumstances, EA social incentives need to be stronger to point people away from interestingness and towards importance.
Some collection of personal experiences:
On the forum: I feel like I get many more points (in the sense of recognition, but also literally EAF karma) per hour for commenting than for top-level posts. The former is faster and has locally better feedback loops.
On my day job (research manager): I feel like the incentive gradient (especially locally) points me to doing local activities like day to day management, giving reviews/feedback on people’s research, and away from forming deeper clarity/high-level strategic thinking.
This despite everyone involved generally being supportive and thinking the latter work is very important.
Grantmaking vs. conceptual research: I think per hour, I gain more recognition, encouragement, etc., from grantmaking than for forming conceptual views and research. The former also has much faster feedback loops.
(This is ameliorated by me personally finding grantmaking much less interesting/fun than my other projects).
Friends in ML academia vs alignment labs: I feel like my friends in academia usually get less points when they want to stay in academia, as opposed to joining Anthropic/Redwood.
And more locally, they seem to get more prestige points per hour for stuff like hosting podcasts, grantmaking, LW posts, etc, than from writing papers and deepening their understanding of ML.
(This isn’t a clear misallocation. Naively you’d expect people staying in academia to get more prestige points from academia/the outside world, so EA giving them relatively less points seem like an appropriate redistribution).
How I got into EA research: I got into EA research in 2020 via one of the EAish projects with the fastest feedback loops ever (coronavirus forecasting). I suspect I’d be a lot slower to either a) find internal traction or b) external engagement/recognition if I were to try to start with, e.g., AI strategy or some of the other research projects I was excited about in early 2020 (e.g. heavy-tailedness of talent).
More broadly, I think the general vibe I get from the world is that you get locally rewarded more for things that have faster feedback loops and are more attention grabbing (related notes: gamification, contrast with Deep Work). So it should be surprising if EA is a stark contrast.
That said, I do think something about people finding “bigger picture” problems more interesting and doing them anyway, despite incentives pointing in the other discussion, is a relevant piece of the puzzle.
I think I’ve been in multiple positions where I strongly advised very junior researchers to focus on narrower, more concrete research questions, as it would be better for their career and (by my personal estimation) their personal learning to start small and concrete. Usually, they ended up tackling the “big picture” nebulous problems anyway. See also Nuno’s notes.
(I’ve also been on the receiving end of such advice)
Still, there’s something that needs to be explained via both your and Duncan’s beliefs that people keep going up the longer feedback loop ladder and the abstraction ladder. If I were to hazard a guess for why people were to believe this, I’d probably go to:
Another important issue is that “it’s easier to pass yourself off as a long-looper when you’re really doing nothing,” but that’s a different discussion.
I think you treat that as a sidenote, but to me maybe this is the whole story. Speaking in a overly general way, we can imagine that EA prestige rewards people in roughly the following ways:
Theoretical wins (e.g. successfully champion a new cause area)> Concrete wins[1](e.g. founding a company doing great work, distributing many bednets) >> Theoretical failure >> Concrete failure.
In this story, it’s easier/much more legible to see somebody fail when they do very concrete work. People are scared of openly failing, so they if they think they can’t get concrete wins (or they don’t believe it has sufficiently high probability), they’re drawn to doing theoretical work that is harder to pinpoint exactly if, or when, they’ve failed. This despite concrete work having perhaps higher EV (both altruistically and selfishly).
An important caveat here is that I think the rest of the world will privilege a concrete win over a theoretical win. So for example, most core EAs would consider “being one of the three most important people instrumental in causing EAs to work on AI alignment as a major cause area” to be more impressive than e.g. founding Wave. But I expect almost anybody else in the world to consider founding Wave a bigger deal.
For what it’s worth, I agree with the theoretical argument that people might be biased to overweight interestingness over importance.
But I think I disagree fairly strong with the following (implied?) points:
The incentive gradient within EA (always, or usually) points people towards work with longer feedback loops than shorter feedback loops.
The incentive gradient within EA rewards you for academia than for more direct work in EA.
The incentive gradient within EA rewards you for more theoretical work than more applied work
I think I agree with a limited version of:
People seem drawn to “bigger picture”, more abstract work over narrow, concrete, specific problems.
People may naturally be overly inclined to do theoretical work over applied work
But I think this is in spite of EA incentive gradients, not because of them.
Perhaps in some specific circumstances, EA social incentives need to be stronger to point people away from interestingness and towards importance.
Some collection of personal experiences:
On the forum: I feel like I get many more points (in the sense of recognition, but also literally EAF karma) per hour for commenting than for top-level posts. The former is faster and has locally better feedback loops.
On my day job (research manager): I feel like the incentive gradient (especially locally) points me to doing local activities like day to day management, giving reviews/feedback on people’s research, and away from forming deeper clarity/high-level strategic thinking.
This despite everyone involved generally being supportive and thinking the latter work is very important.
Grantmaking vs. conceptual research: I think per hour, I gain more recognition, encouragement, etc., from grantmaking than for forming conceptual views and research. The former also has much faster feedback loops.
(This is ameliorated by me personally finding grantmaking much less interesting/fun than my other projects).
Friends in ML academia vs alignment labs: I feel like my friends in academia usually get less points when they want to stay in academia, as opposed to joining Anthropic/Redwood.
And more locally, they seem to get more prestige points per hour for stuff like hosting podcasts, grantmaking, LW posts, etc, than from writing papers and deepening their understanding of ML.
(This isn’t a clear misallocation. Naively you’d expect people staying in academia to get more prestige points from academia/the outside world, so EA giving them relatively less points seem like an appropriate redistribution).
How I got into EA research: I got into EA research in 2020 via one of the EAish projects with the fastest feedback loops ever (coronavirus forecasting). I suspect I’d be a lot slower to either a) find internal traction or b) external engagement/recognition if I were to try to start with, e.g., AI strategy or some of the other research projects I was excited about in early 2020 (e.g. heavy-tailedness of talent).
More broadly, I think the general vibe I get from the world is that you get locally rewarded more for things that have faster feedback loops and are more attention grabbing (related notes: gamification, contrast with Deep Work). So it should be surprising if EA is a stark contrast.
That said, I do think something about people finding “bigger picture” problems more interesting and doing them anyway, despite incentives pointing in the other discussion, is a relevant piece of the puzzle.
I think I’ve been in multiple positions where I strongly advised very junior researchers to focus on narrower, more concrete research questions, as it would be better for their career and (by my personal estimation) their personal learning to start small and concrete. Usually, they ended up tackling the “big picture” nebulous problems anyway. See also Nuno’s notes.
(I’ve also been on the receiving end of such advice)
__
[epistemic status: idle, probably mean, speculation]
Still, there’s something that needs to be explained via both your and Duncan’s beliefs that people keep going up the longer feedback loop ladder and the abstraction ladder. If I were to hazard a guess for why people were to believe this, I’d probably go to:
I think you treat that as a sidenote, but to me maybe this is the whole story. Speaking in a overly general way, we can imagine that EA prestige rewards people in roughly the following ways:
Theoretical wins (e.g. successfully champion a new cause area)> Concrete wins[1](e.g. founding a company doing great work, distributing many bednets) >> Theoretical failure >> Concrete failure.
In this story, it’s easier/much more legible to see somebody fail when they do very concrete work. People are scared of openly failing, so they if they think they can’t get concrete wins (or they don’t believe it has sufficiently high probability), they’re drawn to doing theoretical work that is harder to pinpoint exactly if, or when, they’ve failed. This despite concrete work having perhaps higher EV (both altruistically and selfishly).
So if we celebrate legible failures more, and have less of an act/omission distinction, then perhaps the incentive gradient will point people who fail at conceptual/theoretical work to be more excited to take a shot at doing more concrete work.
Perhaps not, too. I’m far from confident that I pinpointed the exact problem or solution.
___
I also want to quickly note that your post conflates “theoretical/conceptual work” and “slow feedback loops.” I think this is only sort of true.
An important caveat here is that I think the rest of the world will privilege a concrete win over a theoretical win. So for example, most core EAs would consider “being one of the three most important people instrumental in causing EAs to work on AI alignment as a major cause area” to be more impressive than e.g. founding Wave. But I expect almost anybody else in the world to consider founding Wave a bigger deal.