I share several of the worries articulated in this post.
I think you’re wrong about how you characterise 80k’s strategic shift here, and want to try to correct the record on that point. I’m also going to give some concrete examples of things I’m currently doing, to illustrate a bit what I mean, & also include a few more personal comments.
(Context: I run the 80k web programme.)
if you glorify some relatively-value-neutral conception of AI safety as the summum bonum of what is or used to be EA, there is just a good chance that you will lose the plot and end up not pursuing the the actual highest good, the good itself.
Well put. And I agree that there are some concerning signs in this direction (though I’ve also had countervailing, inspiring experiences of AIS-focused people questioning whether some prevailing view about what to do in AIS is actually best for the world.)
I’d also love to see more cause prioritisation research. And it’s gonna be hard to both stay open enough to changing our minds about how to make the world better & to pursue our chosen means with enough focus to be effective. I think this challenge is fairly central to EA.
On 80k’s strategic shift:
You wrote:
Perhaps the clearest and most predictive embodiment of the trend is 80,000 Hours’ new strategic focus on AI. 80k was always fundamentally about providing thorough, practical cause/intervention prioritization and that exercise can be fairly regarded as the core of EA. They’re now effectively saying the analysis is done: doing the most good means steering AI development, so we’ll now focus only on the particulars of what to do in AI. Thanks, we’ll take it from here indeed.
How do we see the relationship between focusing on helping AGI go well and doing the most good?
It has always been the case that people and organisations need to find some intermediary outcome that comes before the good to point at strategically, some proxy for impact. Strategy is always about figuring out what’s gonna be the biggest/most cost-effective causal factor for that (i.e. means), & therefore the best proxy to pursue.
We used to focus on career changes not necessarily inside one specific cause area but it was still a proxy for the good. Now our proxy for the good is helping people work on making AGI go well, but our relationship to the good is the same as it was before: trying our best to point at it, trying to figure out the best means for doing so.
EA values & ideas are still a really important part of the strategy.
As mentioned, we’re still using EA values (e.g. those listed here and here) to determine what to prioritise, including in making this strategic shift.
And we still think it’s important for people to use EA values and ideas as they’re thinking about and pursuing high-impact careers. Some particular examples which feel salient to us:
Scope sensitivity and thinking on the margin seem important for having an impact in any area, including helping AGI go well.
We think there are some roles / areas of work where it’s especially important to continually use EA-style ideas and be steadfastly pointed at having a positive impact in order for it to be good to work in the area. For example, in roles where it’s possible to do a large amount of accidental harm, like working at an AI company, or roles where you have a lot of influence in steering an organisation’s direction.
There are also a variety of areas where EA-style thinking about issues like moral patienthood, neglectedness, leverage, etc. are still incredibly useful – e.g. grand challenges humanity may face due to explosive progress from transformatively powerful AI.
Though one might understandably worry that was paying lip service, just to reassure people. Let me talk about some internal recent goings-on off the top of my head, which hopefully do something to show we mean it:
1. Our internal doc on web programme strategy (i.e. the strategy for the programme I run) currently says that in order for our audience to actually have much more impact with their careers, engagement with the site ideally causes movement along at least 3[1] dimensions:
A. Awareness of transformative AI potential B. EA-mindest (i.e. using ideas like impartiality, scope sensitivity, and thinking on the margin) C. Career position
This makes re-designing the user flow post-strategic-shift a difficult balancing act/full of tradeoffs. How do we both quickly introduce people to AI being a big deal & urgent, and communicate EA ideas, plus help people shift their careers? Which do we do first?
We’re going to lose some simplicity (and some people, who don’t want to hear it) trying to do all this, and it will be reflected in the site being more complex than a strategy like “maximize for engagement or respectability” or “maximize for getting one idea across effectively” would recommend.
My view is that it’s worth it, because there is a danger of people just jumping into jobs that have “AI” or even “AI security/safety” in the name, without grappling with tough questions around what it actually means to help AGI go well or prioritising between options based on expected impact.
(On the term “EA mindset”—it’s really just a nickname; the thing I think we should care about is the focus on impact/use of the ideas.)
2. Our CEO (Niel Bowerman) spent several weeks recently with his top proactive priority helping figure out the top priorities within making AGI go well – i.e. which is more pressing (in the sense of where can additional talented people do the most marginal good) between issues like AI-enabled human coups, getting things right with rights and welfare of digital minds, and catastrophic misalignment. We argued about questions like “how big is the spread between issues within making AGI go well?” and “to what extent is AI rights and welfare an issue human has to get right before AI becomes incredibly powerful, due to potential lock-in effects of bad discourse or policies?”
So, we agree with this:
exactly what people end up doing within “AI safety” matters enormously from the EA perspective. [...] We actually care about the overall moral value of the long run future. Making AI less racist or preventing its use in petty scams doesn’t really cut it in those terms.
In other words, the analysis, as you say, is not done. It’s gonna be hecka hard to figure out “the particulars of what to do with AI.” And we do not “have it from here” – we need people thinking critically about this going forward so they stand the best chance of actually helping AGI go well, rather than just having a career in “something something AI.”
[1](I’m currently debating whether we should add a 4th: tactical sophistication about AI.)
My view is that it’s worth it, because there is a danger of people just jumping into jobs that have “AI” or even “AI security/safety” in the name, without grappling with tough questions around what it actually means to help AGI go well or prioritising between options based on expected impact.
I appreciate the dilemma and don’t want to imply this is an easy call.
For me the central question is all of this is whether you foreground process (EA) or conclusion (AGI go well). It seems like the whole space is uniformly rushing to foreground the conclusion. It’s especially costly when 80k – the paragon of process discourse – decides to foreground the conclusion too. Who’s left as a source of wisdom foregrounding process?
I know you’e trying to do both. I guess you can call me pessimistic that even you (amazing Arden, my total fav) can pull it off.
Hey Matt,
I share several of the worries articulated in this post.
I think you’re wrong about how you characterise 80k’s strategic shift here, and want to try to correct the record on that point. I’m also going to give some concrete examples of things I’m currently doing, to illustrate a bit what I mean, & also include a few more personal comments.
(Context: I run the 80k web programme.)
Well put. And I agree that there are some concerning signs in this direction (though I’ve also had countervailing, inspiring experiences of AIS-focused people questioning whether some prevailing view about what to do in AIS is actually best for the world.)
I’d also love to see more cause prioritisation research. And it’s gonna be hard to both stay open enough to changing our minds about how to make the world better & to pursue our chosen means with enough focus to be effective. I think this challenge is fairly central to EA.
On 80k’s strategic shift:
You wrote:
How do we see the relationship between focusing on helping AGI go well and doing the most good?
It has always been the case that people and organisations need to find some intermediary outcome that comes before the good to point at strategically, some proxy for impact. Strategy is always about figuring out what’s gonna be the biggest/most cost-effective causal factor for that (i.e. means), & therefore the best proxy to pursue.
We used to focus on career changes not necessarily inside one specific cause area but it was still a proxy for the good. Now our proxy for the good is helping people work on making AGI go well, but our relationship to the good is the same as it was before: trying our best to point at it, trying to figure out the best means for doing so.
EA values & ideas are still a really important part of the strategy.
We wrote this in our post on the shift:
Though one might understandably worry that was paying lip service, just to reassure people. Let me talk about some internal recent goings-on off the top of my head, which hopefully do something to show we mean it:
1. Our internal doc on web programme strategy (i.e. the strategy for the programme I run) currently says that in order for our audience to actually have much more impact with their careers, engagement with the site ideally causes movement along at least 3[1] dimensions:
This makes re-designing the user flow post-strategic-shift a difficult balancing act/full of tradeoffs. How do we both quickly introduce people to AI being a big deal & urgent, and communicate EA ideas, plus help people shift their careers? Which do we do first?
We’re going to lose some simplicity (and some people, who don’t want to hear it) trying to do all this, and it will be reflected in the site being more complex than a strategy like “maximize for engagement or respectability” or “maximize for getting one idea across effectively” would recommend.
My view is that it’s worth it, because there is a danger of people just jumping into jobs that have “AI” or even “AI security/safety” in the name, without grappling with tough questions around what it actually means to help AGI go well or prioritising between options based on expected impact.
(On the term “EA mindset”—it’s really just a nickname; the thing I think we should care about is the focus on impact/use of the ideas.)
2. Our CEO (Niel Bowerman) spent several weeks recently with his top proactive priority helping figure out the top priorities within making AGI go well – i.e. which is more pressing (in the sense of where can additional talented people do the most marginal good) between issues like AI-enabled human coups, getting things right with rights and welfare of digital minds, and catastrophic misalignment. We argued about questions like “how big is the spread between issues within making AGI go well?” and “to what extent is AI rights and welfare an issue human has to get right before AI becomes incredibly powerful, due to potential lock-in effects of bad discourse or policies?”
So, we agree with this:
In other words, the analysis, as you say, is not done. It’s gonna be hecka hard to figure out “the particulars of what to do with AI.” And we do not “have it from here” – we need people thinking critically about this going forward so they stand the best chance of actually helping AGI go well, rather than just having a career in “something something AI.”
[1](I’m currently debating whether we should add a 4th: tactical sophistication about AI.)
I appreciate the dilemma and don’t want to imply this is an easy call.
For me the central question is all of this is whether you foreground process (EA) or conclusion (AGI go well). It seems like the whole space is uniformly rushing to foreground the conclusion. It’s especially costly when 80k – the paragon of process discourse – decides to foreground the conclusion too. Who’s left as a source of wisdom foregrounding process?
I know you’e trying to do both. I guess you can call me pessimistic that even you (amazing Arden, my total fav) can pull it off.