Strong upvote on this—it’s an issue that a lot of people have been discussing, and I found the post very clear!
There’s lots more to say, and I only had time to write something quickly but one consideration is about division of effort with respect to timelines to transformative AI. The longer AI timelines are, the more plausible principles-led EA movement-building looks.
Though I’ve updated a lot in the last couple of years on transformative-AI-in-the-next-decade, I think we should still put significant probability mass on “long” timelines (e.g. more than 30 years). For example, though Metaculus’s timelines have shortened dramatically — and now suggest a best-guess of superintelligence within a decade — the community forecast still puts a 10% chance that even “weakly general” AI is only developed after 2050, and it puts about a 20% chance that the time from weakly general AI to superintelligence is more than 30 years. AI progress could slow down a lot, perhaps because of regulation; there could also be some bottleneck we haven’t properly modelled that means that explosive growth never happens. (Relatedly, some things could happen that could make near-term effort on AI less important: AI alignment could end up being easy, or there could be sufficient effort on it, such that the most important questions are all about what happens post-AGI.)
In very short (e.g. <10yr) timelines worlds, then an AI-safety specific movement looks more compelling.
In long timelines worlds (e.g. >30 years), EA looks comparatively more promising for a few reasons:
EA is more adaptive over time. If our state of knowledge changes, or if the environment changes, such that X thing is no longer the best thing to do, then the EA recommendation changes, and (ideally) someone who follows EA reasoning will switch to the better thing.
This is much more likely to be relevant in long timelines worlds: there are decades for things to change (e.g. geopolitical changes, development of new technologies), for there to be further learning about issues relevant to cause-prioritisation, and for the world as a whole to invest dramatically more into AI safety.
Relatedly, EA is accessible to more people. A greater diversity of skills is more useful in long timelines worlds than they are in short timelines worlds, because of greater uncertainty about what will be of most value in long timelines worlds.
Principles-focused EA might have greater long-term compounding benefits (as a result of EAs doing outreach to recruit new EAs, some of whom do outreach to recruit new EAs, in a way similar to the compounding of financial investment over time) than cause-specific movements. I started trying to write this out quantitatively but ran out of time and there are a lot of subtle issues, so I’m not certain how exactly it shakes out; it’s something I’ll have to come back to. (Even better, someone better at maths than me could do this instead!)
This suggests that we might want to focus in particular on both short and fairly long timelines worlds (and pay less attention to in-the-middle timelines worlds). In short timelines worlds, we have outsized impact because the world hasn’t yet caught up to the importance of safely managing advanced AI and the issue is highly neglected. In fairly long timelines worlds, we get particular benefits from the long-run compounding of EA, and its diversity and adaptability.
This could look like some people focusing on AI for the next ten years, and then potentially switching after that time for the rest of their careers. Or, and in addition, it could look like some % of EA-minded people focusing wholly on AI, and some % focusing wholly on principles-first EA.
A final consideration concerns student groups in particular. For someone aged 18 — unless they are truly exceptional — it’ll probably be at least 5 years before they are able to meaningfully contribute to AI safety. If you think that the next 10 years are the particularly-likely time to get an intelligence explosion (because of an expected but unsustainable huge increase in investment into AI, as discussed by Carl Shulman here), then half of that opportunity can’t be capitalised upon by the 18-year-old. This gives an additional boost to the value of EA movement-building vs AI safety specific movement-building when it comes to campus outreach.
Overall, I agree that given the rapid progress in AI there should be some significant reallocation from EA movement-building to AI in particular, and I think we’re already seeing this happening. I’m currently unsure on whether I expect EA as a whole to ultimately under-correct or over-correct on recent AI developments.
Thanks for this comment; I found it helpful and agree with a lot of it. I expect the “university groups are disproportionately useful in long timelines worlds” point to be useful to a lot of people.
On this bit:
EA is more adaptive over time… This is much more likely to be relevant in long timelines worlds
This isn’t obvious to me. I would expect that short timeline worlds are just weirder and changing more rapidly in general, so being adaptive is more valuable.
Caricature example: in a short timeline world we have one year from the first sentient LLM to when we achieve value lock in, and in a long timeline world we have 100 years. In the former case EA seems more useful, because we can marshal a bunch of people to drop what they are doing and focus on digital sentience. In the latter case digital sentience probably just becomes an established field, without any need for EA.
There are counterbalancing factors (e.g. more of our resources should probably go towards AI safety, in short timelines worlds) but it seems pretty plausible to me that this nets out in favor of EA being more useful in shorter timelines.[1]
This suggests that we might want to focus in particular on both short and fairly long timelines worlds [...]
I’ve recently started thinking of this as a playing to your outs strategy, though without the small probabilities that that implies. One other factor in favor of believing that long timelines might happen, and those worlds might be good worlds to focus on, would be that starting very recently it’s begun to look possible to actually slow down AI. In those worlds, it’s presumably easier to pay an alignment tax, which makes those world more likely to survive.
For someone aged 18 — unless they are truly exceptional — it’ll probably be at least 5 years before they are able to meaningfully contribute to AI safety.
This might be true for technical. Less true for things like trying to organise a petition or drum up support for a protest.
Strong upvote on this—it’s an issue that a lot of people have been discussing, and I found the post very clear!
There’s lots more to say, and I only had time to write something quickly but one consideration is about division of effort with respect to timelines to transformative AI. The longer AI timelines are, the more plausible principles-led EA movement-building looks.
Though I’ve updated a lot in the last couple of years on transformative-AI-in-the-next-decade, I think we should still put significant probability mass on “long” timelines (e.g. more than 30 years). For example, though Metaculus’s timelines have shortened dramatically — and now suggest a best-guess of superintelligence within a decade — the community forecast still puts a 10% chance that even “weakly general” AI is only developed after 2050, and it puts about a 20% chance that the time from weakly general AI to superintelligence is more than 30 years. AI progress could slow down a lot, perhaps because of regulation; there could also be some bottleneck we haven’t properly modelled that means that explosive growth never happens. (Relatedly, some things could happen that could make near-term effort on AI less important: AI alignment could end up being easy, or there could be sufficient effort on it, such that the most important questions are all about what happens post-AGI.)
In very short (e.g. <10yr) timelines worlds, then an AI-safety specific movement looks more compelling.
In long timelines worlds (e.g. >30 years), EA looks comparatively more promising for a few reasons:
EA is more adaptive over time. If our state of knowledge changes, or if the environment changes, such that X thing is no longer the best thing to do, then the EA recommendation changes, and (ideally) someone who follows EA reasoning will switch to the better thing.
This is much more likely to be relevant in long timelines worlds: there are decades for things to change (e.g. geopolitical changes, development of new technologies), for there to be further learning about issues relevant to cause-prioritisation, and for the world as a whole to invest dramatically more into AI safety.
Relatedly, EA is accessible to more people. A greater diversity of skills is more useful in long timelines worlds than they are in short timelines worlds, because of greater uncertainty about what will be of most value in long timelines worlds.
Principles-focused EA might have greater long-term compounding benefits (as a result of EAs doing outreach to recruit new EAs, some of whom do outreach to recruit new EAs, in a way similar to the compounding of financial investment over time) than cause-specific movements. I started trying to write this out quantitatively but ran out of time and there are a lot of subtle issues, so I’m not certain how exactly it shakes out; it’s something I’ll have to come back to. (Even better, someone better at maths than me could do this instead!)
This suggests that we might want to focus in particular on both short and fairly long timelines worlds (and pay less attention to in-the-middle timelines worlds). In short timelines worlds, we have outsized impact because the world hasn’t yet caught up to the importance of safely managing advanced AI and the issue is highly neglected. In fairly long timelines worlds, we get particular benefits from the long-run compounding of EA, and its diversity and adaptability.
This could look like some people focusing on AI for the next ten years, and then potentially switching after that time for the rest of their careers. Or, and in addition, it could look like some % of EA-minded people focusing wholly on AI, and some % focusing wholly on principles-first EA.
A final consideration concerns student groups in particular. For someone aged 18 — unless they are truly exceptional — it’ll probably be at least 5 years before they are able to meaningfully contribute to AI safety. If you think that the next 10 years are the particularly-likely time to get an intelligence explosion (because of an expected but unsustainable huge increase in investment into AI, as discussed by Carl Shulman here), then half of that opportunity can’t be capitalised upon by the 18-year-old. This gives an additional boost to the value of EA movement-building vs AI safety specific movement-building when it comes to campus outreach.
Overall, I agree that given the rapid progress in AI there should be some significant reallocation from EA movement-building to AI in particular, and I think we’re already seeing this happening. I’m currently unsure on whether I expect EA as a whole to ultimately under-correct or over-correct on recent AI developments.
Thanks for this comment; I found it helpful and agree with a lot of it. I expect the “university groups are disproportionately useful in long timelines worlds” point to be useful to a lot of people.
On this bit:
This isn’t obvious to me. I would expect that short timeline worlds are just weirder and changing more rapidly in general, so being adaptive is more valuable.
Caricature example: in a short timeline world we have one year from the first sentient LLM to when we achieve value lock in, and in a long timeline world we have 100 years. In the former case EA seems more useful, because we can marshal a bunch of people to drop what they are doing and focus on digital sentience. In the latter case digital sentience probably just becomes an established field, without any need for EA.
There are counterbalancing factors (e.g. more of our resources should probably go towards AI safety, in short timelines worlds) but it seems pretty plausible to me that this nets out in favor of EA being more useful in shorter timelines.[1]
JP has made a related point to me, and deserves credit for a lot of this idea
I like this comment a whole bunch.
I’ve recently started thinking of this as a playing to your outs strategy, though without the small probabilities that that implies. One other factor in favor of believing that long timelines might happen, and those worlds might be good worlds to focus on, would be that starting very recently it’s begun to look possible to actually slow down AI. In those worlds, it’s presumably easier to pay an alignment tax, which makes those world more likely to survive.
This might be true for technical. Less true for things like trying to organise a petition or drum up support for a protest.