William_MacAskill

Karma: 7,065

William_MacAskill 23 Oct 2023 17:06 UTC
14 points
4 ∶ 0
on: Announcing Rob Gledhill as the new CEO of EV UK
Thank you so much for your work with EV over the last year, Howie! It was enormously helpful to have someone so well-trusted, with such excellent judgment, in this position. I’m sure you’ll have an enormous positive impact at Open Phil.

And welcome, Rob—I think it’s fantastic news that you’ve taken the role!

William_MacAskill 21 Sep 2023 15:39 UTC
126 points
6 ∶ 1
on: William_MacAskill’s Shortform
I mentioned a few months ago that I was planning to resign from the board of EV UK: I’ve now officially done so.
Since last November, I’ve been recused from the board on all matters associated with FTX and related topics, which has ended up being a large proportion of board business. (This is because the recusal affected not just decisions that were directly related to the collapse of FTX, but also many other decisions for which the way EV UK has been affected by the collapse of FTX was important context.) I know I initially said that I’d wait for there to be more capacity, but trustee recruitment has moved more slowly than I’d anticipated, and with the ongoing recusal I didn’t expect to add much capacity for the foreseeable future, so it felt like a natural time to step down.
It’s been quite a ride over the last eleven years. Effective Ventures has grown to a size far beyond what I expected, and I’ve felt privileged to help it on that journey. I deeply respect the rest of the board, and the leadership teams at EV, and I’m glad they’re at the helm.
Some people have asked me what I’m currently working on, and what my plans are. This year has been quite spread over a number of different things, including fundraising, helping out other EA-adjacent public figures, support for GPI, CEA and 80,000 Hours, writing additions to What We Owe The Future and helping with the print textbook version of utilitarianism.net that’s coming out next year. It’s also personally been the toughest year of my life; my mental health has been at its worst in over a decade, and I’ve been trying to deal with that, too.
At the moment, I’m doing three main things:
- Some public engagement, in particular around the WWOTF paperback and foreign language book launches and at EAGxBerlin. This has been and will be lower-key than the media around WWOTF last year, and more focused on in-person events; I’m also more focused on fundraising than I was before.
- Research into “trajectory changes”: in particular, ways of increasing the wellbeing of future generations other than ‘standard’ existential risk mitigation strategies, in particular on issues that arise even if we solve AI alignment, like digital sentience and the long reflection. I’m also doing some learning to try to get to grips on how to update properly on the latest developments in AI, in particular with respect to the probability of an intelligence explosion in the next decade, and on how hard we should expect AI alignment to be.
- Gathering information for what I should focus on next. In the medium term, I still plan to be a public proponent of EA-as-an-idea, which I think plays to my comparative advantage, and because I’m worried about people neglecting “EA qua EA”. If anything, all the crises faced by EA and by the world in the last year has reminded me of just how deeply I believe in EA as a project, and how the message of taking a thoughtful, humble, and scientific approach to doing good is more important than ever. The precise options I’m considering are still quite wide-ranging, including: a podcast and/or YouTube show and/or substack; a book on effective giving; a book on evidence-based living; or deeper research into the ethics and governance questions that arise even if we solve AI alignment. I hope to decide on that by the end of the year.
What links here?
- JWS's comment on EA is good, actually by Amy Labenz (30 Nov 2023 12:07 UTC; 16 points)

William_MacAskill 7 Sep 2023 13:03 UTC
81 points
14 ∶ 4
on: Nick Beckstead is leaving the Effective Ventures boards
(My personal views only, and like Nick I’ve been recused from a lot of board work since November.)

Thank you, Nick, for all your work on the Boards over the last eleven years. You helped steward the organisations into existence, and were central to helping them flourish and grow. I’ve always been impressed by your work ethic, your willingness to listen and learn, and your ability to provide feedback that was incisive, helpful, and kind.
Because you’ve been less in the limelight than me or Toby, I think many people don’t know just how crucial a role you played in EA’s early days. Though you joined shortly after launch, given all your work on it I think you were essentially a third cofounder of Giving What We Can; you led its research for many years, and helped build vital bridges with GiveWell and later Open Philanthropy. I remember that when you launched Giving What We Can: Rutgers, you organised a talk with I think over 500 people. It must still be one of the most well-attended talks that we’ve ever had within EA, and helped the idea of local groups get off the ground.
The EA movement wouldn’t have been the same without your service. It’s been an honour to have worked with you.

William_MacAskill 4 Aug 2023 10:40 UTC
170 points
57 ∶ 10
on: University EA Groups Need Fixing
Hey,
I’m really sorry to hear about this experience. I’ve also experienced what feels like social pressure to have particular beliefs (e.g. around non-causal decision theory, high AI x-risk estimates, other general pictures of the world), and it’s something I also don’t like about the movement. My biggest worries with my own beliefs stem around the worry that I’d have very different views if I’d found myself in a different social environment. It’s just simply very hard to successfully have a group of people who are trying to both figure out what’s correct and trying to change the world: from the perspective of someone who thinks the end of the world is imminent, someone who doesn’t agree is at best useless and at worst harmful (because they are promoting misinformation).
In local groups in particular, I can see how this issue can get aggravated: people want their local group to be successful, and it’s much easier to track success with a metric like “number of new AI safety researchers” than “number of people who have thought really deeply about the most pressing issues and have come to their own well-considered conclusions”.
One thing I’ll say is that core researchers are often (but not always) much more uncertain and pluralist than it seems from “the vibe”. The second half of Holden Karnofsky’s recent 80k blog post is indicative. Open Phil splits their funding across quite a number of cause areas, and I expect that to continue. Most of the researchers at GPI are pretty sceptical of AI x-risk. Even among people who are really worried about TAI in the next decade, there’s normally significant support (whether driven by worldview diversification or just normal human psychology) for neartermist or other non-AI causes. That’s certainly true of me. I think longtermism is highly non-obvious, and focusing on near-term AI risk even more so; beyond that, I think a healthy EA movement should be highly intellectually diverse and exploratory.
What should be done? I have a few thoughts, but my most major best guess is that, now that AI safety is big enough and getting so much attention, it should have its own movement, separate from EA. Currently, AI has an odd relationship to EA. Global health and development and farm animal welfare, and to some extent pandemic preparedness, had movements working on them independently of EA. In contrast, AI safety work currently overlaps much more heavily with the EA/rationalist community, because it’s more homegrown.
If AI had its own movement infrastructure, that would give EA more space to be its own thing. It could more easily be about the question “how can we do the most good?” and a portfolio of possible answers to that question, rather than one increasingly common answer — “AI”.
At the moment, I’m pretty worried that, on the current trajectory, AI safety will end up eating EA. Though I’m very worried about what the next 5-10 years will look like in AI, and though I think we should put significantly more resources into AI safety even than we have done, I still think that AI safety eating EA would be a major loss. EA qua EA, which can live and breathe on its own terms, still has huge amounts of value: if AI progress slows; if it gets so much attention that it’s no longer neglected; if it turns out the case for AI safety was wrong in important ways; and because there are other ways of adding value to the world, too. I think most people in EA, even people like Holden who are currently obsessed with near-term AI risk, would agree.
What links here?

William_MacAskill 2 Aug 2023 14:30 UTC
5 points
0 ∶ 0
on: Thoughts on far-UVC after working in the field for 8 months
This was an extremely helpful post, thank you!

William_MacAskill 2 Aug 2023 14:27 UTC
15 points
2 ∶ 0
on: What do you think the vision is behind our biggest critics?
This isn’t answering the question you ask (sorry), but one possible response to this line of criticism is for some people within EA / longtermism to more clearly state what vision of the future they are aiming towards. Because this tends not to happen, it means that critics can attribute particular visions to people that they don’t have. In particular, critics of WWOTF often thought that I was trying to push for some particular narrow vision of the future, whereas really the primary goal, in my mind at least, is to keep our options open as much as possible, and make moral progress in order to figure out what sort of future we should try to create.

Here are a couple of suggestions for positive visions. These are what I’d answer if asked: “What vision of the future are you aiming towards?”:

”Procedural visions”
(Name options: Viatopia—representing the idea of a waypoint, and of keeping multiple paths open—though this mixes latin and greek roots. Optiotopia, though is a mouthful and mixes latin and greek roots. Related ideas: existential security, the long reflection.)

These doesn’t have some vision of what we ultimately want to achieve. Instead they propose a waypoint that we’d want to achieve, as a step on the path to a good future. That waypoint would involve: (i) ending all obvious grievous contemporary harms, like war, violence and unnecessary suffering; (ii) reducing existential risk down to a very low level; (iii) securing a deliberative process for humanity as a whole, so that we make sufficient moral progress before embarking on potentially-irreversible actions like space settlement.

The hope could be that almost everyone could agree on this as a desirable waypoint.

”Utopia for everyone”
(Name options: multitopia or pluritopia, but this mixes latin and greek roots; polytopia, but this is the name of a computer game. Related idea: Paretopia.)

This vision is where a great diversity of different visions of the good are allowed to happen, and people have choice about what sort of society they want to live in. Environmentalists could preserve Earth’s ecosystems; others can build off-world societies. Liberals and libertarians can create a society where everyone is empowered to act autonomously, pursuing their own goals; lovers of knowledge can build societies devoted to figuring out the deepest truths of the universe; philosophical hedonists can create societies devoted to joy, and so on.

The key insight, here, is that there’s just a lot of available stuff in the future, and that scientific, social and moral progress will potentially enable us to produce great wealth with that stuff (if we don’t destroy the world first, or suffer value lock-in). Plausibly, if we as a global society get our act together, the large majority of moral perspectives can get most of what they want.

Like the procedural visions, spelling this vision out more could have great benefits today, via greater collaboration: if we could agree that this is what we’ll aim for, at least in part, then we could reduce the chance of some person or people with some narrow view trying to grab power for itself.

(I write about these a little bit about both of these idea in a fictional short story, here.)

I’d welcome name ideas for these, especially the former. My best guesses so far are “viatopia” and “multitopia”, but I’m not wedded to them and I haven’t spent lots of time on naming. I don’t think that the -topia suffix is strictly necessary.
What links here?
- James Herbert's comment on An Elephant in the Community Building room by Kaleem (21 Aug 2023 19:58 UTC; 29 points)
- James Herbert's comment on James Herbert’s Quick takes by James Herbert (15 Sep 2023 9:25 UTC; 25 points)

William_MacAskill 2 Aug 2023 13:30 UTC
2 points
0 ∶ 0
in reply to: Jack Malde’s comment on: Shaping Humanity’s Longterm Trajectory
This is a good point, and it’s worth pointing out that increasing $¯ v$ is always good whereas increasing $τ$ is only good if the future is of positive value. So risk aversion reduces the value of increasing $τ$ relative to increasing $¯ v$ , provided we put some probability on a bad future.
Agree this is worth pointing out! I’ve a draft paper that goes into some of this stuff in more detail, and I make this argument.

Another potential argument for trying to improve $¯ v$ is that, plausibly at least, the value lost as a result of the gap between expected- $¯ v$ and best-possible- $¯ v$ is greater that the value lost as a result of the gap between expected- $τ$ and best-possible- $τ$ . So in that sense the problem that expected- $¯ v$ is not as high as it could be is more “important” (in the ITN sense) than the problem that the expected $τ$ is not as high as it could be.

William_MacAskill 31 Jul 2023 10:31 UTC
23 points
4 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Existential risk, and an alternative framework

One common issue with “existential risk” is that it’s so easy to conflate it with “extinction risk”. It seems that even you end up falling into this use of language. You say: “if there were 20 percentage points of near-term existential risk (so an 80 percent chance of survival)”. But human extinction is not necessary for something to be an existential risk, so 20 percentage points of near-term existential risk doesn’t entail an 80 percent chance of survival. (Human extinction may also not be sufficient for existential catastrophe either, depending on how one defines “humanity”))
Relatedly, “existential risk” blurs together two quite different ways of affecting the future. In your model: $V = ¯ v τ$ . (That is: The value of humanity’s future is the average value of humanity’s future over time multiplied by the duration of humanity’s future.)
This naturally lends itself to the idea that there are two main ways of improving the future: increasing $¯ v$ and increasing $τ$ .
In What We Owe The Future I refer to the latter as “ensuring civilisational survival” and the former as “effecting a positive trajectory change”. (We’ll need to do a bit of syncing up on terminology.)
I think it’s important to keep these separate, because there are plausible views on which affecting one of these is much more important than affecting the other.
Some views on which increasing $¯ v$ is more important:
- If the future is of zero or net-negative value
- If large drops in future population size are not of enormous importance (e.g. the average view, variable-value views)
- If nonhuman-originating civilisation would use the resources that we would use, and is similarly good
Some views on which increasing $τ$ is more important:
- If there’s a “low” upper bound on value, which we expect almost all future civilisations to meet
- If think that moral convergence, conditional on survival, is very likely
What’s more, changes to $τ$ are plausible binary, but changes to $¯ v$ are not. Plausibly, most probability mass is on $τ$ being small (we go extinct in the next thousand years) or very large (we survive for billions of years or more). But, assuming for simplicity that there’s a “best possible” and “worst possible” future, $¯ v$ could take any value between 100% and −100%. So focusing only on “drastic” changes, as the language of “existential risk” does, makes sense for changes to $τ$ , but not for changes to $¯ v$ .

William_MacAskill 31 Jul 2023 10:30 UTC
4 points
0 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Flow vs fixed resources
In footnote 14 you say: “It has also been suggested (Sandberg et al 2016, Ord 2021) that the ultimate physical limits may be set by a civilisation that expands to secure resources but doesn’t use them to create value until much later on, when the energy can be used more efficiently. If so, one could tweak the framework to model this not as a flow of intrinsic value over time, but a flow of new resources which can eventually be used to create value.”
This feels to me that it would really be changing the framework considerably, rather than just a “tweak”.
For example, consider a “speed up” with an endogenous end time. On the original model, this decreases total value (assuming the future is overall good). But if we’re talking about gaining a pot of fixed resources, speeding up progress forever doesn’t change total value.

William_MacAskill 31 Jul 2023 10:30 UTC
14 points
3 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Humanity
Like the other commenter says, I feel worried that v(.) refers to the value of “humanity”. For similar reasons, I feel worried that existential risk is defined in terms of humanity’s potential.
One issue is that it’s vague what counts as “humanity”. Homo sapiens count, but what about:
- A species that Homo sapiens evolves into
- “Uploaded” humans
- “Aligned” AI systems
- Non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes
I’m not sure where you draw the line, or if there is a principled place to draw the line.
A second issue is that “humanity” doesn’t include the value of:
- Earth-originating but nonhuman civilisations, for example if Homo sapiens go extinct, but some other species later evolves that has technological capability.
- Non-Earth-originating alien civilisation.
And, depending on how “humanity” is defined, it may not include non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes.
I tried to think about how to incorporate this into your model, but ultimately I think it’s hard without it becoming quite unintuitive.
And I think these adjustments are potentially non-trivial. I think one could reasonably hold, for example, that the probability of a technologically-capable species evolving, if Homo sapiens goes extinct, is 90%, that non-Earth-originating alien civilisations settling the solar systems that we would ultimately settle is also 90%, and that such civilisations would have similar value to human-originating civilisation.
(They also change how you should think about longterm impact. If alien civilisations will settle the Milky Way (etc) anyway, then preventing human extinction is actually about changing how interstellar resources are used, not whether they are used at all .)
And I think it means we miss out on some potentially important ways of improving the future. For example, consider scenarios where we fail on alignment. There is no “humanity”, but we can still make the future better or worse. A misaligned AI system that promotes suffering (or promotes something that involves a lot of suffering) is a lot worse than an AI system that promotes something valueless.

William_MacAskill 31 Jul 2023 10:29 UTC
4 points
0 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Humanity
Like the other commenter says, I feel worried that v(.) refers to the value of “humanity”. For similar reasons, I feel worried that existential risk is defined in terms of humanity’s potential.
One issue is that it’s vague what counts as “humanity”. Homo sapiens count, but what about:
- A species that Homo sapiens evolves into
- “Uploaded” humans
- “Aligned” AI systems
- Non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes
I’m not sure where you draw the line, or if there is a principled place to draw the line.
A second issue is that “humanity” doesn’t include the value of:
- Earth-originating but nonhuman civilisations, for example if Homo sapiens go extinct, but some other species later evolves that has technological capability.
- Non-Earth-originating alien civilisation.
And, depending on how “humanity” is defined, it may not include non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes.
I tried to think about how to incorporate this into your model, but ultimately I think it’s hard without it becoming quite unintuitive.
And I think these adjustments are potentially non-trivial. I think one could reasonably hold, for example, that the probability of a technologically-capable species evolving, if Homo sapiens goes extinct, is 90%, that non-Earth-originating alien civilisations settling the solar systems that we would ultimately settle is also 90%, and that such civilisations would have similar value to human-originating civilisation.
(They also change how you should think about longterm impact. If alien civilisations will settle the Milky Way (etc) anyway, then preventing human extinction is actually about changing how interstellar resources are used, not whether they are used at all .)
And I think it means we miss out on some potentially important ways of improving the future. For example, consider scenarios where we fail on alignment. There is no “humanity”, but we can still make the future better or worse. A misaligned AI system that promotes suffering (or promotes something that involves a lot of suffering) is a lot worse than an AI system that promotes something valueless.

William_MacAskill 31 Jul 2023 10:29 UTC
3 points
0 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Humanity
Like the other commenter says, I feel worried that v(.) refers to the value of “humanity”. For similar reasons, I feel worried that existential risk is defined in terms of humanity’s potential.
One issue is that it’s vague what counts as “humanity”. Homo sapiens count, but what about:
- A species that Homo sapiens evolves into
- “Uploaded” humans
- “Aligned” AI systems
- Non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes
I’m not sure where you draw the line, or if there is a principled place to draw the line.
A second issue is that “humanity” doesn’t include the value of:
- Earth-originating but nonhuman civilisations, for example if Homo sapiens go extinct, but some other species later evolves that has technological capability.
- Non-Earth-originating alien civilisation.
And, depending on how “humanity” is defined, it may not include non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes.
I tried to think about how to incorporate this into your model, but ultimately I think it’s hard without it becoming quite unintuitive.
And I think these adjustments are potentially non-trivial. I think one could reasonably hold, for example, that the probability of a technologically-capable species evolving, if Homo sapiens goes extinct, is 90%, that non-Earth-originating alien civilisations settling the solar systems that we would ultimately settle is also 90%, and that such civilisations would have similar value to human-originating civilisation.
(They also change how you should think about longterm impact. If alien civilisations will settle the Milky Way (etc) anyway, then preventing human extinction is actually about changing how interstellar resources are used, not whether they are used at all .)
And I think it means we miss out on some potentially important ways of improving the future. For example, consider scenarios where we fail on alignment. There is no “humanity”, but we can still make the future better or worse. A misaligned AI system that promotes suffering (or promotes something that involves a lot of suffering) is a lot worse than an AI system that promotes something valueless.

William_MacAskill 31 Jul 2023 10:27 UTC
22 points
4 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Enhancements
I felt like the paper gave enhancements short shrift. As you note, they are the intervention most plausibly competes with existential risk reduction, as they scale with $¯ v$ $τ$ .
You say: “As with many of these idealised changes, they face the challenge of why this wouldn’t happen eventually, even without the current effort. I think this is a serious challenge for many proposed enhancements.”
I agree that this is a serious challenge, and that one should have more starting scepticism about the persistence of enhancements compared with extinction risk reduction.
But there is a compelling response as to why the improvements to v(.) don’t happen anyway: which is that future agents don’t want them to happen. Taking a simplified example: In one scenario, society is controlled by hedonists; in another scenario, society is controlled by preference-satisfactionists. But, let us assume, the hedonists do in fact produce more value. I don’t think we should necessarily expect the preference-satisfactionists to switch to being hedonists, if they don’t want to switch.
(Indeed, that’s the explanation of why AI risk is so worrying from a longterm perspective. Future AI agents might want something valueless, and choose not to promote what’s actually of value.)
So it seems to me that your argument only works if one assumes a fairly strong form of moral internalism, that future agents will work out the moral truth and then act on that basis.

William_MacAskill 31 Jul 2023 10:24 UTC
8 points
0 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Gains
“While the idea of a gain is simple — a permanent improvement in instantaneous value of a fixed size — it is not so clear how common they are.”
I agree that gains aren’t where the action is, when it comes to longterm impact. Nonetheless, here are some potential examples:
- Species loss / preservation of species
- Preservation of historical information
- Preservation of Earth’s ecosystem
- Preservation of areas of natural importance, like Olympus Mons
These plausibly have two sources of longterm value. The first is that future agents might have slightly better lives as a result: perhaps one in a billion future people are willing to pay the equivalent of $1 in order to be able to see a real-life panda, or to learn about the life and times of a historically interesting figure. This scales in future population size, so is probably an “enhancement” rather than a “gain” (though it depends a little on one’s population ethics).
The second is if these things have intrinsic value. If so, then perhaps they provide a fixed amount over value at any time. That really would be a gain.
Another possible gain is preventing future wars that destroy resources. Suppose that, for example, there’s a war between two factions of future interstellar civilisation, and a solar system is destroyed as a result. That would be a loss.

William_MacAskill 31 Jul 2023 10:23 UTC
10 points
2 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Speed-ups
You write: “How plausible are speed-ups? The broad course of human history suggests that speed-ups are possible,” and, “though there is more scholarly debate about whether the industrial revolution would have ever happened had it not started in the way it did. And there are other smaller breakthroughs, such as the phonetic alphabet, that only occurred once and whose main effect may have been to speed up progress. So contingent speed-ups may be possible.”
This was the section of the paper I was most surprised / confused by. You seemed open to speed-ups, but it seems to me that a speed-up for the whole rest of the future is extremely hard to do.
The more natural thought is that, at some point in time, we either hit a plateau, or hit some hard limit of how fast v(.) can grow (perhaps driven by cubic or quadratic growth as future people settle the stars). But if so, then what looks like a “speed up” is really an advancement.
I really don’t see what sort of action could result in a speed-up across the whole course of v(.), unless the future is short (e.g. 1000 years).

William_MacAskill 31 Jul 2023 10:23 UTC
8 points
0 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Advancements
I broadly agree with the upshots you draw, but here are three points that make things a little more complicated:
Continued exponential growth
As you note: (i) if v(.) continues exponentially, then advancements can compete with existential risk reduction; (ii) such continued exponential growth seems very unlikely.
However, it seems above 0 probability that we could have continued exponential growth in v(.) forever, including at the end point (and perhaps even at a very fast rate, like doubling every year). And, if so, then the total value of the future would be dramatically greater than if v(.) increases cubically and/or eventually plateaus. So, one might argue: this is where most of the expected value is. So advancements, in expectation, are competitive with existential risk reduction.
Now, I hate this argument: it seems like it’s falling prey to “fanaticism” in the technical sense, letting our expected value calculations be driven by extremely small probabilities.
But it at least shows that, when thinking about longterm impact, we need to make some tough judgment calls about which possibilities we should ignore on the grounds of driving “fanatical”-seeming conclusions, even while only considering only finite amounts of value.

Aliens
Elsewhere, you note the loss of galaxies due to the expansion of the universe, which means that ~one five-billionth of the universe per year becomes inaccessible.
But if the “grabby aliens” model is correct, then that number is too low. By my calculation, if we meet grabby alien civilisations in, for example, one billion years (which I think is about the median estimate from the grabby aliens model), then we “lose” approximately 1 millionth of accessible resources to alien civilisations every year. This is still very small, but three orders of magnitude higher than what we get by just looking at the expansion of the universe.
(Then there’s a hard and relevant question about the value of alien civilisation versus the value of human-originating civilisation.)

Length of advancements / delays
“An advancement of an entire year would be very difficult to achieve: it may require something comparable to the entire effort of all currently existing humans working for a year.”
This is true when considering “normal” economic trajectories. But I think there are some things we could do that could cause much greater advancements or delays. A few examples:
- Preventing the collapse of civilisation. If there were a catastrophe so severe that civilisation went back to pre-industrial or even pre-agricultural levels of technology, this could result in a delay, in expectation, of thousands of years.
- Preventing long-term stagnation. Rapid developments in AI are making this seem increasingly unlikely, but it’s possible at least that the world enters a period of long-term stagnation. Combined, potentially, with other catastrophes, this could lead to a delay of hundreds or thousands of years.
- The world institutes something like a Long Reflection, delaying space settlement by a thousand years.
Combining this with the “grabby aliens” point, there is potentially 0.1% of the value of the future that could be gained from preventing delays (1000 years * 1 millionth loss per year). Still much lower than the loss of value from anthropogenic existential risks, but higher than from non-anthropogenic risks. It’s enough that I think it’s not really action-relevant, but so at the same time not totally negligible.

William_MacAskill 31 Jul 2023 10:21 UTC
16 points
1 ∶ 0
on: Shaping Humanity’s Longterm Trajectory
Hi Toby,

Thanks so much for doing and sharing this! It’s a beautiful piece of work—characteristically clear and precise.
Remarkably, I didn’t know you’d been writing this, or had an essay coming out that volume! Especially given that I’d been doing some similar work, though with a different emphasis.
I’ve got a number of thoughts, which I’ll break into different comments.

William_MacAskill 7 Jul 2023 17:01 UTC
39 points
11 ∶ 0
on: On focusing resources more on particular fields vs. EA per se—considerations and takes
Strong upvote on this—it’s an issue that a lot of people have been discussing, and I found the post very clear!
There’s lots more to say, and I only had time to write something quickly but one consideration is about division of effort with respect to timelines to transformative AI. The longer AI timelines are, the more plausible principles-led EA movement-building looks.
Though I’ve updated a lot in the last couple of years on transformative-AI-in-the-next-decade, I think we should still put significant probability mass on “long” timelines (e.g. more than 30 years). For example, though Metaculus’s timelines have shortened dramatically — and now suggest a best-guess of superintelligence within a decade — the community forecast still puts a 10% chance that even “weakly general” AI is only developed after 2050, and it puts about a 20% chance that the time from weakly general AI to superintelligence is more than 30 years. AI progress could slow down a lot, perhaps because of regulation; there could also be some bottleneck we haven’t properly modelled that means that explosive growth never happens. (Relatedly, some things could happen that could make near-term effort on AI less important: AI alignment could end up being easy, or there could be sufficient effort on it, such that the most important questions are all about what happens post-AGI.)
In very short (e.g. <10yr) timelines worlds, then an AI-safety specific movement looks more compelling.
In long timelines worlds (e.g. >30 years), EA looks comparatively more promising for a few reasons:
- EA is more adaptive over time. If our state of knowledge changes, or if the environment changes, such that X thing is no longer the best thing to do, then the EA recommendation changes, and (ideally) someone who follows EA reasoning will switch to the better thing.
  - This is much more likely to be relevant in long timelines worlds: there are decades for things to change (e.g. geopolitical changes, development of new technologies), for there to be further learning about issues relevant to cause-prioritisation, and for the world as a whole to invest dramatically more into AI safety.
- Relatedly, EA is accessible to more people. A greater diversity of skills is more useful in long timelines worlds than they are in short timelines worlds, because of greater uncertainty about what will be of most value in long timelines worlds.
- Principles-focused EA might have greater long-term compounding benefits (as a result of EAs doing outreach to recruit new EAs, some of whom do outreach to recruit new EAs, in a way similar to the compounding of financial investment over time) than cause-specific movements. I started trying to write this out quantitatively but ran out of time and there are a lot of subtle issues, so I’m not certain how exactly it shakes out; it’s something I’ll have to come back to. (Even better, someone better at maths than me could do this instead!)
This suggests that we might want to focus in particular on both short and fairly long timelines worlds (and pay less attention to in-the-middle timelines worlds). In short timelines worlds, we have outsized impact because the world hasn’t yet caught up to the importance of safely managing advanced AI and the issue is highly neglected. In fairly long timelines worlds, we get particular benefits from the long-run compounding of EA, and its diversity and adaptability.
This could look like some people focusing on AI for the next ten years, and then potentially switching after that time for the rest of their careers. Or, and in addition, it could look like some % of EA-minded people focusing wholly on AI, and some % focusing wholly on principles-first EA.
A final consideration concerns student groups in particular. For someone aged 18 — unless they are truly exceptional — it’ll probably be at least 5 years before they are able to meaningfully contribute to AI safety. If you think that the next 10 years are the particularly-likely time to get an intelligence explosion (because of an expected but unsustainable huge increase in investment into AI, as discussed by Carl Shulman here), then half of that opportunity can’t be capitalised upon by the 18-year-old. This gives an additional boost to the value of EA movement-building vs AI safety specific movement-building when it comes to campus outreach.
Overall, I agree that given the rapid progress in AI there should be some significant reallocation from EA movement-building to AI in particular, and I think we’re already seeing this happening. I’m currently unsure on whether I expect EA as a whole to ultimately under-correct or over-correct on recent AI developments.

William_MacAskill 27 Jun 2023 14:30 UTC
15 points
3 ∶ 1
in reply to: lincolnq’s comment on: Decision-making and decentralisation in EA
Thanks! I agree that we are already (kind of) doing most of these things. So the question is whether further centralisation is tractable (and desirable). Like I say, it seems to me the big thing is if there’s someone, or some group of people, who really wants to make that further centralisation to happen. (E.g. I don’t think I’d be the right person even if I wanted to do it.)

Some things I didn’t understand from your bullet-point list:

Having most of the resources come from one place

By “resources” do you primarily mean funding? (I’ll assume yes.)

Declaring that a certain type of resource is the “official” resource which we “recommend”
Here, by “resource” do you here mean information (books, etc)? (I’ll assume yes.)

Stop publishing critical info publicly, relying on whisper networks to get the word out about things

This doesn’t clearly map onto “centralised” vs “decentralised” to me?

Of your list, the first two bullet-points seem non-desirable to me in a totally ideal world. But of course having lots of funding from OP is much, much better than not having the funding at all!

The second two bullet points seem good to have, even if EA were more decentralised than it is now.

William_MacAskill 27 Jun 2023 13:59 UTC
9 points
2 ∶ 1
in reply to: Larks’s comment on: Decision-making and decentralisation in EA
Maybe what’s going on here is vagueness, and me being unclear.
Jeff’s clarification is helpful. I could have just dropped “part of the EA movement or” and the sentence would have been clearer and better.
The key thing I was meaning in this context is: “Is a project engaging in EA movement-building, such that it would make sense that they at least potentially have obligations or responsibilities towards the EA movement as a whole?” The answer is clearly “no” for LEEP (for example), and “yes” for CEA. On that question, I would say “no” for GovAI, Longview or Wytham, though I’ll caveat I don’t lead any of those projects so that’s just my perception.

“To make it even more clear, many of these projects used to be part of CEA.”

If you mean CEA-the-project (not CEA-the-former-legal-entity), that’s true of EA Funds and GWWC (though GWWC predates CEA and was separate from it prior to merging and then separating again), but not the others.
If you mean CEA-the-legal-entity, the name change from CEA UK and US → Effective Ventures UK and US was when the legal entity started housing more projects that aren’t focused on EA movement-building, and was done in part so that a project’s being housed at EV UK or EV US wouldn’t be understood to mean it was engaged in EA movement-building. (Clearly we should have communicated better about this, as it’s led to a lot of confusion.)

William_MacAskill

Existential risk, and an alternative framework

Flow vs fixed resources

Humanity

Humanity

Humanity

Enhancements

Gains

Speed-ups

Advancements