Toby_Ord

Karma: 3,315

Toby_Ord Aug 2, 2023, 6:00 PM
2 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
The term ‘humanity’ is definitely intended to be interpreted broadly. I was more explicit about this in The Precipice and forgot to reiterate it in this paper. I certainly want to include any worthy successors to homo sapiens. But it may be important to understand the boundary of what counts. A background assumption is that the entities are both moral agents and moral patients — capable of steering the future towards what matters and for being intrinsically part of what matters. I’m not sure if those assumptions are actually needed, but they were guiding my thought.
I definitely don’t intend to include alien civilisations or future independent earth-originating intelligent life. The point is to capture the causal downstream consequences of things in our sphere of control. So the effects of us on alien civilisations should be counted and any effects we have of on whether any earth species evolves after us, but it isn’t meant to be a graph of all value in the universe. My methods wouldn’t work for that, as we can’t plausibly speed that up, or protect it all etc (unless we were almost all the value anyway).

Toby_Ord Aug 2, 2023, 4:04 PM
4 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
I think I may have been a bit too unclear about which things I found more promising than others. Ultimately the chapter is more about the framework, with a few considerations added for and against each of the kinds of idealised changes, and no real attempt to be complete about those or make all-things-considered judgments about how to rate them. Of the marginal interventions I discuss, I am most excited about existential-risk reduction, followed by enhancements.
As to your example, I feel that I might count the point where the world became permanently controlled by preference utilitarians an existential catastrophe — locking in an incorrect moral system forever. In general, lock-in is a good answer for why things might not happen later if they don’t happen now, but too much lock-in of too big a consequence is what I call an existential catastrophe. So your example is good as a non-*extinction* case, but to find a non-existential one, you may need to look for examples that are smaller in size, or perhaps only partly locked-in?

Toby_Ord Aug 2, 2023, 2:53 PM
7 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
I think “open to speed-ups” is about right. As I said in the quoted text, my conclusion was that contingent speed-ups “may be possible”. They are not an avenue for long-term change that I’m especially excited about. The main reason for including them here was to distinguish them from advancements (these two things are often run together) and because they fall out very natural as one of the kinds of natural marginal change to the trajectory whose value doesn’t depend on the details of the curve.
That said, it sounds like I think they are a bit more likely to be possible than you do. Here are some comments on that.
One thing is that it is easier to have a speed-up relative to another trajectory than to have one that is contingent — which wouldn’t have happened otherwise. Contingent speed-ups are the ones of most interest to longtermists, but those that are overdetermined to happen are still relevant to studying the value of the future and where it comes from. e.g. if the industrial revolution was going to happen anyway, then the counterfactual value of it happening in the UK in the late 1700s may be small, but it is still an extremely interesting event in terms of dramatically changing the rate of progress from then onwards compared to a world without an industrial revolution.
The more natural thought is that, at some point in time, we either hit a plateau, or hit some hard limit of how fast v(.) can grow (perhaps driven by cubic or quadratic growth as future people settle the stars).
Even if v(.) hits a plateau, you can still have a speed-up, it is just that it only has an impact on the value achieved before we would have hit the value anyway. That could be a large change (e.g. if the plateau isn’t reached in the first 1% of our lifetime), but even if it isn’t, that doesn’t stop this being a speed-up, it is just that changing the speed of some things isn’t very valuable, which is a result that is revealed by the framework.
Suppose v(.) stops growing exponentially with progress and most of its increase is then governed by growing cubically as a humanity’s descendants settle the cosmos. Expanding faster (e.g. by achieving a faster travel speed of 76%c instead of 75%c) could then count as a speed-up. That said, it is difficult for this to be a contingent speed-up, as it raises the question of why (when this was the main determinant of value) would this improvement not be implemented at a later date? And it is also difficult in this case to see how any actions now could produce such a speed-up.
Overall, I don’t see them as a promising avenue for current efforts to target, and more of a useful theoretical tool, but I also don’t think there are knockdown arguments against them being a practical avenue.

Toby_Ord Aug 2, 2023, 2:06 PM
3 points
0 ∶ 0
in reply to: Muireall’s comment on: Shaping Humanity’s Longterm Trajectory
I think that is indeed a good analogy. In the chapter, I focus on questions of marginal changes to the curve where there is a small shock to some parameter right now, and then its effects play out. If one is instead asking questions that are less about what we now could do to change the future, but are about how humanity over deep time should act to have a good future, then I think the optimal control techniques could be very useful. And I wouldn’t be surprised if in attempting to understand the dynamics, there were lessons for our present actions too.

Toby_Ord Aug 2, 2023, 2:02 PM
7 points
1 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
This naturally lends itself to the idea that there are two main ways of improving the future: increasing v̅ and increasing $τ$ .
I think this is a useful two factor model, though I don’t quite think of avoiding existential risk just as increasing τ. I think of it more as increasing the probability that it doesn’t just end now, or at some other intermediate point. In my (unpublished) extensions of this model that I hint at in the chapter, I add a curve representing the probability of surviving to time t (or beyond), and then think of raising this curve as intervening on existential risk.

Toby_Ord Aug 2, 2023, 1:56 PM
7 points
1 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
One common issue with “existential risk” is that it’s so easy to conflate it with “extinction risk”. It seems that even you end up falling into this use of language. You say: “if there were 20 percentage points of near-term existential risk (so an 80 percent chance of survival)”. But human extinction is not necessary for something to be an existential risk, so 20 percentage points of near-term existential risk doesn’t entail an 80 percent chance of survival.
In this case I meant ‘an 80 percent chance of surviving the threat with our potential intact’, or of ‘our potential surviving the threat’.
While this framework is slightly cleaner with extinction risk instead of existential risk (i.e. the curve may simply stop), it can also work with existential risk as while the curve continues after some existential catastrophes, it usually only sweeps out an a small area. This does raise a bigger issue if the existential catastrophe is that we end up with a vastly negative future, as then the curve may continue in very important ways after that point. (There are related challenges pointed out by another commenter where out impacts on the intrinsic value of other animals may also continue after our extinction.) These are genuine challenges (or limitations) for the current model. One definitely can overcome them, but the question would be the best way to do so while maintaining analytic tractability.

Toby_Ord Jul 31, 2023, 4:20 PM
7 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
Length of advancements / delays
Good point about the fact that I was focusing on some normal kind of economic trajectory when assessing the difficulty of advancements and delays. Your examples are good, as is MichaelStJules’ comment about how changing the timing of transformative AI might act as an advancement.

Toby_Ord Jul 31, 2023, 4:16 PM
7 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
>Aliens
You are right that the presence or absence of alien civilisations (especially those that expand to settle very large regions) can change things. I didn’t address this explicitly because (1) I think it is more likely that we are alone in the affectable universe, and (2) there are many different possible dynamics for multiple interacting civilisations and it is not clear what is the best model. But it is still quite a plausible possibility and some of the possible dynamics are likely enough and simple enough that they are worth analysing.
I’m not sure about the details of your calculation, but have thought a bit about it in terms of Jay Olson’s model of cosmological expanding civilisations (which is roughly how Anders and I think of it, and similar to model Hanson et al independently came up with). On this model, if civilisations expand at a constant fraction of c (which we can call f), the average distance between independently arising civilisations is D light years, and civilisations permanently hold all locations they reach first, then delaying by 1 year loses roughly 3f/D of the resources they could have reached. So if D were 1 billion light years, and f were close to 1, then a year’s delay would lose roughly 1 part in 300 million of the resources. So on my calculation, it would need to be an average distance of about 3 million light years or less, to get the fraction lost down to 1 part in 1 million. And at that point, the arrangement of galaxies makes a big difference. But this was off-the-cuff and I could be overlooking something.
What links here?
- Longtermist implications of aliens Space-Faring Civilizations—Introduction by Maxime Riché 🔸 (Feb 21, 2025, 12:07 PM; 43 points)
- Longtermist implications of aliens Space-Faring Civilizations—Introduction by Maxime Riché (LessWrong; Feb 21, 2025, 12:08 PM; 21 points)

Toby_Ord Jul 31, 2023, 1:55 PM
7 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
>Continued exponential growth
I agree that there is a kind of Pascallian possibility of very small probabilities of exponential growth in value going for extremely long times. If so, then advancements scale in value with v-bar and with τ. This isn’t enough to make them competitive with existential risk reduction ex ante as they are still down-weighted by the very small probability. But it is perhaps enough to cause some issues. Worse is that there is a possibility of growth in value that is faster than an exponential, and this can more than offset the very small probability. This feels very much like Pascal’s Mugging and I’m not inclined to bite the bullet and seek out or focus on outcomes like this. But nor do I have a principled answer to why not. I agree that it is probably useful to put under the label of ‘fanaticism’.

Toby_Ord Jul 31, 2023, 11:27 AM
12 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
Thanks Will, these are great comments — really taking the discussion forwards. I’ll try to reply to them all over the next day or so.

Toby_Ord Jul 31, 2023, 11:25 AM
9 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
As you say, there is an issue that some of these things might really be enhancements because they aren’t of a fixed size. This is especially true for those that have instrumental effects on the wellbeing of individuals, as if those effects increase with total population or with the wellbeing level of those individuals, then they can be enhancements. So cases where there is a clearly fixed effect per person and a clearly fixed number of people who benefit would be good candidates.
As are cases where the thing is of intrinsic non-welfarist value. Though there is also an issue that I don’t know how intrinsic value of art, environmental preservation, species types existing, or knowledge is supposed to interact with time. Is it twice as good to have a masterpiece or landscape or species or piece of knowledge for twice as long? It plausibly is. So at least on accounts of value where things scale like that, there is the possibility of acting like a gain.
Another issue is if the effects don’t truly scale with the duration of our future. For example, on the longest futures that seem possible (lasting far beyond the lifetime of the Sun), even a well preserved site may have faded long before our end point. So many candidates might act like gains on some durations of our future, but not others.

Toby_Ord Jul 31, 2023, 11:09 AM
7 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Shaping Humanity’s Longterm Trajectory
You may be right that this is more than a ‘tweak’. What I was trying to imply is that the framework is not wildly different. You still have graphs, integrals over time, decomposition into similar variables etc — but they can behave somewhat differently. In this case, the resources approach is tracking what matters (according to the cited papers) faithfully until expansion has ended, but then is indifferent to what happens after that, which is a bit of an oversimplification and could cause problems.
I like your example of speed-up in this context of large-scale interstellar settlement, as it also brings another issue into sharp relief. Whether thinking in terms of my standard framework or the ‘tweaked’ one, you are only going to be able to get a pure speed-up if you increase the travel speed too. So simply increasing the rate of technological (or social) progress won’t constitute a speed-up. This happens because in this future, progress ceases to be the main factor setting the rate at which value accrues.

Toby_Ord Jul 31, 2023, 10:47 AM
9 points
1 ∶ 0
in reply to: Benjamin_Todd’s comment on: Shaping Humanity’s Longterm Trajectory
I’ve thought about this a lot and strongly think it should be the way I did it in this chapter. Otherwise all the names are off by one derivative. e.g. it is true that for one of my speed-ups, one has to temporarily accelerate, but you also have to temporarily change every higher derivative too, and we don’t name it after those. The key thing that changes permanently and by a fixed amount is the speed.

Toby_Ord Jul 31, 2023, 8:29 AM
6 points
2 ∶ 0
in reply to: Fai’s comment on: Shaping Humanity’s Longterm Trajectory
It’s because I’m not intending the trajectories to be a measure of all value in the universe, only the value we affect through our choices. When humanity goes extinct, it no longer contributes intrinsic value through its own flourishing and it has no further choices which could have instrumental value, so you might expect its ongoing value to be zero. And it would be on many measures.
Setting up the measures so that it goes to zero at that point also greatly simplifies the analysis, and we need all the simplification we can get if we want to get a grasp on the value of the longterm future. (Note that this isn’t saying we should ignore effects of our actions on others, just that if there is a formal way of setting things up that recommends the same actions but is more analytically tractable, we should use that.)

Toby_Ord Jul 23, 2023, 12:47 PM
13 points
1 ∶ 0
in reply to: Fai’s comment on: Shaping Humanity’s Longterm Trajectory
Good point. I may not be clear enough on this in the piece (or even in my head). I definitely want to value animal wellbeing (positive and negative) in moral choices. The question is whether this approach can cleanly account for that, or if it would need to be additional. Usually, when I focus on the value of humanity (rather than all animals) it is because we are the relevant moral agent making the choices and because we have tremendous instrumental value — in part because we can affect other species for good or for ill. That works for defining existential risk as I do it via instrumental value.
But for these curves, I am trying to focus on intrinsic value. Things look quite different with instrumental value, as the timings of the benefits change. e.g. if we were to set up a wonderful stable utopia in 100 years, then the instrumental value of that is immense. It is as if all the intrinsic value of that utopia is scored at the moment in 100 years (or in the run up to it). Whereas, the curves are designed to track when the benefits are actually enjoyed.
I also don’t want them to track counterfactual value (how much better things are than they would have been) as I think that is cleaner to compare options by drawing a trajectory for each option and then compare those directly (rather than assuming a default and subtracting it off every alternate choice).
It isn’t trivial to reconcile these things. One approach would be to say the curve represents the instrumental effects of humanity on intrinsic value of all beings at that time. This might work, though does have some surprising effects, such as that even after our extinction, the trajectory might not stay at zero, and different trajectories could have different behaviour after our extinction.

Toby_Ord Jul 19, 2023, 10:02 AM
2 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Shaping Humanity’s Longterm Trajectory
Quite right!

Toby_Ord Jul 19, 2023, 10:02 AM
6 points
0 ∶ 0
in reply to: Toby_Ord’s comment on: Shaping Humanity’s Longterm Trajectory
Even in a model where it really matters when one arrives at each point in space (e.g. if we were merely collecting the flow of starlight, and where the stars burning out set the relevant end point for useful expansion) I believe the relevant number is still very small: 4/R where R is the relevant radius of expansion in light years. The 4 is because this grows as a quartic. For my model, it is 3/R, where R is 16.7 billion light years.

Toby_Ord Jul 19, 2023, 9:36 AM
6 points
0 ∶ 0
in reply to: trevor1’s comment on: Shaping Humanity’s Longterm Trajectory
The text of the book is almost completely finalised now, and my guess is that it will be out early 2023 (print publishing is usually slow). It is an academic book, published with Oxford University Press, and I think it will be open access too. My guess is that sales will be only the tiniest fraction of those of WWOTF, and that free downloads will be bigger, but still a small fraction.

Toby_Ord Jul 19, 2023, 9:33 AM
11 points
0 ∶ 0
in reply to: MichaelStJules’s comment on: Shaping Humanity’s Longterm Trajectory
This is a great point and would indeed have been good to include.
There is a plausible case that advancing AI roughly advances everything (as well as having other effects on existential risk …) making advancements easier if they are targetting AI capabilities in particular and making advancements targetting everything else harder. That said, I still think it is easier to reduce risk by 0.0001% than to advance AI by 1 year — especially doing the latter while not increasing risk by a more-than-compensating amount.

Toby_Ord Jul 19, 2023, 8:52 AM
10 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Shaping Humanity’s Longterm Trajectory
Thanks!
The idea is that advancing overall progress by a year means getting a year ahead on social progress, political progress, moral progress, scientific progress, and of course, technological progress. Given that our progress in these is the result of so many people’s work, it seems very hard to me for a small group to lead to changes that improve that by a whole year (even over one’s lifetime). Whereas a small group leading to changes that lead to the neglected area of existential risk reduction of 0.0001% seems a lot more plausible to me — in fact, I’d guess we’ve already achieved that.