tylermjohn

• tylermjohnMar 17, 2025, 4:19 PM

  3 points

  1 ∶ 0

  in reply to: Greg_Colbourn ⏸️ ’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Oh no! But then we are likely to lose out on almost all value because we won’t have the enormous digital workforce needed to settle the stars. It seems like we should bank on having some chance of solving alignment (at least for some architecture, even if not the current deep learning paradigm) and work towards that at least over the next couple hundred years.

• tylermjohnMar 17, 2025, 4:17 PM

  9 points

  1 ∶ 0

  in reply to: Jordan Arel’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Will is thinking about this much more actively and will give the best answer, but here are some key crucial considerations:

  • How tractable is extinction risk reduction and trajectory change work?

    • As a part of that, are there ways that we can have a predictable and persistent effect on the value of the long-term future other than by reducing extinction risk?

  • How good is the future by default?

  • How good are the best attainable futures?

  These are basically Tractability and Importance from the INT framework.

  Some of the biggest disagreements in the field are over how likely we are to achieve eutopia by default (or what % of eutopia we will achieve) and what, if anything, can be done to predictably shape the far future. Populating and refining a list of answers to this last question has been a lot of the key work of the field over the past few years.

• tylermjohnMar 17, 2025, 4:11 PM

  3 points

  1 ∶ 0

  in reply to: Davidmanheim’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  I’ll just share that for me personally the case rests on expected value. I actually think there is a lot that we can do to make AI existential safety go better (governance if nothing else), and this is what I spend most of my time on. But the expected value of better futures seems far higher given the difference in size between the default post-human future and the best possible future.

• tylermjohnMar 17, 2025, 4:06 PM

  1 point

  0 ∶ 0

  in reply to: Maxime Riché 🔸’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  No, I don’t think there’s a misunderstanding. It’s more that I think the future could go many different ways with wide variance in expected value, and I can shape the direction the human future goes but I cannot shape the direction that alien futures go.

  What do you think about just building and letting misaligned AGI loose? That seems fairly similar to letting other civilisations take over. (Apologies that I haven’t read your evaluation.)

• tylermjohnMar 17, 2025, 4:04 PM

  11 points

  1 ∶ 0

  in reply to: Greg_Colbourn ⏸️ ’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Part of what it means that I try to support thinking on this issue, e.g. by seed-funding NYU MEP and doing this discussion, and doing my own thinking on it.

  At this stage the thing I’m most excited about supporting is market-based mechanisms for democratic AI alignment like this. Also excited about trying to get more resources to work on AI welfare, utilitarianism, and to groups like Forethought: A new AI macrostrategy group.

  In practice I spend more resources on extinction risk reduction. Part of this is just because I’d really prefer not to die in my 30s. When an EA cares for their family taking away time from extinction risk they’re valuing their family as much as 10^N people. I see myself as doing something similar here.

• tylermjohnMar 17, 2025, 3:59 PM

  3 points

  0 ∶ 0

  in reply to: William_MacAskill’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Re WWOTF: You can (and should) think that there’s huge amounts of value at stake in the future, and even think that there’s much more value at stake in the future than there is in the present century, without thinking that value is linear in number of happy people. It diminishes the case a bit, but nowhere near enough for longtermism to not go through.

  Sure you could have a view that it’s great to have 10^12 people, but no more than that, but that seems like a really weird thing to have written in the stars. Or that all that matters is creating the Machine God, so we haven’t attained any value yet. But that doesn’t seem great.
  

  Do you have a gloss on a kind of view that threads the needle nicely without being too crazy, even if it doesn’t ultimately withstand scrutiny?

• tylermjohnMar 17, 2025, 3:53 PM

  3 points

  0 ∶ 0

  in reply to: JoA🔸’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Here are three toy existential catastrophe scenarios to think about:

  • A biological catastrophe (potentially from AI) which kills all humans and leaves animals mostly untouched, due to their biology

  • A paperclipping-style AI takeover scenario where AI turns everything into something else

  • A human disempowerment scenario where humans are left alive but substantively lose control over the future and its direction

  I think it would be pretty interesting to think about interventions one could take to make the world persistently better for wild animals in the event that humans go extinct from biological catastrophe. I’m not sure you could do much, but it could be very impactful if worst-case bio gets bad enough!

  My view is that bio x-risk is fairly low, so the scenarios where there are no humans but there are nonhuman animals (in the near future) are pretty unexpected.

• tylermjohnMar 17, 2025, 3:45 PM

  5 points

  0 ∶ 0

  in reply to: Greg_Colbourn ⏸️ ’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  This seems rather arbitrary. Why woiuld preventing extinction now guarantee that we (forever) lose that 1000x potential?
  

  It’s super arbitrary! Just trying to pull out your own model.

  I give one argument in Power Laws of Value.

  I think it is. Gaining the best possible future requires aligning an ASI, which has not been proven to be even theoretically possible afaik.

  One underrated argument for focusing on non-alignment issues like trajectory change is:

  • Either alignment is easy or hopeless

  • If it’s hopeless, we shouldn’t work on it

  • If it’s easy, we shouldn’t work on it

  It only makes sense to work on alignment if we happen to fall in the middle, where marginal efforts can make a difference before it’s too late. If you think this space is small, then it doesn’t look like a very tractable problem.

• tylermjohnMar 17, 2025, 3:42 PM

  5 points

  2 ∶ 1

  in reply to: Maxime Riché 🔸’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Civ-Saturation seems plausible, though only if there are other agents in the affectable universe. I don’t have a good view on this, and yours is probably better.

  Civ-Similarity seems implausible. I at least have some control over what humans do in the future, so I can steer things towards the futures I judge best. I don’t have any control over what aliens do. And there are large differences between the best and middling futures as I argue in Power Laws of Value .

• tylermjohnMar 17, 2025, 3:39 PM

  7 points

  0 ∶ 0

  in reply to: MichaelStJules’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  How asymmetric do you think things are? I tend to deprioritise s-risks (both accidental and intentional) because it seems like accidental suffering and intentional suffering will be a very small portion of the things that our descendants choose to do with energy. In everyday cases I don’t feel a pull to putting a lot of weight on suffering. But I feel more confused when we get to tail cases. Maximising pleasure intuitively feels meh to me, but maximising suffering sounds pretty awful. So I worry that (1) all of the value is in the tails, as per Power Laws of Value and (2) on my intuitive moral tastes the good tails are not that great and the bad tails are really bad.

• tylermjohnMar 17, 2025, 3:34 PM

  3 points

  1 ∶ 0

  in reply to: William_MacAskill’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Hopefully they do not just converge on the same moral views, but also good ones!

• tylermjohnMar 17, 2025, 3:33 PM

  3 points

  0 ∶ 1

  in reply to: William_MacAskill’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  The easiest way, in my view, to make a near-optimal future very likely, conditional on non-extinction, is if value is bounded above.

  Yes.

  Of all the issues facing classical utilitarianism, this seems the most problematic to me.

  So you doubt fanaticism, the view that a tiny chance of an astronomically good outcome can be more valuable than a certainty of a decent outcome. What about in the case of certainty? Do you doubt the utilitarian’s objection to Common-sense Eutopia? This kind of aggregation seems important for the case for longtermism. (See the pages of paper dolls in What We Owe the Future.)

• tylermjohnMar 17, 2025, 3:29 PM

  3 points

  0 ∶ 0

  in reply to: Greg_Colbourn ⏸️ ’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Suppose p(doom) is 90%. Then preventing extinction multiplies the value of the world by 10 in expectation.

  But suppose that the best attainable futures are 1000 times better than the default non-extinction scenario. Then ensuring we are on track to get the best possible future multiplies the value of the world by 100 in expectation, even after factoring in the 90% chance of extinction.

  In this toy model, you should only allocate your resources to reducing extinction if it is 10 times more tractable than ensuring we are on track to get the best possible future, at the current margin.

  You might think that we can just defer this to the future. But I’ve assumed in the set-up that the default future is 1/​1000th as good as the best future. So apparently our descendants are not going to be very good at optimising the future, and we can’t trust them with this decision.

  Where do you think this goes wrong?

• tylermjohnMar 17, 2025, 3:24 PM

  7 points

  0 ∶ 0

  in reply to: OscarD🔸’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  There are funky axiologies where value is superlinear in resources — basically any moral worldview that embraces holism. If you think that the whole, arranged a particular way, is more valuable than the parts, then you will be even more precious about how precisely the world should be arranged than the total utilitarian.

• tylermjohnMar 17, 2025, 3:15 PM

  7 points

  0 ∶ 0

  on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Position statement: I chose 36% disagreement. AMA!

  My view is that Earth-originating civilisation, if we become spacefaring, will attain around 0.0001% of all value. This still makes extinction risk astronomically valuable (it’s equivalent to optimising a millionth of the whole cosmos!), but if we could increase the chance of optimising 1% of the universe by 1%, this would be 100x more valuable than avoiding extinction. (You’re not going to get an extremely well grounded explanation of these numbers from me, but I hope they make my position clearer.)

  My view is that if Earth-originating civilisation becomes spacefaring, over the long term we will settle the entire universe and use nearly all of its energy. However, we could use this energy for many different things. I expect that by default we will mostly use this energy to create copies of ourselves (human brains) or digital slaves that allow us to learn and achieve new things, since humans love nothing more than humans and Earthly things. But these are inefficient media for realising moral value. We could instead use this energy to realise vastly more value, as I argue in Power Laws of Value. But our descendants basically won’t care at all about doing this. So I expect that we will miss out on nearly all value, despite using most of the universe’s energy.

  I think that there will be significant path dependency in what Earth-originating civilisation chooses to make with our cosmic endowment. In particular, I expect that artificial intelligence will drive most growth and most choices after the 21st century. Several factors, such as the number of distinct agents, their values, and the processes by which their values evolve over time will therefore make a decisive difference to what our descendants choose to do with the cosmos.

  So our choices about AI governance and architecture today are likely to make a significant and, at least on some paths we can choose, predictable difference to what our descendants do with all of the energy in the universe. If we do this well, it could make the difference between attaining almost no value and nearly all value.

  Given that I expect us to miss out on almost all value by default, I view the value of avoiding extinction as smaller than those who think we will achieve almost all value by default.

• tylermjohnMar 17, 2025, 2:37 PM

  3 points

  0 ∶ 0

  in reply to: zdgroff’s comment on: Power Laws of Value

  Thanks, @zdgroff! I think MPL is most important if you think that there are going to be some agents shaping things, these agents’ motivations are decisive for what outcomes are achieved, and you might (today) be able to align these agents with tail-valuable outcomes. Then aligning these agents with your moral values is wildly important. And by contrast marginal improvements to the agents’ motivations are relatively unimportant.

  You’re right that if you don’t have any chance of optimizing any part of the universe, then MPL doesn’t matter as much. Do you think that there won’t be agents (even groups of them) with decisive control over what outcomes are achieved in (even parts of) the world?

  It seems to me in the worst case we could at least ask Dustin to try to buy one star and then eventually turn it into computronium.

• tylermjohnMar 17, 2025, 1:51 PM

  3 points

  0 ∶ 0

  in reply to: GideonF’s comment on: Power Laws of Value

  Thanks a lot, @Gideon Futerman! Good additions, which all seem right to me.

  Another question I’ve had on my mind is how much MPL is related to additive separability. You might at a first pass think that moral atomism makes you more likely to buy MPL, since you have so many different spaces of value to optimise. But holistic views can in principle lead to even sharper differences in the value of worlds — for example, you could have a view that says that you need to align all of the stars forever or you don’t capture any value.

  I’d like to have a clearer view on when moral viewpoints will tend to end up at MPL, but I don’t have one yet.

• tylermjohnMar 17, 2025, 1:26 PM

  3 points

  0 ∶ 0

  in reply to: GideonF’s comment on: Power Laws of Value

  Nice point. I shouldn’t have picked averageism as the most extreme version of this view. It would have been more apt to pick a “capped” model where the value on additional utility (or utility of a specific type) becomes zero after enough of it has been achieved.

• tylermjohnMar 17, 2025, 12:23 PM

  3 points

  0 ∶ 0

  in reply to: Davidmanheim’s comment on: Power Laws of Value

  Thanks a lot @Davidmanheim!

  definitely resonates with the empirical EA view of power-law returns, which I was surprised you didn’t mention.

  Thanks, I meant to mention this but I think it got cut when I revised the second section for clarity, alas.

  1. The version of non-naturalist moral realism that the divergence seems both very strong, and strange to me. It assumes that the true moral code is unlike mathematical realism, where it’s accessible with reflection and would be a natural conclusion for those who cared.

  Thanks for this. I’d love to hear non-naturalist moral realists talk about how they think moral facts are epistemically accessible, if it’s not just luck. (Some philosophers do explicitly assume it’s luck.) I think the problem here is extremely hard, including for mathematics, and my own view on mathematics is closest to Millian empiricism (we learn basic math e.g. arithmetic by observing physics and for more advanced mathematics we freely choose axioms that we test against reality for usefulness and explanatory power).

  The best philosopher writing on the epistemology of both mathematics and ethics is Justin Clarke-Doane, who combines a form of pluralist realism about mathematics with expressivism about ethics.

  However we access moral facts, I’d expect my points about dependency on initial conditions to generalise.

  2. “You could accept diminishing returns to value in utility… but you’re unlikely to be a longtermist, laser focused on extinction risk if you do.” I think this is false under the view of near-term extinction risk that is held by most of those who seem concerned about AI extinction risk, or even varieties of the hinge-of-history view whereby we are affected in the near term by longtermist concerns.

  True, you could accept this moral view and also accept that:

  • X-risk is so high and so tractable that reducing it is better than GiveWell on relatively NT grounds

  • Animals don’t matter much, so don’t beat NT x-risk

  And then you’d avoid the objection.

  Or you could think utility doesn’t matter much after you’ve got, say, 10^12 humans, and then x-risk still looks good but making sure the right future happens looks less important.

  In general I think I was too quick here, good catch.

Power Laws of Value