Toby_Ord

• Toby_OrdJul 19, 2023, 8:47 AM

  6 points

  0 ∶ 0

  in reply to: MichaelStJules’s comment on: Shap­ing Hu­man­ity’s Longterm Trajectory

  I’m not sure about Tarsney’s model in particular, but on the model I use in The Edges of Our Universe, a year’s delay in setting out towards the most distant reaches of space results in reaching about 1 part in 5 billion fewer stars before they are pulled beyond our reach by cosmic expansion. If reaching them or not is the main issue, then that is comparable in value to a 1 in 5 billion existential risk reduction, but sounds a lot harder to achieve.

Shap­ing Hu­man­ity’s Longterm Trajectory

• Toby_OrdJul 5, 2023, 1:59 PM

  25 points

  7 ∶ 0

  in reply to: Toby_Ord’s comment on: Three mis­takes in the moral math­e­mat­ics of ex­is­ten­tial risk (David Thorstad)

  I thought section 6.1 on ‘Cumulative risk and intergenerational coordination’ was very good. Many people (including those promoting action on existential risk) neglect how important it is that we get risk down and then keep it down. This is a necessary part of what I call existential security in my section of The Precipice devoted to our longterm strategy. And it is not easy to achieve. One strategy I talk about is implementing a constitution for humanity, committing future generations to work within their own diminishing share of a finite existential risk budget.

  I broadly agree with section 6.2 on dialectical flips, which is why I made broadly the same point in The Precipice (p 275):

  “This is contrary to our intuitions, as people who estimate risk to be low typically use this as an argument against prioritizing work on existential risk.”

  I was thus a bit surprised to see my book cited in that section as something endorsing the Time of Perils, but not as something that had made the same point you are making about why high risk per-period risk reduces the EV of work on single-period risk reduction.

• Toby_OrdJul 5, 2023, 11:53 AM

  20 points

  4 ∶ 1

  in reply to: Toby_Ord’s comment on: Three mis­takes in the moral math­e­mat­ics of ex­is­ten­tial risk (David Thorstad)

  Re the third ‘mistake’, there is a long history of thinking that carrying capacity is a decent proxy for long term population. Is it a good proxy? Probably not in many situations. Is it better than extrapolating out the current growth dynamics for millions of years? Probably. My guess is that it is a simple defensible rough model here. And by laying out separate estimates for different scales being reached, there is also a pretty good sensitivity analysis. I think you are right that this could be improved by adding cases of permanent population collapse to the sensitivity analysis. But it won’t change the EV much. So again, I wonder if a superior critique would be: these estimates are more or less correct in EV terms, but we should be suspicious of EV.

• Toby_OrdJul 5, 2023, 11:47 AM

  37 points

  8 ∶ 1

  in reply to: Toby_Ord’s comment on: Three mis­takes in the moral math­e­mat­ics of ex­is­ten­tial risk (David Thorstad)

  Regarding the ‘second mistake’, I don’t see how it is very different from the first one. If there remains high average per-period risk, then the expected benefits of avoiding nearterm risk is indeed greatly lowered — from ‘overwhelming’ to just ‘large’. In effect, it starts to approach the level of risk to currently existing people (which is sometimes argued to be so large already that we don’t need to talk about future generations).

  But it doesn’t seem unreasonable to me for Millet and Snyder-Beattie to model things with an expected lifespan for humanity equal to that of a typical species. It is true that if risk stays high, then we won’t get that, but risk staying high would be a more contentious assumption. And uncertainty about the final rate, tends to increase the expectation. e.g. If there was even a 1 in 400 chance that we last as long as the Nautilus, then that alone would make M & SB’s assumption an underestimate. Again, I can’t see any ‘mistake’ here.

  I was actually much more intrigued by your comment about a systematic overestimate due to an implicit assumption of independence between the variables they estimate. I’d have loved to see that developed instead.

  There is also room for an interesting critique of EV of risk reduction as the best measure. Your arguments generally put pressure on the idea that the estimate of M & SB (or other people’s duration estimates) are typical of the probability distribution. That is, they might be OK as estimates of the expectations (means), but they get much of that EV from the extreme tail of the distribution. And we might have Pascallian concerns about cases like that, where there is a decent case that we shouldn’t compare prospects like this by their expectations.

• Toby_OrdJul 5, 2023, 11:30 AM

  49 points

  12 ∶ 1

  in reply to: Toby_Ord’s comment on: Three mis­takes in the moral math­e­mat­ics of ex­is­ten­tial risk (David Thorstad)

  Regarding the ‘first mistake’, you correctly show that survival of a species for a billion years requires reaching a low per-period level of risk (averaging roughly 1 in a billion per year). I don’t disagree with that and I doubt Bostrom would either. No complex species has yet survived so long, but that is partly because there have been less than 1 billion years since complex life began. But there are species (or at least families) that have survived almost the whole time, such as the Nautilus (which has survived 500 million years). So risk levels comparable to 1 in a billion per year do occur. For Bostrom’s modelling of the EV of risk reduction to work, he just needs there to be at least a small chance (say 1 in 1 million) that risk declines to such a level or beyond. That sounds eminently plausible to me, and my best guess of this probability would be much higher.

  You say that: “there is a clear sense in which the drop in existential risk that Bostrom envisions is not small, but instead very large”. But note that this is not the drop in existential risk that need be caused by the intervention Bostrom is evaluating. He relies on there being at least a slender possibility that risk levels fall to something like those of the safest species on Earth, but the intervention doesn’t need to bring that about.

  So on this ‘first mistake’, I agree that it is often also useful to think of things in per-period risk, and that this could provide a sanity check. But in this case, I think Bostrom’s estimate passes that sanity check, so don’t think he has made any kind of mistake here.

• Toby_OrdJul 5, 2023, 11:03 AM

  42 points

  13 ∶ 3

  on: Three mis­takes in the moral math­e­mat­ics of ex­is­ten­tial risk (David Thorstad)

  Hi David,

  Thanks for sharing this.

  My main reaction is that I was puzzled by the framing. It is obviously an allusion to Parfit’s ‘Five Mistakes in Moral Mathematics’. But there are major differences. Parfit was objecting to pieces of maths that are embedded in our common-sense understanding of morality, such as the share of the total view. He argued that the maths of morality is different to that. You are complaining about three modelling assumptions about the empirics of risk over time and population over time. You don’t present any disagreement with the moral mathematics (which is just a big expected value calculation over the total wellbeing of all future people). And you don’t even suggest that there were mistakes made in the modelling, just that they may rely on assumptions that weren’t foregrounded and thus be misleading.

  I liked the work you did on foregrounding those assumptions, but felt a bit let down by the framing. I expected a Parfit-mark-II piece that showed how our commonsense understanding of the ethics of longtermism relied on mistaken moral assumptions, but instead found a piece that mainly just suggested different modelling assumptions (and in my view, assumptions that are more misleading than those in the pieces you critique).

  The framing also sounds to my ear to be a bit insulting to your peers. To some extent, any paper in moral philosophy where the author disagrees with another person could be reframed to be about a mistake in the other person’s moral reasoning, but I’m glad authors don’t typically choose that frame. Instead, they say that opponent said P, while this piece argues not P, or that the opponent assumed Q, while here are some reasons that Q is not a safe assumption. This keeps the focus on the content.

  This is a lot of text re a piece’s framing, but I wanted to lay it out because (at least for me) the framing really does distract from the content.

  (I’ll split further reactions on particular parts into separate comments)

• Toby_OrdJun 24, 2023, 10:32 AM

  52 points

  8 ∶ 0

  in reply to: Sarah Levin’s comment on: My ten­ta­tive best guess on how EAs and Ra­tion­al­ists some­times turn crazy

  extremely ideological group houses like early Toby Ord’s

  You haven’t got your facts straight. I have never lived in a group house, let alone an extremely ideological one.

• Toby_OrdJun 13, 2023, 11:59 AM

  99 points

  59 ∶ 0

  on: Why I Spoke to TIME Magaz­ine, and My Ex­pe­rience as a Fe­male AI Re­searcher in Sili­con Valley with Sex­ual Harass­ment/​Abuse

  Thank you so much for writing so clearly and compellingly about what happened to you and the subculture which encourages treating women like this.

  There is no place for such a subculture in EA (or anywhere else).

• Toby_OrdApr 3, 2023, 8:44 AM

  10 points

  2 ∶ 0

  in reply to: Vasco Grilo🔸’s comment on: Cast­ing the De­ci­sive Vote

  Thanks Vasco,

  Interesting analysis. Here are a few points in response:

  • It is best to take my piece as an input into a calculation of whether voting is morally justified on account of changing the outcome — it is an input in the form of helping work out the probability the outcome gets changed. More analysis would be needed to make the overall moral case — especially in the many voting systems that have multiple levels, where it may be much more important to vote in marginal seats and much less in safe seats, so taking the average may be inappropriate.

  • You make a good point that the value depends on who it is and their counterfactuals. Most people looking at this are trying to work out the average value to defend against claims that voting is not typically morally justified, rather than trying to work out the case for particular groups such as EAs — though that is a relevant group for this forum.

  • In such empirical arguments, I’d be cautious about claims that $1 dollar to the LTFF (or similar) is literally worth the same as $30,000 distributed across US citizens. Once the ratios get this extreme, you do need to worry more about issues like 0.1% of the $30,000 flowing through to extremely high value things and then outweighing the small targeted donation.

  • While you were trying to be very conservative by allocating a very large financial benefit to the better of the two parties, it is also relevant that who is in power at the time of the development of transformative AI capabilities could be directly relevant to existential risk, so even your generous accounting may be too small. (This factor will only apply in a small number of elections, but US presidential elections are likely one of them.)

  • I have a general presumption in favour of EAs acting as most people think morally responsible people should. In part because there is a good chance that the common-sense approach is tracking something important that our calculations may have lost sight of, in part because I don’t think we should be trying to optimise all aspects of our behaviour, and in part because it is a legible sign of moral earnestness (i.e. it is reasonable for people to trust you less if you don’t do the things those people see as basic moral responsibilities).

• Toby_OrdMar 31, 2023, 1:00 PM

  2 points

  0 ∶ 0

  in reply to: John G. Halstead’s comment on: Cast­ing the De­ci­sive Vote

  Something like that. Geoffrey Brennan and Lomasky indeed present the binomial formula and suggest using it in their earlier work, but I haven’t found a case of them using it in any particular way (which could get results like Jason Brennan’s or results like Banzhaf’s), so didn’t want to pin this on them. So I cited Jason Brennan whose uses it to produce these crazily low probabilities in his book. It is possible that Jason Brennan didn’t do the calculations himself and that someone else did (either Geoffrey Brennan and Lomasky or others), but I don’t know and haven’t found an earlier source for the crazy numbers.

• Toby_OrdMar 29, 2023, 1:23 PM

  12 points

  1 ∶ 0

  on: Cast­ing the De­ci­sive Vote

  A caution re interpreting of my argument in two-level elections:

  One might read the above piece as an argument that voting is generally worthwhile. But note that the two-level structure of many elections (at least in countries without PR) does dampen the value of voting for many voters. e.g. if you are in the 10%+ of the US population who live in California, then not only are you very unlikely to cast a decisive vote to win the state’s electoral college votes (since the probability that the underdog wins is very low), but it is very likely that in the situation where California comes down to a single vote, the rest of the country has skewed overwhelmingly to the Republicans, making the Californian electoral college votes irrelevant. Similar situations hold for safe seats in the lower house in the US, UK or Australia.

  It might still be that in some sense two-level elections function on average like a single level election, but even if so, that could be because there are some people in marginal seats/​states with disproportionate chances of changing the outcome, while many or most people have very little.

  So while my adjusted formula above does apply in two-level elections, the intuitive interpretation that it supports a moral case for voting for the superior candidate may not apply.

• Toby_OrdMar 29, 2023, 1:03 PM

  3 points

  1 ∶ 0

  in reply to: Toby_Ord’s comment on: How much should gov­ern­ments pay to pre­vent catas­tro­phes? Longter­mism’s limited role

  I’ve just seen your comment further down:

  What we’re arguing for is a criterion: governments should fund all those catastrophe-preventing interventions that clear the bar set by cost-benefit analysis and altruistic willingness to pay. One justification for funding these interventions is the justification provided by CBA itself, but it need not be the only one. If longtermist justifications help us get to the place where all the catastrophe-preventing interventions that clear the CBA-plus-AWTP bar are funded, then there’s a case for employing those justifications too.

  which answers my final paragraph in the parent comment, and suggests that we are not too far apart.

• Toby_OrdMar 29, 2023, 1:00 PM

  3 points

  0 ∶ 0

  in reply to: EJT’s comment on: How much should gov­ern­ments pay to pre­vent catas­tro­phes? Longter­mism’s limited role

  Thanks for the clarifications!

• Toby_OrdMar 29, 2023, 12:58 PM

  4 points

  1 ∶ 0

  in reply to: EJT’s comment on: How much should gov­ern­ments pay to pre­vent catas­tro­phes? Longter­mism’s limited role

  we chose ‘unacceptable’ because we also think there would be something normatively problematic about it.

  I’m not so sure about that. I agree with you that it would be normatively problematic in the paradigm case of a policy that imposed extreme costs on current society for very slight reduction in total existential risk — let’s say, reducing incomes by 50% in order to lower risk by 1 part in 1 million.

  But I don’t know that it is true in general.

  First, consider a policy that was inefficient but small — e.g. one that cost $10 million to the US govt, but reduced the number of statistical lives lost in the US by only 0.1, I don’t think I’d say that this was democratically unacceptable. Policies like this are enacted all the time in safety contexts and are often inefficient and ill-thought-out, and I’m not generally in favour of them, but I don’t find them to be undemocratic. I suppose one could argue that all US policy that doesn’t pass a CBA is undemocratic (or democratically unacceptable), but that seems a stretch to me. So I wonder whether it is correct to count our intuitions on the extreme example as counting against all policies that are inefficient in traditional CBA terms or just against those that impose severe costs.

• Toby_OrdMar 29, 2023, 12:33 PM

  2 points

  0 ∶ 0

  in reply to: EJT’s comment on: How much should gov­ern­ments pay to pre­vent catas­tro­phes? Longter­mism’s limited role

  Thanks Elliott,

  I guess this shows that the case won’t get through with the conservative rounding off that you applied here, so future developments of this CBA would want to go straight for the more precise approximations in order to secure a higher evaluation.

  Re the possibility of international agreements, I agree that they can make it easier to meet various CBA thresholds, but I also note that they are notoriously hard to achieve, even when in the interests of both parties. That doesn’t mean that we shouldn’t try, but if the CBA case relies on them then the claim that one doesn’t need to go beyond it (or beyond CBA-plus-AWTP) becomes weaker.

  That said, I think some of our residual disagreement may be to do with me still not quite understanding what your paper is claiming. One of my concerns is that I’m worried that CBA-plus-AWTP is a weak style of argument — especially with elected politicians. That is, arguing for new policies (or treaties) on grounds of CBA-plus-AWTP has some sway for fairly routine choices made by civil servants who need to apply government cost-effectiveness tests, but little sway with voters or politicians. Indeed, many people who would be benefited by such cost-effectiveness tests are either bored by — or actively repelled by — such a methodology. But if you are arguing that we should only campaign for policies that would pass such a test, then I’m more sympathetic. In that case, we could still make the case for them in terms that will resonate more broadly.

• Toby_OrdMar 28, 2023, 7:51 AM

  11 points

  3 ∶ 0

  in reply to: Robi Rahman’s comment on: Cast­ing the De­ci­sive Vote

  Ah, that’s what I meant by the value your candidate would bring. There isn’t any kind of neutral outcome to compare them against, so I thought it clear that it meant in comparison to the other candidate. Evidently not so clear!

Cast­ing the De­ci­sive Vote

• Toby_OrdMar 24, 2023, 4:05 PM

  58 points

  19 ∶ 0

  on: Reflect­ing on the Last Year — Les­sons for EA (open­ing keynote at EAG)

  I should note that I don’t see a stronger focus on character as the only thing we should be doing to improve effective altruism! Indeed, I don’t even think it is the most important improvement. There have been many other suggestions for improving institutions, governance, funding, and culture in EA that I’m really excited about. I focused on character (and decision procedures) in my talk because it was a topic I hadn’t seen much in online discussions about what to improve, because I have some distinctive expertise to impart, and because it is something that everyone in EA can work on.

Reflect­ing on the Last Year — Les­sons for EA (open­ing keynote at EAG)