Toby_Ord

Karma: 3,312

Toby_Ord Mar 22, 2023, 5:29 PM
11 points
2 ∶ 1
in reply to: EliezerYudkowsky’s comment on: My Objections to “We’re All Gonna Die with Eliezer Yudkowsky”
I think you have put your finger on a key aspect with the coldness requirement.
When ice cream is melted or coke is lukewarm, they both taste far too sweet. I’ve long had a hypothesis that we evolved some kind of rejection of foods that taste too sweet (at least in large quantity) and that by cooling them down, they taste less sweet (overcoming that rejection mechanism) but we still get increased reward when the sugar content enters our bloodstream. I feel that carbonation is similar (flat coke tastes too sweet), so that the cold and carbonation could be hacks we’ve discovered to get around the ‘tastes too sweet’ defence mechanism, while still enjoying extremely high blood sugar based rewards. (Other forms of bitterness or saltiness added to the sweet foods could be similar.)
More speculative and still requires a few sentences to explain though, so a different example may be best.

Toby_Ord Mar 22, 2023, 5:15 PM
19 points
4 ∶ 0
in reply to: Toby_Ord’s comment on: How much should governments pay to prevent catastrophes? Longtermism’s limited role
Overshooting:
Second, the argument overshoots. Given other plausible claims, building policy on this premise would not only lead governments to increase their efforts to prevent catastrophes. It would also lead them to impose extreme costs on the present generation for the sake of miniscule reductions in the risk of existential catastrophe.
I disagree with this.
First, I think that many moral views are compelled to find the possibility that their generation permanently eradicates all humans from the world to be especially bad and worthy of much extra effort to avoid. As I detailed in Chapter 2 of the Precipice, this can be based on considerations about the past or about the future. While longtermism is often associated with the future-directed reasons, I favour a broader definition. If someone is deeply moved by the Burkean partnership of the generations over an unbroken chain stretching back 10,000 generations and thinks this gives additional reason not to be the generation who breaks it, then I’m inclined to say they are a longtermist too. But whether it counts or doesn’t, my arguments in Chapter 2 still imply that many moral views are already committed to a special badness of extinction (and often other existential risks). This means there is a wide set of views that go beyond traditional CBA and I can’t see a good argument why they should all overshoot.
And what about for a longtermist view that is more typical of our community? Suppose we are committed to the idea that each person matters equally no matter when they would live. It doesn’t follow from this that the best policy is one that demands vast sacrifices from the current generation, anymore than this follows from the widely held view that all people matter equally regardless of race or place of birth. One could still have places in ethics or political philosophy where there are limits placed on the sacrifices that can be demanded of you, and especially limits placed on the sacrifices you can force others to endure in order to produce a larger amount of benefits for others. Theories with such limits could still be impartial in time and could definitely qualify as longtermist.
One could also have things tempered by moral uncertainty or political beliefs about pluralism or non-coercion.
And that is before we get to the fact that longtermist policy drafters don’t have to ignore the feasibility of their proposals — another clear way to stop before you overshoot.
I really don’t think it is clear that there are any serious policy suggestions from longtermists that do overshoot here. e.g. in The Precipice (p. 186) my advice on budget is:
We currently spend less than a thousandth of a percent of gross world product on them. Earlier, I suggested bringing this up by at least a factor of 100, to reach a point where the world is spending more on securing its potential than on ice cream, and perhaps a good longer-term target may be a full 1 percent.
And this doesn’t seem too different from your own advice ($400B spending by the US is 2% of a year’s GDP).
A different take might be that I and others could be commended for not going too far, but that in doing so we are being inconsistent with our stated principles. That is an interesting angle, and one raised by Jim Holt in his very good NYT book review. But I ultimately don’t think it works either: I can’t see any strong arguments that longtermism lacks the theoretical resources to consistently avoid overshooting.

Toby_Ord Mar 22, 2023, 12:27 PM
48 points
9 ∶ 0
in reply to: Toby_Ord’s comment on: How much should governments pay to prevent catastrophes? Longtermism’s limited role
Regarding your CBA:
I worry that your stated estimates, aren’t really enough to secure adequate funding on traditional CBA grounds.
For example, it doesn’t work in many countries. There are only a fifth as many people living in the UK, so only a fifth as many beneficiaries. Thus the benefits internalised by the UK are a fifth as high, so the cost-benefit ratio is only a fifth as good. Moreover, the UK is poorer, so people are less willing to pay to avoid risk and the VSLs used by the government are substantially lower (I think about a fifth your stated figure — £2M), such that the package you list would dramatically fail the CBA by a factor of 25 or so in the UK. Many other countries are poorer or smaller than the US in some combination and will have trouble meeting this cost-benefit bar on the estimates you gave.
Even in the US, your estimates for the cost-benefit calculation aren’t really enough to make it go through. It would be if all these interventions were inextricably bound up as a package, or if all funding on them had equal effect. But CBA cares about marginal cost effectiveness and presumably the package can be broken into chunks of differing ex-ante cost-effectiveness (e.g. by intervention type, or by tranches of funding in each intervention). Indeed you suggest this later in the piece. Since the average only just meets the bar, if there is much variation, the marginal work won’t meet the bar, so government funding would cap out at something less than this, perhaps substantially so.
Moreover, your analysis suggests the package only marginally meets the cost-benefit threshold and requires some educated guesswork to do so (the amounts it would lower existential risk). Such speculative estimates that drive a large funding choice are usually frowned on in CBA. So even if it is marginally worth funding if government attention and capacity were free, it wouldn’t be a high priority.
My guess is therefore that a national package of interventions by the US would need to be notably more modest than this one to get funded purely on traditional CBA grounds, and much more modest to get funded in the UK or elsewhere, widening the gap between what traditional CBA gets us and what the addition of longtermist considerations could achieve.

Toby_Ord Mar 22, 2023, 12:09 PM
3 points
0 ∶ 0
in reply to: Toby_Ord’s comment on: How much should governments pay to prevent catastrophes? Longtermism’s limited role
The target of your comparisons:
At times it seems like you are comparing traditional CBA justifications for government work on existential risk vs longtermist justifications. At other times it seems like you are comparing the policy prescriptions and funding that would pass a traditional CBA vs the policy prescriptions and funding that would come out of strong longtermist reasoning (deliberately setting aside political feasibility). I have different views on each of these comparisons.
On the first, I think we should use both traditional CBA justifications as well as longtermist considerations and don’t think the piece really offers much argument against including the latter at all (for one thing, it doesn’t seem to recognise that they can actually be popular and motivating with the public and with policy makers in a way that CBA isn’t, widening the feasibility window).
On the second, I agree that we should generally not use strong longtermist justifications (though the piece usually frames itself as arguing against regular longtermist justifications). For one thing, I don’t actually endorse strong longtermism anyway. I also agree that we shouldn’t set aside political feasibility in our policy recommendations. But it seems something of a straw man to suggest that the choice under discussion is to ignore effects on future generations or to consider all such effects on a total utilitarian basis and ignore the political feasibility. Are there any serious advocates of that position? The inadequacy of the second of these options doesn’t really help you conclude that we should deliberately restrict ourselves to the CBA option, as there are so many other options that were not considered.

Toby_Ord Mar 22, 2023, 11:53 AM
29 points
6 ∶ 0
in reply to: Toby_Ord’s comment on: How much should governments pay to prevent catastrophes? Longtermism’s limited role
‘Democratically unacceptable’
Like Richard, I was quite confused while reading the piece about the use of this term. I agree with him that if you are trying to get normative mileage out of this idea, then it doesn’t work. In representative democracies like the US, we elect people to represent us in the highest levels of national decision-making. They are allowed to act in our narrow self-interest, but also in furtherance of our ideals. Indeed, they are aren’t constrained to leading from the back — to leading people to where they already want to go. They are also allowed to to lead from the front — to use good judgment to see things the public hadn’t yet seen, to show why it is an attractive course of action and to take it. Indeed, when we think of great Presidents or Prime Ministers it is often this quality that we most admire. So it seems entirely fine to me to appeal to people in government on moral grounds — including on longtermist grounds such as the effects on future generations. To say otherwise would involve some surprising judgments about other acts of moral leadership that are widely celebrated (such as in the ending of slavery).
I think the best version of this argument about democratic acceptability is not a moral or political philosophy argument (that it is wrong or unjust or undemocratic for governments to implement policies on these grounds) because there is little evidence it would be any of those things. Before a funding allocation reached those levels it would likely become politically unacceptable — i.e. it couldn’t practically be implemented and you would get less in the long run than if you asked for a more modest amount. I think there is broad agreement among longtermists that political feasibility is a key constraint, so the question is then about where it kicks in and what we can do to loosen the constraint by raising moral awareness.
At times it sounds like you agree with this, such as when you explicitly say “democratically unacceptable, by which we mean it could not be adopted and maintained by a democratic government”, but at other times you seem to trade on the idea that there is something democratically tainted about political advocacy on behalf of the people of the future — this is something I strongly reject. I think the paper would be better if it were clearer on this. I think using the term ‘democratically infeasible’ would have been more apposite and would dispel the confusion.
(I should add that part of this is the question of whether you mean infeasible vs normatively problematic, and part of it is the question of whether you are critiquing use of longtermist considerations (over and above CBA considerations) vs whether you are critiquing attempts to impose the policies one would come up with on strong longtermism + ignoring feasibility. I’ll address that more in another comment, but the short version is that I agree that imposing wildly unpopular policies on voters might be normatively problematic as well as infeasible, but think that comparison is a straw man.)

Toby_Ord Mar 22, 2023, 11:34 AM
47 points
7 ∶ 0
on: How much should governments pay to prevent catastrophes? Longtermism’s limited role
Thanks so much for writing this thought-provoking piece and for cross-posting it here to the forum for discussion. It is a long piece with lots of subtly different arguments that piece together in complex ways. Indeed, by the end of it, I’m not sure if I agree with the overall thrust or not. But there are many strands on which I do strongly agree or disagree and so I’ll try to break them out into different comments here to make the discussion easier.
Overall:
- I’m enthusiastic about your work towards creating CBA for existential risks.
- I’m sympathetic to the argument that concern about presently existing people is sufficient to warrant substantial government spending on existential risk (though I don’t think it quite gets there).
- I’m also sympathetic to the idea that one should lead with this in policy circles (though I favour the view that one should sometimes lead with it and sometimes with future generations).
- I strongly agree that we shouldn’t try to implement policies that are democratically infeasible (though it should be uncontroversial that it is sometimes correct to propose policies that go somewhat beyond current democratic feasibility, since feasibility is not fixed).
- I strongly disagree with the idea that it is in some way undemocratic to argue that the effects on future generations give an especially strong reason to be concerned about existential risks (which I think your paper sometimes suggests).

Toby_Ord Mar 21, 2023, 6:09 PM
31 points
10 ∶ 3
on: My Objections to “We’re All Gonna Die with Eliezer Yudkowsky”
Thanks so much for writing this Quinton. I found it extremely informative — one of the best things I’ve read in recent years on the big picture of alignment research.

Toby_Ord Mar 21, 2023, 12:30 PM
46 points
15 ∶ 1
on: My Objections to “We’re All Gonna Die with Eliezer Yudkowsky”
There was nothing with that much sugar, salt, fat combined together as ice cream.
This example has always frustrated me on the grounds that it is clearly false.
Eliezer correctly points out that honey existed in the ancestral environment, and so did gazelles. But if you simply pour honey over animal fat, you get something much higher in both sugar and fat than ice cream:
Ice cream is typically ~20% sugar, ~10% fat. An experiment found even that ice cream optimised for being liked by the eater involves ~15% sugar, 15% fat.
But honey is 82% sugar and animal fat is 100% fat. A 50-50 mixture is 41% sugar, 50% fat — more than twice as much of each as ice-cream.
Neither is super-salty, but animal fat (0.15%) still contains more salt than ice cream (0.08%), so a 30-70 mixture would have more sugar, fat, and salt. Also, very salty things such as salt deposits (100%) and sea water (3.5%) were sometimes available in the ancestral environment.
Of course there is something about ice cream that makes us want it more than honey-drenched fat! But I don’t know what that is, nor how it relates to AI alignment.

Toby_Ord Mar 9, 2023, 11:38 AM
1 point
0 ∶ 1
in reply to: Tobias Häberli’s comment on: Toby Ord’s new report on lessons from the development of the atomic bomb
You may be right, and these sources to make it less clear. I haven’t looked at the original sources and as with most parts of my report am following the eminent nuclear historian Richard Rhodes, who marshals some pretty convincing evidence that it was accidental. On page 521 of The Making of the Atomic Bomb (which I cited in that paragraph of my report):
…Franklin Roosevelt met with Winston Churchill at Casablanca. In the course of the meeting the two leaders discussed what terms of surrender they would eventually insist upon; the word “unconditional” was discussed but not included in the official joint statement to be read at the final press conference. Then, on January 24, to Churchill’s surprise, Roosevelt inserted the word ad lib: “Peace can come to the world,” the President read out to the assembled journalists and newsreel cameras, “only by the total elimination of German and Japanese war power … The elimination of German, Japanese and Italian war power means the unconditional surrender of Germany, Italy , and Japan.” Roosevelt later told Harry Hopkins that the surprising and fateful insertion was a consequence of the confusion attending his effort to convince French General Henri Girard to sit down with Free French leader Charles de Gaulle:
“We had so much trouble getting those two French generals together that I thought to myself that this was as difficult as arranging the meeting of Grant and Lee — and then suddenly the Press Conference was on, and Winston and I had had no time to prepare for it, and the thought popped into my mind that they had called Grant ‘Old Unconditional Surrender,’ and the next thing I knew I had said it.”
Churchill immediately concurred — “Any divergence between us, even by omission, would on such an occasion and at such a time have been damaging or even dangerous to our war effort” — and unconditional surrender became official Allied policy.
I don’t have time to follow up on all the sources Rhodes used to construct this passage, but it does sound like there is some remaining mystery here. We have direct quotes from Roosevelt and Churchill saying it was accidental, but some other evidence which might contradict that.

Toby_Ord Nov 22, 2022, 9:48 AM
18 points
3 ∶ 0
on: Beyond Simple Existential Risk: Survival in a Complex Interconnected World
While I do suggest a 0.1% probability of existential catastrophe from climate change, note that this is on my more restricted definition, where that is roughly the chance that humanity loses almost all its longterm potential due to a climate catastrophe. On Beard et al’s looser definition, I might put that quite a bit higher (e.g. I think there is something more like a 1% chance of a deep civilisation collapse from climate change, but that in most such outcomes we would eventually recover). And I’d put the risk factor from climate change quite a bit higher than 0.1% too — I think it is more of a risk factor than a direct risk.

Toby_Ord Nov 22, 2022, 9:41 AM
46 points
13 ∶ 2
on: Beyond Simple Existential Risk: Survival in a Complex Interconnected World
Thanks so much for posting this Gideon. I like your way of framing this into these two loose clusters, and especially your claim that it is good to have both. I completely agree. While my work is indeed more within the simple cluster, I feel that a fight over which approach is right would be misguided.
All phenomena can be modelled at lesser or greater degrees of precision, with different advantages and disadvantages of each. Often there are some sweet spots where there is an especially good tradeoff between accuracy and ability to actually use the model. We should try to find those and use them all to illuminate the issue.
There is a lot to be said for simple and for complex approaches. In general, my way forward with all kinds of topics is to start as simple as possible and only add complexity when it is clearly needed to address a glaring fault. We all know the truth is as complex as the universe, so the question is not whether the more complex model is more accurate, but whether it adds sufficient accuracy to justify the problems it introduces, such as reduced usability, reduced clarity, and overfitting. Sometimes it clearly is. Other times I don’t see that it is and am happy to wait for those who favour the complex model to point to important results it produces.
One virtue of a simple model that I think is often overlooked is its ability to produce crisp insights that, once found, can be clearly explained and communicated to others. This makes knowledge sharing easier and makes it easier to build up a field’s understanding from these crisp insights. I think the kind of understanding you gain from more complex models is often more a form of improving your intuitions and is harder to communicate, and doesn’t typically come with a simple explanation that the other person can check to see if you are right without spending a similar amount of time with the model.

Toby_Ord Oct 3, 2022, 8:52 AM
4 points
1 ∶ 1
in reply to: MichaelStJules’s comment on: Puzzles for Everyone
Though note that infinite sequences of choices are a well known paradox-ridden corner of decision theory, so proving that a theory falls down there is not conclusive.
I feel that exotic cases like this are interesting and help build up a picture of difficult cases for theories to cover, but don’t count strongly against particular theories which are shown to fail them. This is because it isn’t clear whether (1) any rival theories can deal with the exotic case, or (2) whether usual conditions (or theories) need to be slightly modified in the exotic setting. In other words, it may be another area where the central idea of Richard’s post (‘Puzzles for Everyone’) applies.

Toby_Ord Dec 16, 2021, 2:31 PM
65 points
0 ∶ 0
in reply to: MichaelStJules’s comment on: Response to Recent Criticisms of Longtermism
To be clear, he is factually incorrect about that claim. I never seriously considered calling it that.
One of the major points of effective altruism in my mind was that it isn’t only utilitarians who should care about doing more good rather than less, and not only consequentialists either. All theories that agree saving 10 lives is substantially more important than saving 1 life should care about effectiveness in our moral actions and could benefit from quantifying such things. I thought it was a great shame that effectiveness was usually only discussed re utilitarianism and I wanted to change that.

Toby_Ord Nov 18, 2021, 1:10 PM
14 points
0 ∶ 0
on: On the assessment of volcanic eruptions as global catastrophic or existential risks
Thanks so much for writing this. There are a lot of new leads on understanding the risks here, and it is great to see people working on what I think is the largest of all the natural risks.
I’d be very interested to see what estimates of the overall existential risk from supervolcanic eruptions you come up with. As direct extinction appears particularly difficult, much of the question comes down to estimates of the chance that an eruption of a given magnitude would cause the collapse of civilisation and the chance of recovering from that collapse. Factoring the risk in this way may be helpful for both showing where disagreements between researchers lie and for understanding where the current uncertainty is.
Note also that there was a numerical mistake in my supervolcanic eruption section which I explain on the book’s Errata page: https://theprecipice.com/errata
What links here?
- Opportunities that surprised us during our Clearer Thinking Regrants program by spencerg (Nov 7, 2022, 1:09 PM; 116 points)
- Opportunities that surprised us during our Clearer Thinking Regrants program by spencerg (LessWrong; Nov 7, 2022, 1:09 PM; 20 points)

Toby_Ord Oct 14, 2021, 12:18 PM
39 points
0 ∶ 1
on: A practical guide to long-term planning – and suggestions for longtermism
Thanks for writing this Sam. I think a large part of the disconnect you are perceiving can be explained as follows:
The longtermist community are primarily using an unusual (or unusually explicit) concern about what happens over the long term as a way of doing cause prioritisation. e.g. it picks out issues like existential risk as being especially important, and then with some more empirical information, one can pick out particular risks or risk factors to work on. The idea is that these are far more important areas to work on than the typical area, so you already have a big win from this style of thinking. In contrast, you seem to be looking at something that could perhaps be called ‘long-term thinking’, which takes any area of policy and tries to work out ways to better achieve its longterm goals using longterm plans.
These are quite different approaches to having impact. A useful analogy would be the difference between using cost-effectiveness as a tool for selecting a top cause or intervention to work on, vs using it to work out the most cost-effective way to do what you are already doing. I think a lot of the advantages of both cost-effectiveness and longtermist thinking are had in this first step of its contribution to cause-prioritisation, rather than to improving the area you are already working on.
That said, there are certainly cases of overlap. For example, while one could use longtermist cause-prioritisation to select nuclear disarmament as an area and then focus on the short term goal of re-establishing the INF treaty, which lapsed under Trump, one could also aim higher, for the best ways to completely eliminate nuclear weapons over the rest of the century, which would require longterm planning. I expect that longtermists could benefit from advances in longterm planning more than the average person, but it is not always required in order to get large gains from a longtermist approach.

Toby_Ord Sep 16, 2021, 1:23 PM
1 point
0 ∶ 0
in reply to: axioman’s comment on: When pooling forecasts, use the geometric mean of odds
Thanks — I hadn’t heard of f-means before and it is a useful concept, and relevant here.

Toby_Ord Sep 16, 2021, 1:20 PM
6 points
0 ∶ 0
in reply to: Jaime Sevilla’s comment on: When pooling forecasts, use the geometric mean of odds
I think we are roughly in agreement on this, it is just hard to talk about. I think that compression of the set of expert estimates down to a single measure of central tendency (e.g. the arithmetic mean) loses information about the distribution that is needed to give the right answer in each of a variety of situations. So in this sense, we shouldn’t aggregate first.
The ideal system would neither aggregate first into a single number, nor use each estimate independently and then aggregate from there (I suggested doing so as a contrast to aggregation first, but agree that it is not ideal). Instead, the ideal system would use the whole distribution of estimates (perhaps transformed based on some underlying model about where expert judgments come from, such as assuming that numbers between the point estimates are also plausible) and then doing some kind of EV calculation based on that. But this is so general an approach as to not offer much guidance, without further development.

Toby_Ord Sep 13, 2021, 9:11 AM
5 points
0 ∶ 0
in reply to: Jaime Sevilla’s comment on: When pooling forecasts, use the geometric mean of odds
I agree with a lot of this. In particular, that the best approach for practical rationality involves calculating things out according to each of the probabilities and then aggregating from there (or something like that), rather than aggregating first. That was part of what I was trying to show with the institution example. And it was part of what I was getting at by suggesting that the problem is ill-posed — there are a number of different assumptions we are all making about what these probabilities are going to be used for and whether we can assume the experts are themselves careful reasoners etc. and this discussion has found various places where the best form of aggregation depends crucially on these kinds of matters. I’ve certainly learned quite a bit from the discussion.
I think if you wanted to take things further, then teasing out how different combinations of assumptions lead to different aggregation methods would be a good next step.

Toby_Ord Sep 13, 2021, 9:03 AM
6 points
0 ∶ 0
in reply to: NunoSempere’s comment on: When pooling forecasts, use the geometric mean of odds
I see what you mean, though you will find that scientific experts often end up endorsing probabilities like these. They model the situation, run the calculation and end up with 10^-12 and then say the probability is 10^-12. You are right that if you knew the experts were Bayesian and calibrated and aware of all the ways the model or calculation could be flawed, and had a good dose of humility, then you could read more into such small claimed probabilities — i.e. that they must have a mass of evidence they have not yet shared. But we are very rarely in a situation like that. Averaging a selection of Metaculus forecasters may be close, but is quite a special case when you think more broadly about the question of how to aggregate expert predictions.

Toby_Ord Sep 10, 2021, 12:41 PM
18 points
0 ∶ 0
in reply to: Toby_Ord’s comment on: When pooling forecasts, use the geometric mean of odds
Thinking about this more, I’ve come up with an example which shows a way in which the general question is ill-posed — i.e. that no solution that takes a list of estimates and produces an aggregate can be generally correct, but instead requires additional assumptions.
Three cards (a Jack, Queen, and King) are shuffled and dealt to A, B, and C. Each person can see their card, and the one with the highest card will win. You want to know the chance C will win. Your experts are A and B. They both write down their answers on slips of paper and privately give them to you. A says 50%, so you know A doesn’t have the King. B also says 50%, which also lets you know B doesn’t have the King. You thus know the correct answer is a 100% chance that C has King. In this situation, expert estimates of (50%, 50%) lead to an aggregate estimate of 100%, while anything where an expert estimates 0% leads to an aggregate estimate of 0%. This violates all central estimate aggregation methods.
The point is that it shows there are additional assumptions of whether the information from the experts is independent etc that is needed for the problem to be well posed, and that without this, no form of mean could be generally correct.
What links here?
- Principled extremizing of aggregated forecasts by Jaime Sevilla (Dec 29, 2021, 6:49 PM; 47 points)
- Jaime Sevilla's comment on When pooling forecasts, use the geometric mean of odds by Jaime Sevilla (Nov 13, 2021, 10:37 AM; 5 points)