I think the science value of exoplanet missions is really underrated. We should build the TPF. We might be able to literally find aliens or at least worlds where alien civilizations had gone extinct in the past. And learning about what had killed them might help us avoid the same fate, at least for a while. Or we might learn something that we cannot foresee.
1.) Do we need to worry about how long it takes to deploy capital? If a patient philanthropist encounters a “hinge”, will there be enough pre-existing institutions to be able to take advantage of it?
2.) How much should we value the information gained from acting now and “learning by doing” about what works?
3.) I imagine there also should be value for a patient philanthropist to still donate considerably to research now, so we can better identify “hinges” and know how and where to take advantage of them.
Also, I expect to see small engineered pandemics, but only after effective genetic engineering is widespread. So the fact that we haven’t seen any so far is not much evidence.
Yes, that was broadly the response I had in mind as well. Same goes for most of the “unforeseen”/”other” anthropogenic risks; those categories are in the chapter on “Future risks”, and are mostly things Ord appears to think either will or may get riskier as certain technologies are developed/advanced.
Sleepy reply to Tobias’ “Ord’s estimates seem too high to me”: An important idea in the book is that “the per-century extinction risks from “natural” causes must be very low, based in part on our long history of surviving such risks” (as I phrase it in this post). The flipside of that is roughly the argument that we haven’t got strong evidence of our ability to survive (uncollapsed and sans dystopia) a long period with various technologies that will be developed later, but haven’t been yet.
Of course, that doesn’t seem sufficient by itself as a reason for a high level of concern, as some version of that could’ve been said at every point in history when “things were changing”. But if you couple that general argument with specific reasons to believe upcoming technologies could be notably risky, you could perhaps reasonably arrive at Ord’s estimates. (And there are obviously a lot of specific details and arguments and caveats that I’m omitting here.)
Did this end up happening?
What is the significance of the people on the ISS? Are you suggesting that six people could repopulate the human species? And what sort of disaster takes less time than a flight, and only kills people on the ground?
I haven’t looked at this in much detail, but Ord’s estimates seem too high to me. It seems really hard for humanity to go extinct, considering that there are people in remote villages, people in submarines, people in mid-flight at the time a disaster strikes, and even people on the International Space Station. (And yes, there are women on the ISS, I looked that up.) I just don’t see how e.g. a pandemic would plausibly kill all those people.
Also, if engineered pandemics, or “unforeseen” and “other” anthropogenic risks have a chance of 3% each of causing extinction, wouldn’t you expect to see smaller versions of these risks (that kill, say, 10% of people, but don’t result in extinction) much more frequently? But we don’t observe that.
(I haven’t read Ord’s book so I don’t know if he addresses these points.)
I find this very hard to understand.
This is useful feedback. I might need to work on the wording.
Without that I think it is hard to say if 17% is high or not right?
I don’t think I agree with that—I think the important consideration is the number of identified advertised roles of a particular type relative to the number of identified currently filled roles of the same type. Not the number of advertised roles of type A relative to advertised roles of type B. But FWIW the full report is now published.
this seems like weak evidence for bottleneck claims
I agree its weak evidence; I think it’s the weakest of the 5 bullet points above. I find weak evidence useful.
Thanks for the input! If the above bullet points were evidence of funding constraints, then this “more negative reading” would be a plausible alternative explanation. But I’m not following how the above bullet points could be read in this way. Apologies if I’m missing something.
Are you thinking this applies to all 5 of the above bullet points? Or specific bullet points within that group?
We’re doing some message testing on this and can get back to you.
Great work, thanks for writing this up!
I’m wondering how this might affect the public debate on factory farming. Animal advocates sometimes argue that factory farms contribute to antibiotic resistance, and this point may carry much more force in the future. So perhaps one key conclusion is that advocates should emphasise this angle more in the future. (That said, AFAIK the coronavirus doesn’t have anything to do with farmed animals, and my impression from a quick Google search is that the issue of antibiotic resistance is manageable with the right regulations.)
One minor quibble with this post’s language, rather than any of its actual claims: The title includes the phrase “safety by default”, and the terms “optimism” and “optimist” are repeatedly applied to these researchers or their views. The title is reasonable in a sense, as these interviews were partially/mostly about whether AI would be “safe by default”, or why we might believe that it would be, or why these researchers believe that that’s likely. And the use of “optimism”/”optimist” are reasonable in a sense, as these researchers were discussing why they’re relatively optimistic, compared to something like e.g. the “typical MIRI view”.
But it seems potentially misleading to use those phrases here without emphasising (or at least mentioning) that at least some of these researchers think there’s a greater than 1% chance of extinction or other existential catastrophe as a result of AI. E.g., the statement “Rohin reported an unusually large (90%) chance that AI systems will be safe without additional intervention” implies a 10% credence that that won’t be the case. Relevant quote from The Precipice:
In 1939, Enrico Fermi told Szilard the chain reaction was but a ‘remote possibility’ [...]
Fermi was asked to clarify the ‘remote possibility’ and ventured ‘ten percent’. Isidor Rabi, who was also present, replied, ‘Ten percent is not a remote possibility if it means that we may die of it. If I have pneumonia and the doctor tells me that there is a remote possibility that I might die, and it’s ten percent, I get excited about it’
And in this case, the stakes are far greater (meaning no offence to Isidor Rabi).
My guess would be that a decent portion of people who (a) were more used to something like the FHI/80k/Oxford views, and less used to the MIRI/Bay Area views, and (b) read this without having read the interviews in great detail, might think that these researchers believe something like “The chance things go wrong is too small to be worth anyone else worrying about.” Which doesn’t seem accurate, at least for Rohin and I believe for Paul and Adam. (I’m less sure about Robin, and I don’t recall if anyone but Rohin explicitly gave relevant probability estimates.)
To be clear: I don’t think you’re intending to convey that message. And I definitely wouldn’t want to try shut down any statements about AI that don’t sound like “this is a huge deal, everyone get in here now!” I’m just a bit concerned about posts accidentally conveying an overly optimistic/sanguine message when that wasn’t actually their intent, and when it wasn’t supported by the arguments/evidence provided.
(Something informing this comment is my past experience reading a bunch of cognitive science work on how misinformation spreads and can be sticky. Some discussion here, and a particularly relevant paper here.)
Thanks for this post!
There are lots of calls for individuals with views around AI risk to engage with each other and understand the reasoning behind fundamental disagreements.
I strongly share this view, and have therefore quite appreciated this project by AI Impacts. A lot of resources are going into this field, which I’m broadly very supportive of, but it does seem worth gaining a clearer understanding of precisely why, and precisely which approaches should get which portions of those resources.
One other post that I personally found really useful for understanding the various views, the underlying assumptions, and their interconnections was Clarifying some key hypotheses in AI alignment (coauthored by Rohin). I’ve also collected here ~30 works I found that strongly relate to this goal (I plan to update that collection over time, and have now added this post to it as well).
And I’m currently working on a post with a very similar objective, but for longtermist/x-risk strategies more broadly. Hopefully that’ll be out soon.
The link has an extra ‘.’ - https://www.causal.app/
Looks neat, good luck!
Interesting view. It seems to me like it makes sense, but I also feel like it’d be valuable for it to be fleshed out and critiqued further to see how solid it is. (Perhaps this has already been done somewhere—I do feel like I’ve heard vaguely similar arguments here and there.)
Also, arriving at this thread 5 months late, I notice Toby Ord makes a similar argument in The Precipice. He writes about:
a subtle form of correlation—not between two risks, but between risks and the value of the future. There might be risks that are much more likely to occur in worlds with high potential. For example, if it is possible to create artificial intelligence that far surpasses humanity in every domain, this would increase the risk from misaligned AGI, but would also increase the value we could achieve using AGI that was aligned with human values. By ignoring this correlation, the total risk approach underweights the value of work on this risk.
This can be usefully understood in terms of there being a common cause for the risk and the benefit, producing the correlation. A high ceiling on technological capability might be another common cause between a variety of risks and extremely positive futures. I will set this possibility aside in the rest of the book, but it is an important issue for future work to explore.
I think the sentence does make sense, though it is certainly badly written!
Minor: The in-text links to endnotes link to the page for making a new forum post, which presumably is not intended.
Minor: The following sentence right before section 4 starts seems jumbled?
This is true for when the magnitude of warming was comparable to what we are in for in the next 200 years, and when, on a regional basis, the rate of warming was comparable to what we are in for in the next 200 years.
Excellent post. I highly value such summaries of known and relevant factual information, and views on lessons we can learn. Thank you for putting this together.
I’m also curious what prompted you to look into this topic?
This seems like an important consideration.
My thinking is that downvotes should be used sparingly, and only with feedback (or some cited reason). If someone with a high-status/”authoritative” viewpoint is being downvoted, even they deserve to know why. If someone is brave enough to reply with criticism, others could upvote that criticism anonymously (upvotes could remain anonymous).
Maybe the voting can remain anonymous, but each downvote has to be tied to an explanation? E.g. each downvote must reference a comment in the post as an explanation. Once one person properly criticizes with a comment, the others can refer to that comment anonymously as the reason for their downvotes. This gives whoever’s been downvoted a chance to know why and a chance to defend themselves.