I too find this an interesting topic. More specifically, I wonder why Iâve seen as little discussion published in the last few years (rather than from >10 years ago) of nanotech as I have. I also wonder about the limited discussion of things like very long-lasting totalitarianism - though there I donât have reason to believe people recently had reasonably high x-risk estimates; I just sort-of feel like I havenât yet seen good reason to deprioritise investigating that possible risk. (Iâm not saying that there should be more discussion of these topics, and that there are no good reasons for the lack of it, just that I wonder about that.)
I realize that Ordâs risk estimates are his own while the 2008 data is from a survey, but I assume that his views broadly represent those of his colleagues at FHI and others the GCR community.
Iâm not sure thatâs a safe assumption. The 2008 survey youâre discussing seems to have itself involved widely differing views (see the graphs on the last pages). And more generally, the existential risk and GCR research community seems to have widely differing views on risk estimates (see a collection of side-by-side estimates here).
I would also guess that each individualâs estimates might themselves be relatively unstable from one time you ask them to another, or one particular phrasing of the question to another.
Relatedly, Iâm not sure how decision-relevant differences of less than an order of magnitude between different estimates are. (Though such differences could sometimes be decision-relevant, and larger differences more easily could be.)
Thanks Michael, I had seen that but hadnât looked at the links. Some comments:
The cause report from OPP makes the distinction between molecular nanotechnology and atomically precise manufacturing. The 2008 survey seemed to be explicitly considering weaponised molecular nanotechnology as an extinction risk (I assume the nanotechnology accident was referring to molecular nanotechnology as well). While there seems to be agreement that molecular nanotechnology could be a direct path to GCR/âextinction, OPP presents atomically precise manufacturing as being more of an indirect risk, such as through facilitating weapons proliferation. The Grey goo section of the report does resolve my question about why the community isnât talking about (molecular) nanotechnology as an existential risk as much now (the footnotes are worth reading for more details):
âGrey gooâ is a proposed scenario in which tiny self-replicating machines outcompete organic life and rapidly consume the earthâs resources in order to make more copies of themselves.40 According to Dr. Drexler, a grey goo scenario could not happen by accident; it would require deliberate design.41 Both Drexler and Phoenix have argued that such runaway replicators are, in principle, a physical possibility, and Phoenix has even argued that itâs likely that someone will eventually try to make grey goo. However, they believe that other risks from APM are (i) more likely, and (ii) very likely to be relevant before risks from grey goo, and are therefore more worthy of attention.42 Similarly, Prof. Jones and Dr. Marblestone have argued that a âgrey gooâ catastrophe is a distant, and perhaps unlikely, possibility.43
OPPâs discussion on why molecular nanotechnology (and cryonics) failed to develop as scientific fields is also interesting:
First, early advocates of cryonics and MNT focused on writings and media aimed at a broad popular audience, before they did much technical, scientific work âŚ
Second, early advocates of cryonics and MNT spoke and wrote in a way that was critical and dismissive toward the most relevant mainstream scientific fields âŚ
Third, and perhaps largely as a result of these first two issues, these âneighboringâ established scientific communities (of cryobiologists and chemists) engaged in substantial âboundary workâ to keep advocates of cryonics and MNT excluded âŚ
It least in the case of molecular nanotechnology, the simple failure of the field to develop may have been lucky (at least from a GCR reduction perspective) as it seems that the research that was (at the time) most likely to lead to the risky outcomes was simply never pursued.
I too find this an interesting topic. More specifically, I wonder why Iâve seen as little discussion published in the last few years (rather than from >10 years ago) of nanotech as I have. I also wonder about the limited discussion of things like very long-lasting totalitarianism - though there I donât have reason to believe people recently had reasonably high x-risk estimates; I just sort-of feel like I havenât yet seen good reason to deprioritise investigating that possible risk. (Iâm not saying that there should be more discussion of these topics, and that there are no good reasons for the lack of it, just that I wonder about that.)
Iâm not sure thatâs a safe assumption. The 2008 survey youâre discussing seems to have itself involved widely differing views (see the graphs on the last pages). And more generally, the existential risk and GCR research community seems to have widely differing views on risk estimates (see a collection of side-by-side estimates here).
I would also guess that each individualâs estimates might themselves be relatively unstable from one time you ask them to another, or one particular phrasing of the question to another.
Relatedly, Iâm not sure how decision-relevant differences of less than an order of magnitude between different estimates are. (Though such differences could sometimes be decision-relevant, and larger differences more easily could be.)
In case you hadnât seen it: 80,000 Hours recently released a post with a brief discussion of the problem area of atomically precise manufacturing. That also has links to a few relevant sources.
Thanks Michael, I had seen that but hadnât looked at the links. Some comments:
The cause report from OPP makes the distinction between molecular nanotechnology and atomically precise manufacturing. The 2008 survey seemed to be explicitly considering weaponised molecular nanotechnology as an extinction risk (I assume the nanotechnology accident was referring to molecular nanotechnology as well). While there seems to be agreement that molecular nanotechnology could be a direct path to GCR/âextinction, OPP presents atomically precise manufacturing as being more of an indirect risk, such as through facilitating weapons proliferation. The Grey goo section of the report does resolve my question about why the community isnât talking about (molecular) nanotechnology as an existential risk as much now (the footnotes are worth reading for more details):
OPPâs discussion on why molecular nanotechnology (and cryonics) failed to develop as scientific fields is also interesting:
It least in the case of molecular nanotechnology, the simple failure of the field to develop may have been lucky (at least from a GCR reduction perspective) as it seems that the research that was (at the time) most likely to lead to the risky outcomes was simply never pursued.
Update: Probably influenced a bit by this discussion, Iâve now made a tag for posts about Atomically Precise Manufacturing, as well as a link post (with commentary) for that Open Phil report.