Don’t panic.
jai
Karma: 939
Will AI kill everyone? Here’s what the godfathers of AI have to say [RA video]
I was pretty happy to see the animated-birds-explaining-things crossover with the animated-dogs-explaining-things, especially as I’m now working with the animated explainer dogs.
We could combine them as “Eradication Day” on May 8, like the US does for “President’s Day” in February.
Replying a year later—I think you have a point about the soldier mindset. The reason I wrote it this way originally was because I was writing it for myself first. I felt afraid to involve myself in things for fear I would feel responsible ever after, I was often afraid of doing a little because I could not commit myself to doing a lot, and I felt an aversion to a lot of positive-sum interactions. I’ve gotten better at those things, but it’s still hard.
I think a lot has been written to this effect since by other people, but if I were to write a follow-up it would be focusing on the alternative rather than attacking what I called “The Copenhagen Interpretation” here. It’s okay to help a little. If something helps you a lot and someone else a little and no one gets hurt or is worse off, then the world is better off. Be wary of the temptations to overlook harms—but always remember that the true goal is to help, and sacrifice is only a means. If you can help without sacrifice, so much the better.
Apologies for making the coordination problem worse. I actually picked December 9th before I knew about the two dates, inspired by this comment from B_For_Bandana on LessWrong in 2013 - in particular the last paragraph:
Because Smallpox Eradication Day marks one of the most heroic events in the history of the human species, it is not surprising that it has become a major global holiday in the past few decades, instead of inexplicably being an obscure piece of trivia I had to look up on Wikipedia. I’m just worried that as time goes on it’s going to get too commercialized. If you’re going to a raucous SE Day party like I am, have fun and be safe.
I thought about this when I was reading Kelsey’s piece. Thanks in part to this comment, the alternate reality it itself described is actually starting to become real.
Anyway—in the absence of a solution to the coordination problem I can at least help make the discrepancy more explicit:
Thank you for the praise. I just want more people to know this story.
In addition to being just a really impressive person, I can’t get over how cinematic Dr. Sapara’s life is.
This is profoundly silly, but it’s a thought I can’t get out of my head: “The doctor, in the course of his travels, investigates a mysterious force that’s killing people, discovers a cult whose priests are using the living weapon to kill and keep the people in fear. He confronts them with the truth and defeats them without firing a shot” sounds like the synopsis of an episode of Doctor Who. He even spends some time in the UK! “The doctor served as a rogue spy in the war against Smallpox and singlehandedly defeated a cult of death by talking” is an actual thing that really happened, and no one knows about it.
The Legend of Dr. Oguntola Sapara
Today, May 7 2023, I am planning to make some updates to this piece for historical accuracy. As part of this update, I’m including some citations and commentary in this comment.
I feel torn writing about some of these, because it’s attempting to summarize and quantify an incomprehensible quantity of suffering and death. The numbers are important, to recognize what was lost, to honor the fallen as best we can, and to emphasize the importance of killing Smallpox. But quantifying that loss, for me at least, requires temporarily embracing a detached attitude, especially when the clues are many and uncertainty is abundant. So that is how I will be describing citations and numbers for the remainder of this comment—and then, when I’ve collected as many citations and clarified as many estimations as I can, I will grapple with them as a human.
Much of the narrative is just from Wikipedia’s page on the History of Smallpox. This has changed a bit since 2014.
The figure “500 Million” comes from Donald Henderson by way of Our World in Data.
”Five million in India” comes from this article by Eugene D. Robin, Robert F. McCauley in the Indian Journal of Medical Ethics, 1997.“[A] thousand Europeans every day in the 18th century” is from the Wikipedia page. Specifically, “400,000“ annual deaths.
[Update] The text originally said that Smallpox killed “more than fifty million Native Americans”. I cannot find any source I might have originally used for this claim. When attempting to research this more recently, I came across the 2019 paper “Earth system impacts of the European arrival and Great Dying in the Americas after 1492”, which estimates 55 million all-cause deaths from European contact across the Americas between 1492 and 1600. “Counting the Dead: Estimating the Loss of Life in the Indigenous Holocaust, 1492-Present” is part of a body of literature exploring the interplay between disease and intentional violence, which further complicates things. Lots of things can make people vulnerable to disease. If a family is forced to relocate, goes hungry, and subsequently are weaker when they fall ill, it seems incorrect to say that the illness alone killed them. All told, disease is widely agreed to be the most significant contributor to the great dying of the Indigenous Peoples of the Americas, and of the diseases Smallpox was probably the most significant killer. Given the complexities and uncertainties here, when I finish writing this comment, I intend to update the text to “over ten million Native Americans” as a lower bound that I feel 80% confident about. I think I could refine this further with more research—this is still extremely ad hoc—but this is as much time as I can afford at the moment.
“more soldiers and civilians than any weapon, any soldier, any army”: See Wikipedia’s “List of anthropogenic disasters by death toll”
[Update] “In China, in the 10th century, humanity began to fight back.” I believe I originally got this figure from Our World In Data, but now believe it to be inaccurate. There are “references” that say the tradition dates back to the tenth century, but the evidence is sparse. Going by Wikipedia’s article on Variolation, I’m updating this to “the 15th century”.
Edward Jenner, 1796: Wikipedia. I’m choosing the day of the first inoculation (James Phipps) on May 14, 1796 rather than the publication date of the resulting paper, Inquiry into the Variolae vaccinae known as the Cow Pox, in 1798.
From here on most of the narrative can be traced through the Eradication section of Wikipedia’s article on Smallpox:
- “On August 26, 1807, Bavaria became the first country in the world to introduce compulsory vaccinations.”
—“In Northern Europe a number of countries had eliminated smallpox by 1900”.
- “In 1958 Professor Viktor Zhdanov, Deputy Minister of Health for the USSR, called on the World Health Assembly to undertake a global initiative to eradicate smallpox.”
—“In 1966 an international team, the Smallpox Eradication Unit, was formed under the leadership of an American, Donald Henderson.[127] In 1967, the World Health Organization intensified the global smallpox eradication by contributing $2.4 million annually to the effort, and adopted the new disease surveillance method promoted by Czech epidemiologist Karel Raška.”
—“To eradicate smallpox, each outbreak had to be stopped from spreading, by isolation of cases and vaccination of everyone who lived close by.[129] This process is known as “ring vaccination”.”
—“The WHO established a network of consultants who assisted countries in setting up surveillance and containment activities.”
—“donations of vaccine were provided primarily by the Soviet Union and the United States”
—“The last naturally occurring case of indigenous smallpox (variola minor) was diagnosed in Ali Maow Maalin, a hospital cook in Merca, Somalia, on 26 October 1977”
—“The global eradication of smallpox was certified, based on intense verification activities, by a commission of eminent scientists on 9 December 1979 and subsequently endorsed by the World Health Assembly on 8 May 1980.”
Most Leading AI Experts Believe That Advanced AI Could Be Extremely Dangerous to Humanity
The Rational Animations team has animated this, with narration by Robert Miles. It’s great, as is the rest of their growing body of work.
The current form appears to only allow uploading image files; I can upload a PNG, but not an SVG. This is probably just as well in my case, as the SVG only makes it more painfully obvious that I have no idea how to use Inkscape, but it seems like unintended behavior you might want to change.
I’d like to apologize for “What Almost Was” being a dead link now—I’ve been trying to retire that blog for a while now while preserving the content elsewhere. “What Almost Was” can now be found here.
Here’s a quick attempt at a subset of conjunctive assumptions in Nate’s framing:
- The functional ceiling for AGI is sufficiently above the current level of human civilization to eliminate it
- There is a sharp cutoff between non-AGI AI and AGI, such that early kind-of-AGI doesn’t send up enough warning signals to cause a drastic change in trajectory.
- Early AGIs don’t result in a multi-polar world where superhuman-but-not-godlike agents can’t actually quickly and recursively self-improve, in part because none of them wants any of the others to take over—and without being able to grow stronger, humanity remains a viable player.
It doesn’t seem to me that titotal is assuming MI is solved; having direct access to the brain doesn’t give you full insight into someone’s thoughts either, because neuroscience is basically a pile of unsolved problems with growing-but-still-very-incomplete-picture of low-level and high-level details. We don’t even have a consensus on how memory is physically implemented.
Nonetheless, if you had a bunch of invasive probes feeding you gigabytes/sec of live data from the brain of the genius general of the opposing army, it would be extremely likely to be useful information.
A really interesting thing is that, at the moment, this appears in practice to be a very-asymmetrical advantage. The high-level reasoning processes that GPT-4 implements don’t seem to be able to introspect about fine-grained details, like “how many tokens are in a given string”. The information is obviously and straightforwardly part of the model, but absent external help the model doesn’t seem to bridge the gap between low-level implementation details and high-level reasoning abilities—like us.
Interesting perspective, although I’m not sure how much we actually disagree. “Complicated and open”, to me reads as “difficult”
Is there a rephrasing of the initial statement you would endorse that makes this clearer? I’d suggest “If you apply a security mindset (Murphy’s Law) to the problem of AI alignment, it should quickly become apparent that we do not currently possess the means to ensure that any given AI is safe.”
But that’s not a function of whether “a lot of the fixed costs (like transaction costs) have been paid”. That is very specifically referring to break-even relative to costs already expended, not to counterfactual spending. It’s okay if you’ve changed your mind, but I don’t think that what you said originally is consistent with this comment.
Reply here—https://forum.effectivealtruism.org/posts/xof7iFB3uh8Kc53bG/why-did-cea-buy-wytham-abbey?commentId=tZP9seDyxvS7ot4t7
tldr: this is way more ink than should be spilt over a minor rhetorical point, but I believe it’s a real and meaningful (though minor) strawmanning that selects a weaker opponent for the “keep” position to face.
I labeled this a nit in the hopes of avoiding this kind of deep spiral, but—“commit” is a significant intensifier that specifically implies overriding or ignoring opposing reasons beyond what would normally be expected.
“Why not sell the abbey?” is seeking countervailing reasons while implicitly framing “sell” as the default option to overcome.
”Should they commit to selling the abbey?” is seeking to ignore all possible countervailing reasons and implicitly frames “keep” as the default option which can only be overcome with an overwhelmingly strong argument.
Replacing the former with the latter is choosing a weaker opponent for the “keep” position to overcome.
This is a minor rhetorical thing that does not warrant as many words as this, but it is a real and meaningful thing.
Nit: I was very explicitly asking why not sell, not suggesting a commitment to sell; I don’t appreciate the rhetorical pivot to argue against a point I was not making.
Yes, they should check that it makes financial sense. Though break-even shouldn’t be an anchor or decision point—what matters is whether the money from a sale could be better spent elsewhere, which doesn’t really have anything to do with money already spent.
If it doesn’t make sense to sell, then they shouldn’t sell—but they should also be ready to admit that they made an error in their initial analysis, because it would mean that the bulk of the cost is effectively gone and the true price in opportunity cost was significantly higher than the marginal costs implied.
To a first approximation, it’s either a mostly-recoverable expense that didn’t really cost that much when you consider it maintains its value as a durable good, or it’s not financially feasible to sell and all of those funds are sunk. It can’t be both.
Updates on Luthien since I filled this out (which I probably should have sent to Alexandra, sorry):
- We’re now definitively a non-profit
- Our site is at https://luthienresearch.org
- You can contact us at contact@luthienresearch.org
- We’ve gotten a seed grant from the AI Safety Tactical Opportunities Fund