Founder of CEEALAR (née the EA Hotel; ceealar.org)
Greg_Colbourn(Greg Colbourn)
Karma: 3,350
I could see this backfiring. What if insilling false beliefs just later led to the meta-belief that deception is useful for control?
I do actually think B is likely, but also don’t think it’s particularly relevant (as A is enough for doom). Mesa-optimisation is a mechanism for box escape that seems very difficult to patch.
The AI that causes doom likely won’t be “evil”; it will just have other uses for the Earth’s atoms. I don’t think we can be confident in buggy AI-related warning shots. Or at least, I can’t see how there would be any that are significant enough to not cause doom, but cause the world to coordinate to stop AGI development, especially given the precedent of Covid and gain-of-function research.
It’s not even necessarily about the AGI directly persuading people to let it out. If the AGI is in anyway useful or significantly economically valuable, people will voluntarily connect it to the internet (assuming they don’t appreciate the existential risk!) e.g. people seem to have no qualms about connecting LLMs/Transformers to the internet already. Regarding your A and B, A is already sufficient for our doom! It doesn’t require every single AGI to escape; one is one too many.
Mesa-optimisation is where an optimiser emerges internal to the AI that is optimising for something other than the goal given to the AI. Convergent instrumental goals also come into it (e.g. gaining access to the internet). So you could imagine a mesa-optimiser emerging that has the goal of gaining or access to information, or gaining access to more resources in general (with the subgoal of taking out humanity to make this easier).
With that in mind, can you reanswer questions 1 and 2, this time with no internet. Just the man, his memories of a hunter gatherer lifestyle, and a million years to think and ponder.
That would obviously be no for both. But that isn’t relevant here. The AGI will have access to the internet and its vast global array of sensors, and it will be able to communicate with millions of people and manipulate them into doing things for it (via money or otherwise). If it doesn’t have access to begin with—i.e. it’s boxed—it wouldn’t remain that way for long (it would easily be able to persuade someone to let it out, or otherwise engineer a way out, e.g. via a mesaoptimiser).
Assuming the man in the cave has full access to the Internet (which would be very easy for an AGI to get), 1. yes, 2. yes, 3. maybe, 4. yes. And for 3, it would very likely escape the box, so would end up as yes.
I think it’s a failure of imagination to think otherwise. A million years is a really long time! You mention combinatorial explosions making things “impossible”, but we’re talking about AGIs (and humans) here—intelligences capable of collapsing combinatorial explosions with leaps of insight.
Do you think, in the limit of a simulation on the level of recreating the entire history of evolution, including humans and our civilisations, these things would still be impossible? Do you think that we are at the upper limit (or very close to it) of theoretically possible intelligence? Or theoretically possible technology?
The hypothetical is that the “cave” man has access to our Internet! (As the AI would). So they would know about the periodic table. They would also have access to labs throughout the world via being able to communicate with the workers in them (as the AI would), view camera and data feeds etc. Imagine what you could achieve if you could think 1,000,000x faster and use the internet—inc chatting/emailing with many thousands of humans—at that speed. A lifetime’s worth of work done every 10 minutes. And that’s just assuming the AI is only human level (and doesn’t get smarter!)
Belfort ended up paying 50% of his future income towards restitution
No he didn’t! From Wikipedia [my emphasis in bold]:
Restitution
Belfort’s restitution agreement required him to pay 50% of his income towards restitution to the 1,513 clients he defrauded until 2009, with a total of $110 million in restitution further mandated. About $10 million of the $110 million that had been recovered by Belfort’s victims as of 2013 was the result of the sale of forfeited properties.[32]
In October 2013, federal prosecutors filed a complaint against Belfort. Several days later, the U.S. government withdrew its motion to find Belfort in default of his payments, after his lawyers argued that he had only been responsible for paying 50% of his salary to restitution up until 2009, and not since. The restitution he paid during his parole period (after leaving prison) amounted to $382,910 in 2007, $148,799 in 2008, and $170,000 in 2009. Following this period, Belfort began negotiating a restitution payment plan with the U.S. government.[33] The final deal that Belfort made with the government was to pay a minimum of $10,000 per month for life towards the restitution, after a judge ruled that Belfort was not required to pay 50% of his income past the end of his parole. Belfort has claimed that he is additionally putting the profits from his U.S. public speaking engagements and media royalties towards the restitution, in addition to the $10,000 per month.[34]
Prosecutors also said that he had fled to Australia to avoid taxes and conceal his assets from his victims,[35] but later recanted their statement, which had been given to The Wall Street Journal,[36] by issuing Belfort an official apology and requesting that The Wall Street Journal print a retraction.[37] Belfort also claimed on his website and elsewhere that he intended to request that “100% of the royalties” from his books and The Wolf of Wall Street film be turned over to victims. But in June 2014, spokesmen for the U.S. attorney said that Belfort’s claim was “not factual”,[38] and that he had received money from the initial sale of the movie rights that was not entirely put towards his restitution repayment.[36] BusinessWeek reported that Belfort had paid only $21,000 toward his restitution obligations out of approximately $1.2 million paid to him in connection with the film before its release.[39] Belfort has stated that the government refused his offer to put 100% of his book deal money towards his restitution.[39][40]
I think this is pretty outrageous (he’s paid only ~11% of what he owes, and will likely only ever pay ~12% max), and sends completely the wrong message. i.e. basically he’s still a rich celebrity making lots of money off the stories of his wrongdoing; he only served 22 months in prison; this could easily be interpreted by a lot of people in terms of “financial crime pays off”(!). Who knows, maybe FTX/Alameda were even inspired by it!?
Dustin Moskovitz and Jaan Tallinn were already EA ~billionaire donors well before 2018. They haven’t done anything analogous to what SBF/FTX/Alameda did. What examples are you thinking of?
It’s not “absolute perfection” to create nanotech. Biology has already done it many times via evolution. And extinctions of species happen regularly in nature. Also, there is the Internet and a vast array of sensors attached to it, so it’s nothing like being in a cave. Testing can be done very rapidly in parallel and with viewing things at very high temporal and spatial resolution, so plenty of empirical evidence can be accumulated in a short (wall clock) time (but long thinking time for the AI).
The same prehistoric man with access to the Internet in a speeded up simulation thinking for fifty thousand years of subjective time (and the ability to communicate with hundreds of thousands of humans simultaneously given the speed advantage) could also make nanotech (or other new tech current humans haven’t yet produced).
It is quantity, and speed as well. And access to information. A prehistoric immortal human with access to the Internet who could experience fifty thousand years of thinking time in a virtual world in 5 hours of wall clock time totally could build a nuke!
Awesome news, thanks! Looking forward to hearing more about the operationalization, and logistics.
Wondering if there could be a way to incorporate the fact that doom is conditional on year that TAI is developed (i.e how well developed AI Alignment/strategy/governance is when TAI is possible)? P(doom|TAI in year 20xx) and P(10% chance of TAI in year 20xx) are both important questions.
They’re doing it! Well, something similar, in early 2023. Thanks OpenPhil!
Ah, but are you factoring in thinking speed? An AI could do tens of thousands of years thinking in a few hours if it took over significant amounts of the world’s computing power.
It sounds ridiculously difficult to us, but that’s because we are human. I imagine that a chimp would think that “take over the world and produce enough food for billions of people” is similarly difficult (or indeed, “kill all chimps”), or an ant colony not being able to conceive of its destruction by human house builders. There is nothing in the laws of physics to say that we are anywhere close to the upper limit of optimisation capability (intelligence). A superintelligent AI won’t just be like a super-smart human, it will be on a completely different level (as we are to chimps, or ants). There is more than enough information out there (online) for it to reverse engineer anything it needed.
I think even the foom skeptics (e.g. Christiano) think that a foom will eventually happen, even if there is a slow-takeoff over many years first.
Why is it far more likely? Sounds kind of Just-World Fallacy / Hollywood / human-like fallibility to me. Nature doesn’t care about our survival, we are Beyond the Reach of God etc.
Re Geeks, MOPs, and Sociopaths, SBF was one of the original Geeks (into EA when it was still small), so I don’t think it makes sense to use this framing here. SBF seems to have developed into a Sociopath after he made it big (and EA became similarly big, somewhat independently). Does Chapman have anything to say about that (Geeks → Sociopaths)?
How about nanoprobes covering every cubic meter of the Earth’s habitable environment undetected, and then giving everyone a lethal dose of botulinum toxin simultaneously? AGI x-risk is usually thought of in terms of an adversary that can easily outsmart all of humanity put together. The first AGI might be fallible, but what if the first extinction-threatening AGI was not (and never blew it’s cover until it was too late for us). Can we take that risk?
Re “fuck regulators”, I guess it’s possible that in the mental state he was in, he thought this would go down well with the crypto community and he could regain some of their trust that way, or something. Recently the crypto community had turned against him for being too cosy with regulators in the US. See e.g. this clip that went viral on crypto twitter recently (28 Oct), and the reaction to his proposed regulations.
AGI might be rare/hard to build at first. But proliferation seems highly likely—once one company makes AGI, how much longer until 5 companies do? Evolutionary pressure will be another thing. More capable AGIs will outcompete less capable ones, once rewriting of code or mesa-optimisation starts. They will be more likely to escape boxes.
Even with relatively minor warning shots, what’s to stop way worse happening 6-24 months later? Would there really be a rigorously enforced global moratorium on AGI research after a few thousand deaths?
Whether or not Covid was a lab leak, gain of function research still hasn’t been meaningfully regulated. Despite the now very clear danger and extreme cost of pandemics. It seems that the curiosity, misguided values and prestige incentives of a small number of academic scientists trumps the safety of billions of humans and $trillions in GDP. What hope do we have for regulating an AGI industry that has thousands of times more resources backing it?