But this wouldn’t be global domination in any conventional sense. When humans implement such things, its methods are extremely harsh and inhibit freedoms on all levels of society. A human-run domination would need to enforce such measures with harsh prison time, executions, fear and intimidation, etc. But this is mostly because humans are not very smart, so they don’t know any other way to stop human y from doing x. A powerful AGI wouldn’t have this problem. I don’t think it would even have to be as crude as “burn all GPUs”. It could probably monitor and enforce things so efficiently that trying to create another AGI would be like trying to fight gravity. For a human, it would simply be that you can’t achieve it, no matter how many times you try, almost a new rule interwoven into the fabric of reality. This could probably be made less severe with an implementation such as “can’t achieve AGI that is above intelligence threshold X” or “poses X amount of risk to population”. In this less severe form, humans would still be free to develop AIs that could solve aging, cancer, space travel, etc., but couldn’t develop anything too powerful or dangerous.
Prometheus
I think you make some good points about the assumption an AGI will be a goal-directed agent, but I wouldn’t be so certain that this makes Doom Scenarios less probable, only opens new doors that currently aren’t being researched enough.
In terms of AGI that are just beyond human-level not being much of a threat, I think there are a lot of key assumptions that misunderstand the radical scope of change this would cause. One is speed. Such an intelligence would probably be several orders of magnitude faster than any human intelligence. A second is the ability to replicate. Such a breakthrough would spark radical economic incentive to ramp up computational ability. Even if the first one takes a huge amount of space, the radical amount of investment to scale it I think would quickly change this in a matter of a few years. This would enable a vast number of copies of the AGI to be created. The third is coherence. These AGI copies could all work together in a far more coherent way than any corporation. Corporations are not unified entities. There is a huge amount of disorder within each, and the key decisions are still normally made by just a few individuals, radically slowing progress they can make in terms of company-wide direction and planning. The fourth change that seems very likely is the one that you credited humanity’s power for: communication. These copies could share and communicate with each other with extremely high bandwidth. Humans have to talk, write, read, and listen to share information. This is very low-bandwidth. AGIs can just share their weights with each other. Imagine if every person working on the Manhattan Project had access to all of Von Neumann’s insights, skills, and knowledge. And Einstein’s. And that of the most experienced mechanical engineers, chemists, etc. How long do you think it would have taken them to develop the atom bomb? And given this large scale of new mental power, I don’t see why no one would try to tweak it so that the AGIs start working on self-optimization. The massive incentive for outcompeting other AGIs and mere humans seems far, far too high for this not to be attempted, and I don’t see any reason why this would somehow be impossible or even extremely difficult once you have already created AGIs. Most of the progress in current capabilities of AI have come from a few basic insights from a small number of individuals. In the scope of all of humanity’s available mental power, this was unbelievably low-hanging fruit. If anything, creating more efficient and effective copies seems too easy for an AGI to do. I suspect that this will be achievable before we create AGIs that can even do everything a human can do. In other words, I expect we’ll cross into the danger/weird zone of AI before we even realize it.
Five Areas I Wish EAs Gave More Focus
“Also, the limiting factor for cryonics seems to be more it’s weirdness rather than research?”
Not really. The perfusion techniques haven’t really updated in decades. And the standby teams to actually perform preservation in the event of an accident are extremely spread out and limited. I think some new organizations need to breath life back into cryonics, with clear benchmarks for standards they hope to achieve over a certain timeline. I think Tomorrow Biostasis is doing the kind of thing I’m speaking of, but would love to see more organizations like them.
My guess a big reason is there doesn’t really seem to be any framework to go about working on it, except perhaps on the policy side. Testing out various forms of nanotechnology to see if they’re dangerous might be very bad. Even hypothetically doing that might create information hazards. I imagine we would have to see a few daring EAs blaze the trail for others to follow in. There’s also the obvious skill and knowledge gap. You can’t easily jump into something like nanotech the way you could for something like animal welfare.
Is it actually more cost effective, though? Someone in suspended animation does not eat or consume resources. Unless you mean sometime in the future, but in that future we don’t know what resource constraints will actually be, and we don’t know what we will value most. Preventing irrecoverable loss of sentient minds still seems like a wiser thing to do, given this uncertainty. As for AI Safety, I think we’re facing a talent deficit much more than a financial deficit right now. I’m not sure how much adding, say, 5 more million to the cause will really change at this time.
Thanks! I’ll update to correct this.
4 Key Assumptions in AI Safety
Are there plans from any organizations to support former FTX grantees? I’m not one of them, but know of many who received funding from the FTX regranting program, who are now suddenly without funding and might face clawbacks.
Widening Overton Window—Open Thread
I’m planning on actually doing that. It’ll be the first time I’ve done so in years.
I’m currently thinking that if there are any political or PR resources available to orgs (AI-related or EA) now is the time to use them. Public interest is fickle, and currently most people don’t seem to know what to think, and are looking for serious-seeming people to tell them whether or not to see this as a threat. If we fail to act, someone else will likely hijack the narrative, and push it in a useless or even negative direction. I don’t know how far we can go, or how likely it is, but we can’t assume we’ll get another chance before the public falls back asleep or gets distracted (the US has an election next year, so most discourse will then likely become poisoned). This is especially important for those in the community who are viewed as “serious people” or “serious organizations” (lots of academic credentials, etc.)
It’s almost as if there’s no difference between the two, until someone tells them which side their team supports.
“The strategy of “get a lot of press about our cause area, to get a lot of awareness, even if they get the details wrong” seems to be the opposite of what EA is all about” Yes, and I think this is a huge vulnerability for things like this. Winning the narrative actually matters in the real world.
Humans are not prepared to operate outside their moral training distribution
My quick rebuttal is the flaw you seem to also acknowledge. These different factors that you calculate are not separate variables. They all likely influence the probabilities of each other. (greater capabilities can give rise to greater scaling of manufacturing, since people will want more of it. Greater intelligence can find better forms of efficiency, which means cheaper to run, etc.) This is how you can use probabilities to estimate almost anything is extremely improbable, as you noted.
I think he’s asking if your margin of error is >.01
The primary issue I guess is that the normal rules don’t easily apply here. We don’t have good past data to make predictions, so every new requirement added introduces more complexity (and chaos), which might make it less accurate than using fewer variables. Thinking in terms of “all other factors remaining, what are the odds of x” sounds less accurate, but might be the only way to avoid being consumed by all potential variables. Like, ones you don’t even mention that I could name include “US democracy breaksdown”, “AIs hack the grid”, “AIs break the internet/infect every interconnected device with malware”, etc.* You could just keep adding more requirements until your probabilities drop to near 0, because it’ll be difficult to say with much confidence that any of them are <.01 likely to occur, even though a lot of them probably are. It’s probably better just to group several constraints together, and just give a probability that one or more of them occurs (example: “chance that recession/war/regulation/other slows or halts progress”), rather than trying to assess the likelihood of each one. Ordinarily, this wouldn’t be a problem, but we don’t have any data we could normally work with.
Here’s a brief writeup of some agreements/disagreements I have with the individual contraints.
“We invent algorithms for transformative AGI”
I don’t know how this is only 60%. I’d place >.5 before 2030, let alone 2043. This is just guesswork, but we seem to be one or two breakthroughs away.
“We invent a way for AGIs to learn faster than humans 40%”I don’t really know what this means, why it’s required, or why it’s so low. I see in the paper that it mentions humans being sequential learners that takes years, but AIs don’t seem to work that way. Imagine if GPT4 took years just to learn basic words. AIs also seem to already be able to learn faster than humans. They currently need more data, but less compute than a human brain. Computers can already process information much faster than a brain. And you don’t even need them to learn faster than humans, since once they learn a task, they can just copy that skill to all other AIs. This is a critical point. A human will spend years in Med School just because a senior in the field can’t copy their weights and send them to a grad student.
Also, I’m confused how this at .4, given that its conditional of TAI happening. If you have algorithms for TAI, why couldn’t they also invent algorithms that learn faster than humans? We already see how current AIs can improve algorithmic efficiency (as just one recent example: https://www.deepmind.com/blog/alphadev-discovers-faster-sorting-algorithms). Improving algorithms is probably one of the easiest things a TAI could do, without having to do any physical world experimentation.
“AGI inference costs drop below $25/hr (per human equivalent) 16%”I really don’t see how this is 16%. Once an AI is able to obtain a new capability, it doesn’t seem to cost much to reuse that capability. Example: GPT4, very expensive to train, but it can be used for cents on a dollar afterward. These aren’t mechanical humans, they don’t need to go through repeated training, knowledge expertise, etc. They only need to do it once, and then it just gets copied.
And, like above, if this is conditional on TAI and faster-than-human learning occurring, how is this only at .016? A faster-than-human TAI can (very probably) improve algorithmic efficiency to radically drive down the cost.
“We invent and scale cheap, quality robots 60%”This is one where infrastructure and regulation can bottleneck things, so I can understand at least why this is low.
“We massively scale production of chips and power 46%”If we get TAIs, I imagine scaling will continue or else radically increase. We’re already seeing this, and current AIs have much more limited economic potential. We also don’t know if we actually need to keep scaling or not, since (as I mentioned), algorithmic efficiency might make this unimportant.
“We avoid derailment by human regulation 70%”Maybe?
“We avoid derailment by AI-caused delay 90%”In the paper, it describes this as “superintelligent but expensive AGI may itself warn us to slow progress, to forestall potential catastrophe that would befall both us and it.”
That’s interesting, but if the AI hasn’t coup’d humanity already, wouldn’t this just fall under ‘regulation derails TAI’? Unless there is some other way progress halts that doesn’t involve regulations or AI coups...
“We avoid derailment from wars (e.g., China invades Taiwan) 70%”Possible, but I don’t think this would derail things for 20 years. Maybe 5.
“We avoid derailment from pandemics 90%”Pandemics also increase with the chances of TAI (or, maybe, they go down, depending, AI could possibly detect and predict a pandemic much better). This is one of the issues with all of this, everything is so entangled, and it’s not actually that easy to say which way variables will influence each other. I’m pretty sure it’s not 50⁄50 it goes one way or the other, so it probably does greatly influence it.
“We avoid derailment from severe depressions”Not sure, here. It’s not as though everyone will be going out and buying TPUs with or without economic worries. Not all industries slow or halt, even during a depression. Algorithmic efficiency especially seems unlikely to be affected by this.
Overall, I think the hardware and regulatory constraints are the most likely limiting factors. I’m not that sure about anything else.
*I originally wrote up another AI-related scenario, but decided it shouldn’t be publicly stated at the moment.
If someone manages to create a powerful AGI, and the only cost for most humans is that it burns their GPUs, this seems like an easy tradeoff for me. It’s not great, but it’s mostly a negligible problem for our species. But I do agree using governance and monitoring is a possible option. I’m normally a hardline libertarian/anarchist, but I’m fine going full Orwellian in this domain.