But this wouldn’t be global domination in any conventional sense. When humans implement such things, its methods are extremely harsh and inhibit freedoms on all levels of society. A human-run domination would need to enforce such measures with harsh prison time, executions, fear and intimidation, etc. But this is mostly because humans are not very smart, so they don’t know any other way to stop human y from doing x. A powerful AGI wouldn’t have this problem. I don’t think it would even have to be as crude as “burn all GPUs”. It could probably monitor and enforce things so efficiently that trying to create another AGI would be like trying to fight gravity. For a human, it would simply be that you can’t achieve it, no matter how many times you try, almost a new rule interwoven into the fabric of reality. This could probably be made less severe with an implementation such as “can’t achieve AGI that is above intelligence threshold X” or “poses X amount of risk to population”. In this less severe form, humans would still be free to develop AIs that could solve aging, cancer, space travel, etc., but couldn’t develop anything too powerful or dangerous.
Prometheus
I think you make some good points about the assumption an AGI will be a goal-directed agent, but I wouldn’t be so certain that this makes Doom Scenarios less probable, only opens new doors that currently aren’t being researched enough.
In terms of AGI that are just beyond human-level not being much of a threat, I think there are a lot of key assumptions that misunderstand the radical scope of change this would cause. One is speed. Such an intelligence would probably be several orders of magnitude faster than any human intelligence. A second is the ability to replicate. Such a breakthrough would spark radical economic incentive to ramp up computational ability. Even if the first one takes a huge amount of space, the radical amount of investment to scale it I think would quickly change this in a matter of a few years. This would enable a vast number of copies of the AGI to be created. The third is coherence. These AGI copies could all work together in a far more coherent way than any corporation. Corporations are not unified entities. There is a huge amount of disorder within each, and the key decisions are still normally made by just a few individuals, radically slowing progress they can make in terms of company-wide direction and planning. The fourth change that seems very likely is the one that you credited humanity’s power for: communication. These copies could share and communicate with each other with extremely high bandwidth. Humans have to talk, write, read, and listen to share information. This is very low-bandwidth. AGIs can just share their weights with each other. Imagine if every person working on the Manhattan Project had access to all of Von Neumann’s insights, skills, and knowledge. And Einstein’s. And that of the most experienced mechanical engineers, chemists, etc. How long do you think it would have taken them to develop the atom bomb? And given this large scale of new mental power, I don’t see why no one would try to tweak it so that the AGIs start working on self-optimization. The massive incentive for outcompeting other AGIs and mere humans seems far, far too high for this not to be attempted, and I don’t see any reason why this would somehow be impossible or even extremely difficult once you have already created AGIs. Most of the progress in current capabilities of AI have come from a few basic insights from a small number of individuals. In the scope of all of humanity’s available mental power, this was unbelievably low-hanging fruit. If anything, creating more efficient and effective copies seems too easy for an AGI to do. I suspect that this will be achievable before we create AGIs that can even do everything a human can do. In other words, I expect we’ll cross into the danger/weird zone of AI before we even realize it.
“Also, the limiting factor for cryonics seems to be more it’s weirdness rather than research?”
Not really. The perfusion techniques haven’t really updated in decades. And the standby teams to actually perform preservation in the event of an accident are extremely spread out and limited. I think some new organizations need to breath life back into cryonics, with clear benchmarks for standards they hope to achieve over a certain timeline. I think Tomorrow Biostasis is doing the kind of thing I’m speaking of, but would love to see more organizations like them.
My guess a big reason is there doesn’t really seem to be any framework to go about working on it, except perhaps on the policy side. Testing out various forms of nanotechnology to see if they’re dangerous might be very bad. Even hypothetically doing that might create information hazards. I imagine we would have to see a few daring EAs blaze the trail for others to follow in. There’s also the obvious skill and knowledge gap. You can’t easily jump into something like nanotech the way you could for something like animal welfare.
Is it actually more cost effective, though? Someone in suspended animation does not eat or consume resources. Unless you mean sometime in the future, but in that future we don’t know what resource constraints will actually be, and we don’t know what we will value most. Preventing irrecoverable loss of sentient minds still seems like a wiser thing to do, given this uncertainty. As for AI Safety, I think we’re facing a talent deficit much more than a financial deficit right now. I’m not sure how much adding, say, 5 more million to the cause will really change at this time.
Thanks! I’ll update to correct this.
Are there plans from any organizations to support former FTX grantees? I’m not one of them, but know of many who received funding from the FTX regranting program, who are now suddenly without funding and might face clawbacks.
I’m planning on actually doing that. It’ll be the first time I’ve done so in years.
I’m currently thinking that if there are any political or PR resources available to orgs (AI-related or EA) now is the time to use them. Public interest is fickle, and currently most people don’t seem to know what to think, and are looking for serious-seeming people to tell them whether or not to see this as a threat. If we fail to act, someone else will likely hijack the narrative, and push it in a useless or even negative direction. I don’t know how far we can go, or how likely it is, but we can’t assume we’ll get another chance before the public falls back asleep or gets distracted (the US has an election next year, so most discourse will then likely become poisoned). This is especially important for those in the community who are viewed as “serious people” or “serious organizations” (lots of academic credentials, etc.)
It’s almost as if there’s no difference between the two, until someone tells them which side their team supports.
“The strategy of “get a lot of press about our cause area, to get a lot of awareness, even if they get the details wrong” seems to be the opposite of what EA is all about” Yes, and I think this is a huge vulnerability for things like this. Winning the narrative actually matters in the real world.
My quick rebuttal is the flaw you seem to also acknowledge. These different factors that you calculate are not separate variables. They all likely influence the probabilities of each other. (greater capabilities can give rise to greater scaling of manufacturing, since people will want more of it. Greater intelligence can find better forms of efficiency, which means cheaper to run, etc.) This is how you can use probabilities to estimate almost anything is extremely improbable, as you noted.
I think he’s asking if your margin of error is >.01
The primary issue I guess is that the normal rules don’t easily apply here. We don’t have good past data to make predictions, so every new requirement added introduces more complexity (and chaos), which might make it less accurate than using fewer variables. Thinking in terms of “all other factors remaining, what are the odds of x” sounds less accurate, but might be the only way to avoid being consumed by all potential variables. Like, ones you don’t even mention that I could name include “US democracy breaksdown”, “AIs hack the grid”, “AIs break the internet/infect every interconnected device with malware”, etc.* You could just keep adding more requirements until your probabilities drop to near 0, because it’ll be difficult to say with much confidence that any of them are <.01 likely to occur, even though a lot of them probably are. It’s probably better just to group several constraints together, and just give a probability that one or more of them occurs (example: “chance that recession/war/regulation/other slows or halts progress”), rather than trying to assess the likelihood of each one. Ordinarily, this wouldn’t be a problem, but we don’t have any data we could normally work with.
Here’s a brief writeup of some agreements/disagreements I have with the individual contraints.
“We invent algorithms for transformative AGI”
I don’t know how this is only 60%. I’d place >.5 before 2030, let alone 2043. This is just guesswork, but we seem to be one or two breakthroughs away.
“We invent a way for AGIs to learn faster than humans 40%”I don’t really know what this means, why it’s required, or why it’s so low. I see in the paper that it mentions humans being sequential learners that takes years, but AIs don’t seem to work that way. Imagine if GPT4 took years just to learn basic words. AIs also seem to already be able to learn faster than humans. They currently need more data, but less compute than a human brain. Computers can already process information much faster than a brain. And you don’t even need them to learn faster than humans, since once they learn a task, they can just copy that skill to all other AIs. This is a critical point. A human will spend years in Med School just because a senior in the field can’t copy their weights and send them to a grad student.
Also, I’m confused how this at .4, given that its conditional of TAI happening. If you have algorithms for TAI, why couldn’t they also invent algorithms that learn faster than humans? We already see how current AIs can improve algorithmic efficiency (as just one recent example: https://www.deepmind.com/blog/alphadev-discovers-faster-sorting-algorithms). Improving algorithms is probably one of the easiest things a TAI could do, without having to do any physical world experimentation.
“AGI inference costs drop below $25/hr (per human equivalent) 16%”I really don’t see how this is 16%. Once an AI is able to obtain a new capability, it doesn’t seem to cost much to reuse that capability. Example: GPT4, very expensive to train, but it can be used for cents on a dollar afterward. These aren’t mechanical humans, they don’t need to go through repeated training, knowledge expertise, etc. They only need to do it once, and then it just gets copied.
And, like above, if this is conditional on TAI and faster-than-human learning occurring, how is this only at .016? A faster-than-human TAI can (very probably) improve algorithmic efficiency to radically drive down the cost.
“We invent and scale cheap, quality robots 60%”This is one where infrastructure and regulation can bottleneck things, so I can understand at least why this is low.
“We massively scale production of chips and power 46%”If we get TAIs, I imagine scaling will continue or else radically increase. We’re already seeing this, and current AIs have much more limited economic potential. We also don’t know if we actually need to keep scaling or not, since (as I mentioned), algorithmic efficiency might make this unimportant.
“We avoid derailment by human regulation 70%”Maybe?
“We avoid derailment by AI-caused delay 90%”In the paper, it describes this as “superintelligent but expensive AGI may itself warn us to slow progress, to forestall potential catastrophe that would befall both us and it.”
That’s interesting, but if the AI hasn’t coup’d humanity already, wouldn’t this just fall under ‘regulation derails TAI’? Unless there is some other way progress halts that doesn’t involve regulations or AI coups...
“We avoid derailment from wars (e.g., China invades Taiwan) 70%”Possible, but I don’t think this would derail things for 20 years. Maybe 5.
“We avoid derailment from pandemics 90%”Pandemics also increase with the chances of TAI (or, maybe, they go down, depending, AI could possibly detect and predict a pandemic much better). This is one of the issues with all of this, everything is so entangled, and it’s not actually that easy to say which way variables will influence each other. I’m pretty sure it’s not 50⁄50 it goes one way or the other, so it probably does greatly influence it.
“We avoid derailment from severe depressions”Not sure, here. It’s not as though everyone will be going out and buying TPUs with or without economic worries. Not all industries slow or halt, even during a depression. Algorithmic efficiency especially seems unlikely to be affected by this.
Overall, I think the hardware and regulatory constraints are the most likely limiting factors. I’m not that sure about anything else.
*I originally wrote up another AI-related scenario, but decided it shouldn’t be publicly stated at the moment.
I’m starting to think that the EA community will scare themselves into never taking any action at all. I don’t really feel like going over this point-for-point, because I think this post demonstrates a much greater failure at rationality. The short answer is: you’re doing cost/benefit analysis wrong. You’re zooming in on every possible critique, and determining it’s an organization that shouldn’t have talent directed toward it. Every alignment organization right now has relatively poor results. But the strategy for what to do with that isn’t funneling talent into ones that have slightly better results, but encouraging the spread of talent among many different approaches.
Thank you for voicing your concerns and frustrations. I think many in the EA Community need to hear this sort of thing, even if they don’t like it. To give an idea of why I think EAs have responded this way, I think it’s because many are stuck in certain defense mechanisms that make them think they need a certain PR face at all times. We’re used to being criticized, slammed by the media, and have our views either not taken seriously, or have them depicted as dangerous or cultish. In a sense, I think the branding of ‘EA’ inhibits many from being willing to act quickly. They like the longterm strategy. The calculated, carefully built strategies, where every piece is carefully added on top of the other. But I agree with you that the situation has changed. We may not have much time to act, and most of the progress made toward getting the public’s attention on taking AI Risk seriously has happened outside of EA. I think EA has an ivory tower problem, and needs to provide channels for people outside it to take action.
The following is a conversation between myself in 2022, and a newer version of myself earlier this year.
On the Nature of Intelligence and its “True Name”:
2022 Me: This has become less obvious to me as I’ve tried to gain a better understanding of what general intelligence is. Until recently, I always made the assumption that intelligence and agency were the same thing. But General Intelligence, or G, might not be agentic. Agents that behave like RLs may only be narrow forms of intelligence, without generalizability. G might be something closer to a simulator. From my very naive perception of neuroscience, it could be that we (our intelligence) is not agentic, but just simulates agents. In this situation, the prefrontal cortex not only runs simulations to predict its next sensory input, but might also run simulations to predict inputs from other parts of the brain. In this scenario, “desire” or “goals”, might be simulations to better predict narrowly-intelligent agentic optimizers. Though the simulator might be myopic, I think this prediction model allows for non-myopic behavior, in a similar way GPT has non-myopic behavior, despite only trying to predict the next token (it has an understanding of where a future word “should” be within the context of a sentence, paragraph, or story). I think this model of G allows for the appearance of intelligent goal-seeking behavior, long-term planning, and self-awareness. I have yet to find another model for G that allows for all three. The True Name of G might be Algorithm Optimized To Reduce Predictive Loss.
2023 Me: interesting, me’22, but let me ask you something: you seem to think this majestic ‘G’ is something humans have, but other species do not, and then name the True Name of ‘G’ to be Algorithm Optimized To Reduce Predictive Loss. Do you *really* think other animals don’t do this? How long is a cat going to survive if it can’t predict where it’s going to land? Or where the mouse’s path trajectory is heading? Did you think it was all somehow hardcoded in? But cats can jump up on tables, and those weren’t in the ancestry environment, there’s clearly some kind of generalized form of prediction occurring. Try formulating that answer again, but taboo “intelligence”, “G”, “agent”, “desire”, and “goal”. I think the coherence of it breaks down.
Now, what does me’23 think? Well, I’m going to take a leaf from my own book, and try to explain what I think without the words mentioned above. There are predictive mechanisms in the Universe that can run models of what things in the Universe might do in future states. Some of these predictive mechanism are more computationally efficient than others. Some will be more effective than others. A more effective and efficient predictive mechanism, with a large input of information about the Universe, could be a very powerful tool. If taken to the theoretical (not physical) extreme, that predictive mechanism would hold models of all possible future states. It could then, by accident or intention, guide outcomes toward certain future states over others.
2022 Me: according to this model, humans dream because the simulator is now making predictions without sensory input, gradually creating a bigger and bigger gap from reality. Evidence to support this is from sensory-deprivation tanks, where humans, despite being awake, have dream-like states. I also find it interesting that people who exhibit Schizophrenia, which involves hallucinations (like dreams do), can tickle themselves. Most people can be tickled by others, but not themselves. But normal people on LSD can do this, and also can have hallucinations. My hairbrained theory is that something is going wrong when initializing new tokens for the simulator, which results in hallucinations from the lack of correction from sensory input, and a less strong sense of self because of a lack of correction from RL agents in other parts of the brain.
2023 Me: I don’t want to endorse crackpot theories from Me’22, so I’m just going to speak from feelings and fuzzy intuitions here. I will say hallucinations from chatbots are interesting. When getting one to hallucinate, it seems to be kind of “making up reality as it goes along”. You say it’s a Unicorn, and it will start coming up with explanations for why it’s a Unicorn. You say it told you something you never told it, and it will start acting as though it did. I have to admit it does have a strange resemblance to dreams. I find myself in New York, but remember that I had been in Thailand that morning, and I hallucinate a memory of boarding a plane. I wonder where I got the plane ticket, and I hallucinate another memory of buying one. These are not well-reasoned arguments, though, so I hope Me’24 won’t beat me up too much about them.
2022 Me: I have been searching for how to test this theory. One interest of mine has been mirrors.
2023 Me: Don’t listen to Me’22 on this one. He thought he understood something, but he didn’t. Yes, the mirror thing is interesting in animals, but it’s probably a whole different thing, not the same thing.
Maybe the wording people found off=putting, but I think the point is correct. AIs haven’t really started to get creative yet, which shouldn’t be underestimated. Creativity is expanding the matrix of possibilities. In chess, that matrix remains constrained. Sure, there are physical constraints, but an ASI can run circles around us before it has to resort to reversing entropy.
I generally agree, regarding the public at large. I’m speaking mostly from experience of people in the AIS Community speaking with people either working in AI or some related field, and I’ve found many often can get stuck debating these concepts. The general public seems to get more hung up on concepts like consciousness, sentience, etc. (from my experience)
If someone manages to create a powerful AGI, and the only cost for most humans is that it burns their GPUs, this seems like an easy tradeoff for me. It’s not great, but it’s mostly a negligible problem for our species. But I do agree using governance and monitoring is a possible option. I’m normally a hardline libertarian/anarchist, but I’m fine going full Orwellian in this domain.