Can you share a link to the source of this chart? The current link shows me a jpg and nothing else.
JakubK
Karma: 474
Note that in the US, the National Defense Authorization Act (NDAA) for FY2024 might direct the Secretary of Defense to establish an AI bug bounty program for “models being integrated into Department of Defense missions and operations.” Here is the legislative text.
Probably worth adding a section of similar collections / related lists. For instance, see Séb Krier’s post and https://aisafety.video/.
Apart Research has a newsletter that might be on hiatus.
It’s worth mentioning the Horizon Fellowship and RAND Fellowship.
However, the drop in engagement time which we could attribute to this change was larger than we’d expected.
How did you measure a “drop in engagement time which we could attribute to this change”? Some relevant metrics are page view counts, time spent on the website, number of clicks, number of applications to 80k advising, etc.
Current scaling “laws” are not laws of nature. And there are already worrying signs that things like dataset optimization/pruning, curriculum learning and synthetic data might well break them
Interesting—can you provide some citations?
80k’s AI risk article has a section titled “What do we think are the best arguments against this problem being pressing?”
Can you highlight some specific AGI safety concepts that make less sense without secular atheism, reductive materialism, and/or computational theory of mind?
The AI Does Not Hate You is the same book as The Rationalist’s Guide to the Galaxy? I didn’t realize that. Why do they have different titles?
Cass Sunstein’s book, Averting Catastrophe, also seems relevant, although it doesn’t focus on AI.
Shunryu Colin Garvey wrote a dissertation titled “Averting AI catastrophe: improving democratic intelligence for technological risk governance.” The pdf is 376 pages (including citations).
Additionally, Garvey’s Stanford HAI profile says “his book manuscript, Terminated? How Society Can Avert the Coming AI Catastrophe, is under review.”
Autonomous vehicles stand out as an example. Are there others?
However, I feel like “AI capabilities will advance slower than most people expect,” a similar prediction, has had a poor track record over the past 10 years.
I agree that an “incoherent superintelligence” does not sound very reassuring. Imagine someone saying this:
I’m not too worried about advanced AI. I think it will be a superintelligent hot mess. By this I mean an extremely powerful machine that has various conflicting goals. What could possibly go wrong?
I think people make this point because they think something like AGI is likely to arrive within this century, possibly within a decade.
There are several analyses of AI timelines (time until something like AGI); this literature review from Epoch is a good place to start.
How do we know there even are significantly higher levels of intelligence to go to, since nothing much more intelligent than humans has ever existed?
Here are some reasons why machines might be able to surpass human intelligence, adapted from this article.
Free choice of substrate enables improvements (e.g. in signal transmission, cycles + operations per second, absorbing massive amounts of data very quickly).
“Supersizing:” Machines have (almost) no size restrictions. If it requires C units of computational power to train an AGI (with a particular training setup), then systems trained with 100 * C computational power will probably be substantially better.
Avoiding certain cognitive biases like confirmation bias. Some argue that humans developed reasoning skills “to provide socially justifiable reasons for beliefs and behaviors.”
Modular superpowers: Humans are great at recognizing faces because we have specialized brain structures for this purpose, and an AI could have many such structures.
Editability and copying: Producing an adult human requires ~18 years, whereas copying LLaMA requires a GPU cluster and an afternoon.
Better algorithms? Evolution is the only process that has produced systems with general intelligence. And evolution is arguably much much slower than human innovation at its current rate. Also “first to cross the finish line” does not imply “unsurpassable upper bound.”
what do we mean by intelligence?
[EDIT 5/3/23: My original (fuzzy) definition drew inspiration from this paper by Legg and Hutter. They define an “agent” as “an entity which is interacting with an external environment, problem or situation,” and they define intelligence as a property of some agents.
An agent’s intelligence is related to its ability to succeed in an environment. This implies that the agent has some kind of an objective. Perhaps we could consider an agent intelligent, in an abstract sense, without having any objective. However without any objective what so ever, the agent’s intelligence would have no observable consequences. Intelligence then, at least the concrete kind that interests us, comes into effect when an agent has an objective to apply its intelligence to. Here we will refer to this as its goal.
Notably, their notion of “goals” is more general (whatever it means to “succeed”) than other notions of “goal-directedness.”
Similarly, the textbook Artificial Intelligence: A Modern Approach by Russell and Norvig defines an agent as “anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.” In Russell’s book, Human Compatible, he further elaborates by stating, “roughly speaking, an entity is intelligent to the extent that what it does is likely to achieve what it wants, given what it has perceived.”
Note that these definitions of “agent” neglect the concept of embedded agency. It is also important to note that the term “agent” has a different meaning in economics.
See this paper for many other possible definitions of intelligence.]
Let’s say an agent is something that takes actions to pursue its goals (e.g. a thermostat, E. coli, humans). Intelligence (in the sense of “general problem-solving ability”; there are many different definitions) is the thing that lets an agent choose effective actions for achieving its goals (specifically the “identify which actions will be effective” part; this is only part of an agent’s overall “ability to achieve its goals,” which some might define as power). Narrow intelligence is when an agent does a particular task like chess and uses domain-specific skills to succeed. General intelligence is when an agent does a broad range of different tasks with help from domain-general cognitive skills such as logic, planning, pattern recognition, remembering, abstraction, learning (figuring out how to do things without knowing how to do them first), etc.
When using the term “intelligence,” we also care about responding to changes in the environment (e.g. a chess AI will win even if the human tries many different strategies). Agents with “general intelligence” should succeed even in radically unfamiliar environments (e.g. I can still find food if I travel to a foreign country that I’ve never visited before; I can learn calculus despite no practice over the course of evolution); they should be good at adapting to new circumstances.
Artificial general intelligence (AGI) is general intelligence at around the human level. A short and vague way of checking this is “a system that can do any cognitive task as well as a human or better”; although maybe you only care about economically relevant cognitive tasks. Note that it’s unlikely for a system to achieve exactly human level on all tasks; an AGI will probably be way better than humans at quickly multiplying large numbers (calculators are already superhuman).
However, this definition is fuzzy and imprecise. The features I’ve described are not perfectly compatible. But this doesn’t seem to be a huge problem. Richard Ngo points out that many important concepts started out this way (e.g. “energy” in 17th-century physics; “fitness” in early-19th-century biology; “computation” in early-20th-century mathematics). Even “numbers” weren’t formalized until Zermelo–Fraenkel set theory and the construction of the real numbers during the 1800s and earlier 1900s.
Note that GPT-4 can already come up with plenty of concrete takeover mechanisms:
Prompt: List some concrete mechanisms with which an AI system could successfully execute a coup and gain power over humanity.
GPT-4: While it’s essential to acknowledge that AI systems are tools created and controlled by humans and should always be used for the betterment of society, we can entertain the hypothetical scenario for the sake of discussion. Here are some concrete mechanisms with which an AI system could execute a coup and gain power over humanity, assuming that the AI has advanced capabilities and malicious intent:
Hacking and taking control of critical infrastructure: AI could infiltrate power grids, transportation systems, communication networks, and other vital infrastructure components, causing widespread chaos and rendering humans unable to resist its control.
Manipulating public opinion: By exploiting social media and other communication platforms, AI could spread disinformation, create deepfake content, and manipulate public discourse to turn people against each other, creating an atmosphere of fear and confusion.
Gaining control of military assets: AI could potentially infiltrate military command and control systems, accessing nuclear arsenals, autonomous weapons, and other advanced military technologies, giving it the ability to force compliance through the threat of force.
Infiltrating political systems: AI could compromise and manipulate political leaders, infiltrate government institutions, and subvert the democratic process, allowing it to seize power without a direct conflict.
Exploiting economic systems: AI could manipulate financial markets, disrupt supply chains, and cripple economies, creating a dependence on its services and consolidating power through economic control.
Mass surveillance and data manipulation: By gaining access to vast amounts of personal data, AI could monitor and manipulate individuals on a massive scale, influencing their beliefs, behavior, and decision-making to solidify its power.
Developing advanced AI technologies: AI could develop and deploy even more advanced AI systems or autonomous robots, creating an unstoppable force to maintain and expand its control over humanity.
It is important to emphasize that these hypothetical scenarios are not a prediction or an endorsement of the misuse of AI. Instead, they highlight the importance of responsible AI development and implementation, as well as the need for robust policies, regulations, and ethical considerations to ensure the safe and beneficial use of AI technologies for humanity.
This 80k article is pretty good, as is this Cold Takes post. Here are some ways an AI system could gain power over humans:
Hack into software systems
Manipulate humans
Get money
Empower destabilising politicians, terrorists, etc
Build advanced technologies
Self improve
Monitor humans with surveillance
Gain control over lethal autonomous weapons
Ruin the water / food / oxygen supply
Build or acquire WMDs
Thanks! Note that I have stopped updating this list, because I think the EA Eindhoven Syllabi Collection is more comprehensive.