An independent researcher of ethics, AI safety, and AI impacts. LessWrong: https://www.lesswrong.com/users/roman-leventov. Twitter: https://twitter.com/leventov. E-mail: leventov.ru@gmail.com (the preferred mode of communication).
Roman Leventov
Karma: 128
AI romantic partners will harm society if they go unregulated
[Question] Has private AGI research made independent safety research ineffective already? What should we do about this?
[Question] Workshop (hackathon, residence program, etc.) about for-profit AI Safety projects?
You can say exactly the same about Pause AI.
However, talented individuals who have invested in upskilling themselves to go do AIS research (e.g. SERI MATS graduates) are largely unable to secure research positions.
It would be interesting to see the actual numbers, I think Ryan Kidd should have them.
Proposal for improving the global online discourse through personalised comment ordering on all websites
It’s hard to imagine more general and capability-demanding activity as doing good (superhuman!) science in such an absurdly cross-disciplinary field as AI safety (and among the disciplines that are involved there are those that are notoriously not very scientific yet: psychology, sociology, economics, the studies of consciousness, ethics, etc.). So if there is an AI that can do that but still is not counted as AGI, I don’t know what the heck ‘AGI’ should even refer to. Compare with chess, which is a very narrow problem which can be formally defined and doesn’t require AI to operate with any science (and world models) whatsoever.
AI safety is a field concerned with preventing negative outcomes from AI systems and ensuring that AI is beneficial to humanity.
This is a bad definition of “AI safety” as a field, which muddles the water somewhat. I would say that AI safety is a particular R&D branch (plus we can add here meta and proxy activities for this R&D field, such as AI safety fieldbuilding, education, outreach and marketing among students, grantmaking, and platform development such as what apartresearch.com are doing), of the gamut of activity that strives to “prevent the negative result of civilisational AI transition”.
There are also other sorts of activity that strive for that more or less directly, some of which are also R&D (such as governance R&D (cip.org), R&D in cryptography, infosec, and internet decentralisation (trustoverip.org)), and others are not R&D: good old activism and outreach to the general public (StopAI, PauseAI), good old governance (policy development, UK foundational model task force), and various “mitigation” or “differential development” projects and startups, such as Optic, Digital Gaia, Ought, social innovations (I don’t know about any good examples as of yet, though), innovations in education and psychological training of people (I don’t know about any good examples as of yet). See more details and ideas in this comment.
It’s misleading to call this whole gamut of activities “AI safety”. It’s maybe “AI risk mitigation”. By the way, 80000 hours, despite properly calling “Preventing an AI-related catastrophe”, also suggest that the only two ways to apply one’s efforts to this cause is “technical AI safety research” and “governance research and implementation”, which is wrong, as I demonstrated above.
Somebody may ask, isn’t technical AI safety research more direct and more effective way to tackle this cause area? I suspect that it might not be the case for people who don’t work at AGI labs. That is, I suspect that independent or academic AI safety research might be inefficient enough (at least for most people attempting it) that it would be more effective to apply themselves to various other activities, and “mitigation” or “differential development” projects of the likes that are described above. (I will publish a post that details reasoning behind this suspicion later, but for now this comment has the beginning of it.)
Note: this comment is cross-posted on LessWrong.
Classification of AI safety work
Here I proposed a systematic framework for classifying AI safety work. This is a matrix, where one dimension is the system level:
A monolithic AI system, e.g., a conversational LLM
AGI lab (= the system that designs, manufactures, operates, and evolves monolithic AI systems and systems of AIs)
A cyborg, human + AI(s)
A system of AIs with emergent qualities (e.g., https://numer.ai/, but in the future, we may see more systems like this, operating on a larger scope, up to fully automatic AI economy; or a swarm of CoEms automating science)
A human+AI group, community, or society (scale-free consideration, supports arbitrary fractal nestedness): collective intelligence, e.g., The Collective Intelligence Project
The whole civilisation, e.g., Open Agency Architecture, or the Gaia network
Another dimension is the “time” of consideration:
Design time: research into how the corresponding system should be designed (engineered, organised): considering its functional (“capability”, quality of decisions) properties, adversarial robustness (= misuse safety, memetic virus security), and security. AGI labs: org design and charter.
Manufacturing and deployment time: research into how to create the desired designs of systems successfully and safely:
AI training and monitoring of training runs.
Offline alignment of AIs during (or after) training.
AI strategy (= research into how to transition into the desirable civilisational state = design).
Designing upskilling and educational programs for people to become cyborgs is also here (= designing efficient procedures for manufacturing cyborgs out of people and AIs).
Operations time: ongoing (online) alignment of systems on all levels to each other, ongoing monitoring, inspection, anomaly detection, and governance.
Evolutionary time: research into how the (evolutionary lineages of) systems at the given level evolve long-term:
How the human psyche evolves when it is in a cyborg
How humans will evolve over generations as cyborgs
How AI safety labs evolve into AGI capability labs :/
How groups, communities, and society evolve.
Designing feedback systems that don’t let systems “drift” into undesired state over evolutionary time.
Considering system property: property of flexibility of values (i.e., the property opposite of value lock-in, Riedel (2021)).
IMO, it (sometimes) makes sense to think about this separately from alignment per se. Systems could be perfectly aligned with each other but drift into undesirable states and not even notice this if they don’t have proper feedback loops and procedures for reflection.
There would be 6*4 = 24 slots in this matrix, and almost all of them have something interesting to research and design, and none of them is “too early” to consider.
Richard’s directions within the framework
Scalable oversight: (monolithic) AI system * manufacturing time
Mechanistic interpretability: (monolithic) AI system * manufacturing time, also design time (e.g., in the context of the research agenda of weaving together theories of cognition and cognitive development, ML, deep learning, and interpretability through the abstraction-grounding stack, interpretability plays the role of empirical/experimental science work)
Alignment theory: Richard phrases it vaguely, but referencing primarily MIRI-style work reveals that he means primarily “(monolithic) AI system * design, manufacturing, and operations time”.
Evaluations, unrestricted adversarial training: (monolithic) AI system * manufacturing, operations time
Threat modeling: system of AIs (rarely), human + AI group, whole civilisation * deployment time, operations time, evolutionary time
Governance research, policy research: human + AI group, whole civilisation * mostly design and operations time.
Takeaways
To me, it seems almost certain that many current governance institutions and democratic systems will not survive the AI transition of civilisation. Bengio recently hinted at the same conclusion.
Human+AI group design (scale-free: small group, org, society) and the civilisational intelligence design must be modernised.
Richard mostly classifies this as “governance research”, which has a connotation that this is a sort of “literary” work and not science, with which I disagree. There is a ton of cross-disciplinary hard science to be done about group intelligence and civilisational intelligence design: game theory, control theory, resilience theory, linguistics, political economy (rebuild as hard science, of course, on the basis of resource theory, bounded rationality, economic game theory, etc.), cooperative reinforcement learning, etc.
I feel that the design of group intelligence and civilisational intelligence is an under-appreciated area by the AI safety community. Some people do this (Eric Drexler, davidad, the cip.org team, ai.objectives.institute, the Digital Gaia team, and the SingularityNET team, although the latter are less concerned about alignment), but I feel that far more work is needed in this area.
There is also a place for “literary”, strategic research, but I think it should mostly concern deployment time of group and civilisational intelligence designs, i.e., the questions of transition from the current governance systems to the next-generation, computation and AI-assisted systems.
Also, operations and evolutionary time concerns of everything (AI systems, systems of AIs, human+AI groups, civilisation) seem to be under-appreciated and under-researched: alignment is not a “problem to solve”, but an ongoing, manufacturing-time and operations-time process.
Assessment of AI safety agendas: think about the downside risk
AGI will be made of heterogeneous components, Transformer and Selective SSM blocks will be among them
[...] we are impressed by [...] ‘Eliciting Latent Knowledge’ [that] provided conceptual clarity to a previously confused concept
To me, it seems that ELK is (was) attention-captivating (among the AI safety community) but doesn’t assume a solid basis: logic and theories of cognition and language, and therefore is actually confusing, which prompted at least several clarification and interpretation atttempts (1, 2, 3). I’d argue that most people leave original ELK writings more confused than they were before. So, I’d classify ELK as a mind-teaser and maybe problem-statement (maybe useful than distracting, or maybe more distracting than useful; it’s hard to judge as of now), but definitely not as great “conceptual clarification” work.
If OpenAI still had a moral compass, and were still among the good guys, they would pause AGI (and ASI) capabilities research until they have achieved a viable, scalable, robust set of alignment methods that have the full support and confidence of AI researchers, AI safety experts, regulators, and the general public.
I disagree with multiple things in this sentence. First, you take a deontology stance, whereas OpenAI clearly acts within consequentialist stance, assuming that if they don’t create ‘safe’ AGI, reckless open-source hackers will (upon the continuing exponential decrease in the cost of effective training compute, and/or the next breakthrough in DNN architecture or training that will make it much more efficient, and/or will enable effective online training). Second, I largely agree with OpenAI as well as Anthropic that iteration is important for building an alignment solution. One probably cannot design a robust, safe AI without empirical iteration, including with increasing capabilities.
I agree with your assessment of the strategy they are taking probably will fail, but mainly because I think we have inadequate human intelligence, human psychology, and coordination mechanisms to execute it. That is, I would support Yudkowsky’s proposal: halt all AGI R&D, develop narrow AI and tech for improving the human genome, make humans much smarter (von Neumann-level of intelligence should be just the average) and have much more peaceful psychology, like bonobos, reform coordination and collective decision-making, and only then re-visit the AGI project roughly with the same methodology as OpenAI proposes, albeit with more diversified methodology: I agree with your criticism that OpenAI is too narrowly focused on some sort of computationalism, to the detriment of the perspectives from psychology, neuroscience, biology, etc. BTW, it seems that DeepMind is more diversified in this regard.
The things that the proposed startup is going to do seems to overlap in various ways with MATS, AI Safety Camp, Orthogonal (https://www.lesswrong.com/posts/b2xTk6BLJqJHd3ExE/orthogonal-a-new-agent-foundations-alignment-organization), European Network for AI Safety (ENAIS, https://forum.effectivealtruism.org/posts/92TAmcppCL7t54Ajn/announcing-the-european-network-for-ai-safety-enais), Nonlinear.org, and LTFF (if you plan to ‘hire’ researchers and pay them salary, i.e., effectively fund them, you basically plan to increase the total fundraising for AI safety, which is currently the LTFF’s role).
Detailing similarities, differences, and partnerships with these projects and orgs would be useful
Worrisome misunderstanding of the core issues with AI transition
Open Agency model can solve the AI regulation dilemma
Gaia Network: a practical, incremental pathway to Open Agency Architecture
SociaLLM: proposal for a language model design for personalised apps, social science, and AI safety research
The “nature coin” complicates this experiment a lot. Also, it sounds like a source of inherent randomness in policy’s outcomes, i.e., aleatoric uncertainty, which is perhaps rarely or never the case for actual policies and therefore evaluating such policies ethically is unnatural: the brain is not trained to do this. When people discuss and think about the ethics of policies, even the epistemic uncertainty is often assumed away, though it is actually very common that we don’t know whether a potential policy will turn out good or bad.
Due to this, I would say I have a preference for the intervention E because it’s the only one which actually doesn’t depend on the “nature coin”.
Except that RSPs don’t concern with long-term economic, social, and political implications. The ethos of AGI labs is to assume, for the most part, that these things will sort out themselves, and they only need to check technical and momentary implications, i.e., do “evals”.
The public should push for “long-term evals”, or even mandatory innovation in political and economic systems coupled with the progress in AI models.
The current form of capitalism is simply unprepared for autonomous agents, no amount of RLHF and “evals” will fix this.