An independent researcher of ethics, AI safety, and AI impacts. LessWrong: https://www.lesswrong.com/users/roman-leventov. Twitter: https://twitter.com/leventov. E-mail: leventov.ru@gmail.com (the preferred mode of communication).
Roman Leventov
Karma: 128
I’m excited to see you posting this. My views are very closely agreed with yours. I summarised my views a few days ago here.
One of the most important similarities is that we both emphasise the importance of decision-making and supporting it with institutions. This could be seen as “enactivist” view on agent (human, AI, hybrid, team/organisation) cognition.
The biggest difference between our views is that I think the “cognitivist” agenda (i.e., agent internals and algorithms) is as important as the “enactivist” agenda (institutions), whereas you seem to almost disregard the “cognitivist” agenda.
Try to constrain, delay, or obstruct AI, in order to reduce risk, mitigate negative impacts, or give us more time to solve essential issues. This includes, for example, trying to make sure AIs aren’t able to take certain actions (i.e. ensure they are controlled).
I disagree with putting risk-detection/mitigation mechanisms, algorithms, monitorings in that bucket. I think we should just separate between engineering (cf. A plea for solutionism on AI safety) and non-engineering (policy, legislature, treaties, commitments, advocacy) approaches. In particular, the “scheming control” agenda that you link will be concrete engineering practice that should be used in the training of safe AI models in the future, even if we have good institutions, good decision-making algorithms wrapped on top of these AI models, etc. It’s not an “alternative path” just for “non-AI-dominated worlds”. The same applies ftoor monitoring, interpretability, evals, etc. processes. All of these will require very elaborate engineering on their own.
I 100% agree with your reasoning about Frames 1 and 2. I want to discuss the following point in detail because it’s a rare view in EA/LW circles:
It (IMO) wrongly imagines that the risk of coups comes primarily from the personal values of actors within the system, rather than institutional, cultural, or legal factors.
In my post, I also made a similar point: “aligning LLMs with human values” is hardly a part of [the problem of context alignment] at all”. But my framing was in general not very clear, so I’d try to improve it and integrate it with your take here:
Context alignment is a pervasive process that happens (and sometimes needed) on all timescales: evolutionary, developmental, and online (the examples of the latter in humans: understanding, empathy, rapport). The skill of context alignment is extremely important and should be practiced often by all kinds of agents in their interactions (and therefore we should build this skill into AIs), but it’s not something that we should “iron out once and for all”. That would be neither possible (agents’ contexts are constantly diverging from each other), nor desirable: the (partial) misalignment is also important, it’s the source of diversity that enables the evolution[1]. Institutions (norms, legal systems, etc.) are critical for channelling and controlling this misalignment so that it’s optimally productive and doesn’t pose excessive risk (though some risk is unavoidable: that’s the essence of misalignment!).
Flexible yet resilient legal and social structures that can adapt to changing conditions without collapsing
This is interesting. I’ve also discussed this issue as “morphological intelligence of socioeconomies” just a few day ago :)
Good incentives for agents within the system, e.g. the economic value of trade is mostly internalized
Rafael Kaufmann and I have a take on this in our Gaia Network vision. Gaia Network’s term for internalised economic value of trade is subjective value. The unit of subjective accounting is called FER. Trade with FER induces flow that defines the intersubjective value, i.e., the “exchange rates” of “subjective FERs”. See the post for more details.
While sharing some features of the other two frames, the focus is instead on the institutions that foster AI development, rather than micro-features of AIs, such as their values
As I mentioned in the beginning, I think you are too dismissive of the “cognitivist” perspective. We shouldn’t paint all “micro-features of AIs” with the same brush. I agree that value alignment is over-emphasized[2], but other engineering mechanisms and algorithms, such as decision-making algorithms, “scheming control” procedures, context alignment algorithms, as well as architectural features: namely being world-model-based[3] and being amenable to computational proofs[4] are very important and couldn’t be recovered on the institutional/interface/protocol level. We demonstrated in the post about Gaia Network above that for for the “value economy” to work as intended, agents should make decisions based on maximum entropy rather than maximum likelihood estimates[5] and they should share and compose their world models (even if in a privacy-preserving way with zero-knowledge computations).
- ^
Indeed, this observation makes evident that the refrain question “AI should be aligned with whom?” doesn’t and shouldn’t have a satisfactory answer if “alignment” is meant to be “totalising value alignment as often conceptualised on LessWrong”; on the other hand, if “alignment” is meant to be context alignment as a practice, the question becomes as non-sensical (in the general form) as the question “AI should interact with whom?”—well, with someone, depending on the situation, in the way and to the degree appropriate!
- ^
However, still not completely irrelevant, at least for practical reasons: having shared values on the pre-training/hard-coded/verifiable level, as a minimum, reduces transaction costs because the AI agents shouldn’t then painstakingly “eval” each other’s values before doing any business together.
- ^
Both Bengio and LeCun argue for this: see “Scaling in the service of reasoning & model-based ML” (Bengio and Hu, 2023) and “A Path Towards Autonomous Machine Intelligence” (LeCun, 2022).
- ^
See “Provably safe systems: the only path to controllable AGI” (Tegmark and Omohundro, 2023).
- ^
Which is just another way of saying that they should minimise their (expected) free energy in their model updates/inferences and the course of their actions.
- ^
This post has led me to this idea: “Workshop (hackathon, residence program, etc.) about for-profit AI Safety projects?”
2. Historically, the Federal Reserve has failed to subsequently tighten monetary stimulus I think a Keynesian idea that lies at the heart of Federal Reserve philosophy is that during economic recessions you offer monetary stimulus to support virtuous (re-)investment cycles, and during boom years you tighten market access to funds to prevent debt-fuelled excesses.
I would be really interested in reading somebody more qualified than me revising this point after the two years (or, this post more generally). It seems than right now, the economy and the financial system are closer to a third mode than they have been in decades (?): in 2024 in the US, we could see relatively low GDP growth and sustained relatively high interest rate if the Fed decides that lowering the rate prematurely could spike the inflation again.
However, this still pretty mild and benign in the US ,whereas the UK faces a real risk of downright stagflation, especially if the Labour party wins the next elections, which currently seems almost certain.
Another point, could be seen as generalisation of your first point: more money supply than optimal for the economy just makes the entire economy less efficient. It’s entirely analogous to how an organism gets excessive fat and becomes inefficient (and incurs health risks) when over-fed (and I strongly suspect that this is deeper than just a superficial analogy, that there is a general system development law behind this).
More concretely:
Excessive bureaucratisation and creation of ‘bullshit jobs’, i.e., unproductive jobs whose main functions are in organisational politics and labor union relationships. BTW, these bullshit jobs will also mask deeper issues with employment: for example, participation rate (which is different from ’100% - unemployment’ though) is an important economic metric, but it seems to be misled (Goodhearted if you wish): what we are really interested in is productive participation rate.
Low productive participation rate is not only due to bullshit jobs, but also due to excessively many rentiers or early retirees created by too high money supply. Paradoxically, this leads to a decrease in the quality of life: many people have enough money to retire from productive jobs ⇒ shortage in real sector labour: doctors, public transport workers, street cleaners, canalisation maintenance workers, electricians, etc. ⇒ public services start to crumble, and wait times for non-public services (e.g., private doctors) increase, and the prices for both rise ⇒ lower quality of life even for those rentiers and retirees!
Creation of totally inefficient startups that burn money, don’t have a path towards profitability, but still raise a lot of funds. It is possible to overshoot here in another direction, too, but the current situation when there are so many deeply unprofitable unicorns in late venture investment rounds and even long IPO-ed is unique.
Downstream effect of this unhealthy startup environment is https://en.wikipedia.org/wiki/Enshittification, which is at least partially caused or exacerbated by the monetary overhang. Too many startups are funded and are confident that they can raise money for a very long time ⇒ Price war and winner-take-all mentality ⇒ Quasi-monopolies and enshittification. We have seen this with Uber, and we are likely see it on a massive scale with AI startups, which are currently free or massively subsided, but later will come to eat our arm and leg. But note that I don’t say that the over-supply of money is the only cause of this dynamic.
Another incentive for private firms to do R&D is to make their product at least 30% more valuable than the competition to compel customers to switch. Applies to b2b mostly and long-term products and contracts.
And another reason for why governments don’t do more R&D is that they cannot do it effectively, outside of the context of concrete problems and without resources (infrastructure, tools, know-how, data, etc.) available to a concrete firm.
Thanks!
And also: about the “AI race” risk a.k.a. Moloch a.k.a. https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic
Announcement
I think SociaLLM has a good chance of getting OpenAI’s “Research into Agentic AI Systems” grant because it addresses both the challenges of the legibility of AI agent’s behaviour by making the agent’s behaviour more “human-like” thanks to weight sharing and regularisation techniques/inductive biases described the post, as well as automatic monitoring: detection of duplicity or deception in AI agent’s behaviour by comparing agent’s ToMs “in the eyes” of different other interlocutors, building on the work “Collective Intelligence in Human-AI Teams”.
I am looking for co-investigators for this (up to $100k, up to 8 months long) project with hands-on academic or practical experience in DL training (preferably), ML, Bayesian statistics, or NLP. The deadline for the grant application itself is the 20th of January, so I need to find a co-investigator by the 15th of January.
Another requirement for the co-investigator is that they preferably should be in academia, non-profit, or independent at the moment.
I plan to be hands-on during the project in data preparation (cleansing, generation by other LLMs, etc.) and training, too. However, I don’t have any prior experience with DL training, so if I apply for the project alone, this is a significant risk and a likely rejection.
If the project is successful, it could later be extended for further grants or turned into a startup.
If the project is not a good fit for you but you know someone who may be interested, I’d appreciate it a lot if you shared this with them or within your academic network!
Please reach out to me in DMs or at leventov.ru@gmail.com.
How about coordination and multi-scale planning (optimising both for short term and long term) failures? They both have economic value (i.e., economic value is lost when these failures happen), and they are both at least in part due to the selfish, short-term, impulsive motives/desires/”values” of humans.
E.g., I think people would like to buy an AI that manipulated them into following their exercise plan through some tricks, and likewise they would like to “buy” (build) collectively an AI that restricts their selfishness for the median benefit and the benefit of their own children and grandchildren.
I’ve earlier argued against the sentiment that “we need as many technical approaches (‘stabs’) to solve the AI alignment problem”, and therefore, probably, new organisations to pursue these technical agendas, too.
Outreach to more traditional entrepreneurs, attempting to bridge the EA vs traditional entrepreneurship cultural divide.
I second this.
Also worth mentioning https://arxiv.org/abs/2306.06924 - a paper by Critch and Russell of the very similar genre.
(Cross-posted from LW)
Roko would probably call “the most important century” work “building a stable equilibrium to land an AGI/ASI on”.
I broadly agree with you and Roko that this work is important and that it would often make more sense for people to do this kind of work than “narrowly-defined” technical AI safety.
An aspect for why this may be the case that you didn’t mention is money: technical AI safety is probably bottlenecked on funding, but more of the “most important century/stable equilibrium” are more amenable to conventional VC funding, and the funders shouldn’t even be EA/AI x-risk/”most important century”-pilled.
In a comment to Roko’s post, I offered my classification of this “stable equilibrium” systems and work that should be done. Here I will reproduce it, with extra directions that appeared to me later:
Digital trust infrastructure: decentralised identity, secure communication (see Layers 1 and 2 in Trust Over IP Stack), proof-of-humanness, proof of AI (such as, a proof that such and such artifact is created with such and such agent, e.g., provided by OpenAI—watermarking failed, so need new robust solutions with zero-knowledge proofs).
Infrastructure for collective sensemaking and coordination: the infrastructure for communicating beliefs and counterfactuals, making commitments, imposing constraints on agent behaviour, and monitoring the compliance. We at Gaia Consortium are doing this.
Infrastructure and systems for collective epistemics: next-generation social networks (e.g., https://subconscious.network/), media, content authenticity, Jim Rutt’s “info agents” (he advises “three different projects that are working on this”).
Related to the previous item, in particular, to content authenticity: systems for personal data sovereignty (I don’t know any good examples besides Inrupt), dataset verification/authenticity more generally, dataset governance.
The science/ethics of consciousness and suffering mostly solved, and much more effort in biology to understand whom (or whose existence, joy, or non-suffering) the civilisation should value, to better inform the constraints and policy for the economic agents (which is monitored and verified through the infra from item 2.)
Systems for political decision-making and collective ethical deliberation: see Collective Intelligence Project, Policy Synth, simulated deliberative democracy. These types of systems should also be used for governing all of the above layers.
Accelerating enlightenment using AI teachers (Khanmigo, Quantum Leap) and other tools for individual epistemics (Ought) so that the people who participate in governance (the previous item) could do a better job.
The list above covers all the directions mentioned in the post, and there are a few more important ones.
Except that RSPs don’t concern with long-term economic, social, and political implications. The ethos of AGI labs is to assume, for the most part, that these things will sort out themselves, and they only need to check technical and momentary implications, i.e., do “evals”.
The public should push for “long-term evals”, or even mandatory innovation in political and economic systems coupled with the progress in AI models.
The current form of capitalism is simply unprepared for autonomous agents, no amount of RLHF and “evals” will fix this.
You can say exactly the same about Pause AI.
(Cross-posted from LW)
@Nathan Helm-Burger’s comment made me think it’s worthwhile to reiterate here the point that I periodically make:
Direct “technical AI safety” work is not the only way for technical people (who think that governance & politics, outreach, advocacy, and field-building work doesn’t fit them well) to contribute to the larger “project” of “ensuring that the AI transition of the civilisation goes well”.
Now, as powerful LLMs are available, is the golden age to build innovative systems and tools to improve[1]:
Politics: see https://cip.org/, Audrey Tang’s projects
Social systems: innovative LLM/AI-first social networks that solve the social dilemma? (I don’t have a good existing examples of such projects, though)
Psychotherapy, coaching: see Inflection
Economics: see Verses, One Project, the Gaia Consortium
Epistemic infrastructure: see Subconscious Network, Ought, the Cyborgism agenda, Quantum Leap (AI safety edtech)
Authenticity infrastructure: see Optic, proof-of-personhood projects
Cybersec/infosec: see various AI startups for cybersecurity, trustoverip.org
More?
I believe that if such projects are approached with integrity, thoughtful planning, and AI safety considerations at heart rather than with short-term thinking (specifically, not considering how the project will play out if or when AGI is developed and unleashed on the economy and the society) and profit-extraction motives, they could shape to shape the trajectory of the AI transition in a positive way, and the impact may be comparable to some direct technical AI safety/alignment work.
In the context of this post, it’s important that the verticals and projects mentioned above could either be conventionally VC-funded because they could promise direct financial returns to the investors, or could receive philanthropic or government funding that wouldn’t otherwise go to technical AI safety projects. Also, there is a number of projects in these areas that are already well-funded and hiring.
Joining such projects might also be a good fit for software engineers and other IT and management professionals who don’t feel they are smart enough or have the right intellectual predispositions to do good technical research, anyway, even there was enough well-funded “technical AI safety research orgs”. There should be some people who do science and some people who do engineering.
- ^
I didn’t do serious due diligence and impact analysis on any of the projects mentioned. The mentioned projects are just meant to illustrate the respective verticals, and are not endorsements.
The “nature coin” complicates this experiment a lot. Also, it sounds like a source of inherent randomness in policy’s outcomes, i.e., aleatoric uncertainty, which is perhaps rarely or never the case for actual policies and therefore evaluating such policies ethically is unnatural: the brain is not trained to do this. When people discuss and think about the ethics of policies, even the epistemic uncertainty is often assumed away, though it is actually very common that we don’t know whether a potential policy will turn out good or bad.
Due to this, I would say I have a preference for the intervention E because it’s the only one which actually doesn’t depend on the “nature coin”.
I think a lot of people who are aware of AI risk for some time but nevertheless choose to work on some other causes, such as climate change, may implicitly hold this view.
Fertility rate may be important but to me it’s not worth restricting (directly or indirectly) people’s personal choices for.
This is a radical libertarian view that most people don’t share. Is it worth restricting people’s access to hard drugs? Let’s abstract for a moment from the numerous negative secondary effects that come with the fact that hard drugs are illegal, as well as from the crimes committed by drug users: if we can imagine that hard drugs could be just eliminated from Earth completely, with a magic spell, should we do it, or we “shouldn’t restrict people’s choices”? With AI romantic partners, and other forms of tech, we do have a metaphorical magic wand: we could decide whether such products ever get created or not.
A lot of socially regressive ideas have been justified in the name of “raising the fertility rate” – for example, the rhetoric that gay acceptance would lead to fewer babies (as if gay people can simply “choose to be straight” and have babies the straight way).
The example that you give doesn’t work as evidence for your argument at all, due to the direct disanalogy: the “young man” from the “mainline story” which I outlined could want to have kids in the future or even wants to have kids already when he starts his experiment with the AI relationship, but his experience with the AI partner will prevent him from realising this desire and value over his future life.
I think it’s better to encourage people who are already interested in having kids to do so, through financial and other incentives.
Technology, products, and systems are not value-neutral. We are so afraid of consciously shaping our own values that we are happy to offload this to the blind free market whose objective is not to shape values that reflectively endorse the most.
Hello Agustín, thanks for engaging with our writings and sharing your feedback.
Regarding the ambitiousness, low chances of overall success, and low chances of uptake by human developers and decision-makers (I emphasize “human” because if some tireless near-AGI or AGI comes along it could change the cost of building agents for participation in the Gaia Network dramatically), we are in complete agreement.
But notice Gaia Network could be seen as a much-simplified (from the perspectives of mathematics and Machine Learning) version of Davidad’s OAA, as we framed it in the first post. Also, Gaia Network tries to leverage (at least approximately and to some degree) the existing (political) institutions and economic incentives. In contrast, it’s very unclear to me how the political economy in the “OAA world” could look like, and what is even a remotely plausible plan for switching from the incumbent political economy of the civilisation to OAA, or “plugging” OAA “on top” of the incumbent political economy (and hasn’t been discussed publicly anywhere, to the best of our knowledge). We also discussed this in the first post. Also, notice that due to its extreme ambitiousness, Davidad doesn’t count on humans implementing OAA with their bare hands, it’s a deal-breaker if there isn’t an AI that can automate 99%+ of technical work needed to convert the current science into Infra-Bayesian language.[1] And yes, the same applies to Gaia Network: it’s not feasible without massive assistance from AI tools that can do most of the heavy lifting. But if anything, this reliance on AI is less extreme in the case of Gaia Network than in the case of OAA.
The above makes me think that you should therefore be even more skeptical of OAA’s chances of success than you are about Gaia’s chances. Is this correct? If not, what do you disagree about in the reasoning above, or what elements of OAA make you think it’s more likely to succeed?
Adoption
The “cold start” problem is huge for any system that counts on network effect, and Gaia Network is no exception. But this also means that the cost of convincing most decision-makers (businesses, scientists, etc.) to use the system is far smaller than the cost of convincing the first few, multiplied by the total number of agents. We have also proposed how various early adopters could get value out of the “model-based and free energy-minimising way” of doing decision-making (we don’t need the adoption of Gaia Network right off the bat, more on this below) very soon, in absolutely concrete terms (monetary and real-world risk mitigation) in this thread.
In fact, we think that if there are sufficiently many AI agents and decision intelligence systems that are model-based, i.e., use some kinds of executable state-space (“world”) models to do simulations, hypothesise counterfactually about different courses of actions and external conditions (sometimes in collaboration with other agents, i.e., planning together), and deploy regularisation techniques (from Monte Carlo aggregation of simulation results to amortized adversarial methods suggested by Bengio on slide 47 here) to permit compositional reasoning about risk and uncertaintly that scales beyond the boundary of a single agent, the benefits of collaborative inference of the most accurate and well-regularised models will be so huge that something like Gaia Network will emerge pretty much “by default” because a lot of scientists and industry players will work in parallel to build some versions and local patches of it.
Blockchains, crypto, DeFi, DAOs
I understand why the default prior when hearing anything about crypto, DeFi, and DAOs now is that people who propose something like this are either fantaseurs, or cranks, or, worse, scammers. That’s unfortunate to everyone who just wants to use the technical advances that happen to be loosely associated with this field, which now includes almost anything that has to do with cryptography, identity, digital claims, and zero-knowledge computation.
Generally speaking, zero-knowledge (multi-party) computation is the only solution to make some proofs (of contribution, of impact, of lack of deceit, etc.) without compromising privacy (e.g., proprietary models, know-how, personal data). The ways to deal with this dilemma “in the real world” today inevitably come down to some kind of surveillance which many people become very uneasy about. For example, consider the present discussion of data center audits and compute governance. It’s fine with me and most other people except for e/accs, for now, but what about the time when the cost of training powerful/dangerous models will drop so much that anyone can buy a chip to train the next rogue AI for 1000$? How does compute governance look in this world?
Governance
I don’t think AI Safety governance is that special among other kinds of governance. But more generally on this point, of course, governance is important, and Gaia Network doesn’t claim to “solve” it; rather, it plans to rely on some solutions developed by other projects (see numerous examples in CIP ecosystem map, OpenAI’s “Democratic Inputs to AI” grantees, etc.).
We just mention in passing incorporating preferences of system’s stakeholders into Gaia agents’ subjective value calculations (i.e., building reward models for these agents/entities, if you wish), but there is a lot to be done there: how the preferences of the stakeholders are aggregated and weighted, who can claim to be a stakeholders of this or that system in the first place, etc. Likewise, on the general Gaia diagram in the post, there is a small arrow from “Humans and collectives” box to “Decision Engines” box labelled “Review and oversight”, and, as you can imagine, there is a lot to be going on there as well.
IDK, convinced that this is a safe approach? Being coerced (including economically, not necessary by force) by the broader consensus of using such Gaia Network-like systems? This is a collective action problem. This question could be addressed to any AI Safety agenda and the answer would be the same.
Moloch
I wouldn’t say that we “claim to be able to slay Moloch”. Rafael is more bold in his claims and phrasing than me, but I think even he wouldn’t say that. I would say that the project looks very likely to help to counteract Molochian pressures. But this seems to me almost a self-evident statement, given the nature of the proposal.
Compare with Collective Intelligence Project. It has started with the mission to “fix governance” (and pretty much “help to counteract Moloch” in the domain of political economy, too, they barely didn’t use this concept, or maybe they even did, I don’t want to check it now), and now they “pivoted” to AI safety and achieved great legibility on this path: e.g., they partner with OpenAI, apparently, on more than one project now. Does this mean that CIP is a “solution looking for a problem”? No, it’s just the kind of project that naturally lends to helps both with Moloch and AI safety. I’d say the same could be said of Gaia Network (if it is realised in some forms) and this lies pretty much in plain sight.
Furthermore, this shouldn’t be surprising in general, because AI transition of the economy is evidently an accelerator and a risk factor in the Moloch model, and therefore these domains (Moloch and AI safety) almost merge in my overall model of risk. Cf. Scott Aaronson’s reasoning that AI will inevitably be in the causal structure of any outcome of this century so “P(doom from AI)” is not well defined; I agree with him and only think about “P(doom)” without specification what this doom “comes from”. Again, note that it seems that most narratives about possible good outcomes (take OpenAI’s superalignment plan, Conjecture’s CoEm agenda, OAA, Gaia Network) all rely on developing very advanced (if not superhuman) AI along the way.
Notice here again: you mention that most scientists don’t know about Bayesian methods, but perhaps at least two orders of magnitude still fewer scientists have even heard of Infra-Bayesianism, let alone being convinced it’s a sound and a necessary methodology for doing science. Whereas for Bayesianism, from my perspective, it seems there is quite a broad consensus of its soundness: there are numerous pieces and even books written about how P-values are a bullshit value of doing science and that scientists should take up (Bayesian) causal inference instead.
There are a few notable voices that dismiss Bayesian inference, for example, David Deutsch, but then no less notable voices, such as Scott Aaronson and Sean Carroll (of the people that I’ve heard, anyway), that dismiss Deutsch’s dismissal in turn.