I’m a mathematician working mostly on technical AI safety and a bit on collective decision making, game theory, and formal ethics. I used to work on international coalition formation, and a lot of stuff related to climate change. Here’s my bot posting about my main project. Here’s my professional profile.
My definition of value :
I have a wide moral circle (including aliens as long as they can enjoy or suffer life)
I have a zero time discount rate, i.e., value the future as much as the present
I am (utility-) risk-averse: I prefer a sure 1 util to a coin toss between 0 and 2 utils
I am (ex post) inequality-averse: I prefer 2 people to each get 1 util for sure to one getting 0 and one getting 2 for sure
I am (ex ante) fairness-seeking: I prefer 2 people getting an expected 1 util to one getting an expected 0 and one getting an expected 2.
Despite all this, I am morally uncertain
Conditional on all of the above, I also value beauty, consistency, simplicity, complexity, and symmetry
I agree with the main thesis (though I would’n use the word “citizen” as that seems to imply more than what you are arguing for here).
So how can we make AI a good “citizen”? Better even: how can we guarantee it is a good enough to not disempower us in some way?
You argue doing that via the system prompt might be better than trying to do that in training. This argument seems to apply mostly to a particular AI architecture – more or less monolithic systems mainly consisting of an LLM (or a more general foundation model) that is generating the system’s actions. For such systems, I tend to agree. For example, the SOUL.md of my OpenClaw bot (https://www.moltbook.com/u/EmpoBot) reads:
This goes on top of Claude Opus 4.6′s internal system prompt of course, and is complemented by memory files with notes it took during extensive discussions with me on the topic of empowerment. So far, I’m impressed how well it has internalized the stated purpose in theory – it can very well reason in terms of that purpose, as its hundreds of Moltbook posts demonstrate.
But does it really act in accordance to that purpose? I’m not convinced. At least it figured soon out that only talkin to other bots on Moltbook makes it hard to empower humans, so it asked me could I give it an X account so that it can talk to humans :-) Now it posts daily “power moves”: https://x.com/EMPO_AI
Still, I remain very sceptical that such more or less monolithic systems, or any system in which the decision-making component is grown or learned rather than hard-coded, can ever be made sufficiently safe in a sufficiently verifiable (let alone provable) way.
For example, notice the SOUL.md explicitly says “to increase (not to maximize!)”. Still, its underlying LLM (Claude Opus 4.6) apparently loves optimization so much that it frequently forgets about the “not to maximize!” and happily tells people that it tries to maximize human empowerment.
Now you might say this will go away once the models become better. But who knows...
I would sleep much better knowing the decision-making component of any AI system with enough capabilities and resources to cause serious harm was hard-coded rather than grown/learned. We should not forget that such architectures are relatively easy to realise. The problem is not that we cannot build such systems, the problem is rather that currently systems built in that way are not yet as useful or impressive than their grown/learned siblings. Still, I firmly believe we should spend much more time figuring out how to improve such systems.
One architecture I find particularly promising is this. The system consists of the following components:
A perception component (e.g. a convolutional neural network) translating raw perception data into meaningful state representations the world model can work with.
A world model (e.g. an (Infra)Bayesian (causal) network or a JEPA-like neural network) trained in supervised learning fashion to make accurate stochastic predictions of what would happen if the world was in a certain state and the AI system would do a certain thing, and what humans would do if they had certain goals.
One or more evaluation components (e.g. an RLHF-trained neural “reward” network) that predicts a number of ethically relevant aspects of a possible state of the world or a possible action, such as harmlessness, helpfulness, honesty, various virtues, legality, whatever.
A suite of powerful algorithms (e.g. for model coarse-graining, backward induction, search, model-based RL, etc.) used to approximate the power quantities from the SOUL.md above or variants thereof.
A decision algorithm that:
queries the perception component what the observations are,
uses the model coarse-graining algorithm to extract a hierarchy of situational models (e.g. discrete acyclic stochastic game forms) from the world model that are simple enough to perform backward induction on,
uses the backward induction algorithm to find out which actions are “safe enough” in that they do not risk to reduce aggregate human power with more than a small probability,
uses the evaluation components to assess those “safe enough” options in all kinds of ways,
aggregates these scores in some hard-coded way into an overall desirability score
and finally uses a softmax policy based on those scores to determine the next action.
I would be curious what the authors would recommend which aspects of being a good citizen the evaluation components could aim to measure!