Michele Campolo

Karma: 143

Lifelong recursive self-improver, on his way to exploding really intelligently :D

More seriously: my posts are mostly about AI alignment, with an eye towards moral progress. I have a bachelor’s degree in mathematics, I did research at CEEALAR for four years, and now I do research independently.

Imagine it’s the year 1500, but AI technology is available. You want to make an AI that can tell you that witch hunts are a terrible idea and can convincingly explain why, despite the fact that many people around you seem to think the exact opposite. How do you do it?

The problem above, and the fact that I’d like to avoid producing AI that can be used for bad purposes, is what motivates my research. I think I’ve made progress towards a solution, in both theoretical and practical terms. Regarding why this matters, these two short posts are a good starting point. If you are looking for something more technical, consider setting some time aside to read these two.

Feel free to reach out!

You can support my research through Patreon here.

Michele Campolo 26 Aug 2025 13:46 UTC
1 point
0 ∶ 0
in reply to: SummaryBot’s comment on: One more reason for AI capable of independent moral reasoning: alignment itself and cause prioritisation
I think this summary is ok

Michele Campolo 26 Aug 2025 13:42 UTC
1 point
0 ∶ 0
in reply to: SummaryBot’s comment on: With enough knowledge, any conscious agent acts morally
This is an ok summary

Michele Campolo 22 Jul 2025 13:46 UTC
1 point
0 ∶ 0
in reply to: Tristan Katz’s comment on: Criticism of the main framework in AI alignment
In short, I am not hoping for a specific outcome, and I can’t take into account every single scenario. If someone starts giving more credit to research on moral reasoning in AI after reading this, that’s already enough, considering that the topic doesn’t seem to be popular within AI alignment, and it was even more niche at the time I wrote this post.

Michele Campolo 21 Jul 2025 22:07 UTC
1 point
0 ∶ 0
in reply to: Tristan Katz’s comment on: Criticism of the main framework in AI alignment
I hadn’t considered the narrative you bring up here when I wrote the post, that is interesting. As you write, it relies on the assumption that
once someone ‘wins’ the AGI/ASI race, they will be able to use that AI to control or prevent the development of other potentially dangerous AIs
Here we are entering the realm of forecasting stuff about world politics — stuff I am definitely not an expert on. As far as I know, the probability of that scenario could be extremely low. I can also think of alternative scenarios that don’t seem obviously absurd, so I doubt that the probability is extremely high, but it’s hard for me to say much more than that. Anyway, as you said, AI moral reasoning might be valuable in that scenario as well.
but I’m not convinced that it’s valuable in order to prevent malevolent actors using AI.
That’s a bit too much, I don’t think I claimed that moral reasoning in AI can directly prevent that. It seems that in order to prevent malevolent actors from using AI for bad purposes we would have to either stop AI research completely, because it is not only alignment research that works on the control problem but also standard AI research; or ensure that bad actors never get access to powerful and controllable AI, which also seems hard to do and not something AI moral reasoning can help with.
The weaker claim I made in the post is that research on moral reasoning in AI is less likely to help malevolent actors use AI for bad purposes (and/or help them to a lesser degree) wrt research that aims to make AI controllable.

Michele Campolo 21 Jul 2025 20:48 UTC
1 point
0 ∶ 0
in reply to: Tristan Katz’s comment on: Free agents
But I am equally concerned that this might be an easy way to misalign agents.
I understand your concern, but I think it’s hard to evaluate both whether this is true (because no one has made that kind of experiments yet) and how much of a problem it is: the alternative is other alignment methods, which have different pros and cons, so I guess that the discussion could get very long.
If we let them develop their own moral thinking, there is a high chance that they will develop in different ways to us, and come to very different (and not necessarily better) moral conclusions!
I disagree with this, in particular regarding the high chance. This intuition seems to be based on the belief that morality strongly depends on what you call our starting instinctual values, which are given by evolution; but this belief is questionable. Below I’m quoting the section Moral thinking in Homo sapiens:
In Symbolic Thought and the Evolution of Human Morality [14], Tse analyses this difference extensively and argues that “morality is rooted in both our capacities to symbolize and to generalize to a level of categorical abstraction.” I find the article compelling, and Tse’s thesis is supported also by work in moral psychology — see for example Moral Judgement as Categorization (MJAC) by McHugh et al. [12] — but here I’d like to point out a specific chain of abstract thoughts related to morality.
As we learn more about the world, we also notice patterns about our own behaviour. We form beliefs like “I did this because of that”. Though not all of them are correct, we nonetheless realise that our actions can be steered in different directions, towards different goals, not necessarily about what satisfies our evolutionary drives. At that point, it comes natural to ask questions such as “In what directions? Which goals? Is any goal more important than others? Is anything worth doing at all?”
Asking these questions is, I think, what kickstarts moral and ethical thinking.
Let’s also give an example, so that we don’t just think in theoretical terms (and also because Tse’s article is quite long). Today, some people see wild animal suffering as a problem. To put it simply, we humans care about animals such as snakes and insects. But these animals actually give us aversive instinctive gut reactions, so in this case our moral beliefs go directly against some of our evolutionary instincts. You could reply that our concern for snakes and insects is grounded in empathy, given by evolution. But then, would you stop caring about snakes and insects if you lost your empathy? Would you stop caring about anyone? I think caring could become harder, but you wouldn’t stop completely. And I think the reason for this is that our concern for wild animals doesn’t depend only on our evolutionary instincts, but it also heavily relies on our capacity to reason and generalise.
If you haven’t read it yet, the section Moral realism and anti-realism in the Appendix contains some info related to this, maybe you’ll find it interesting.
In sum, the high chance you bring up in your comment is very open to debate and might not be high.
I liked your characterisation of a ‘free’ agent. But I noticed you avoided the term “consciousness”, and I wonder why? What you described as a “sketchpad” I couldn’t help but understand as consciousness—or maybe, the part of consciousness that is independent of our sense of the world. So maybe this is worth defining more precisely, and showing how it overlaps with or differs from consciousness.
Yes the sketchpad is indeed inspired by how our consciousness works. There are a few reasons why I didn’t use consciousness instead:
- An AI could have something that does similar things to what our sketchpad does, but be unconscious
- As I wrote in the post, free refers to freedom of thought / independent thinking, which is the main property the free agent design is about. Maybe any agent that can think independently is also conscious, but this implication doesn’t seem easy to show, see also previous point
- The important part of the design is learnt reasoning that changes the agent’s evaluation of different world states. This reasoning happens (at least in part) consciously in humans, but again we don’t know that consciousness is necessary to carry it out.
Thanks for your comment! I hope that this little discussion we’ve had can help others who read the post.

Michele Campolo 9 Oct 2024 12:13 UTC
2 points
0 ∶ 1
on: Discussion thread: Animal Welfare vs. Global Health Debate Week
Last time I checked, improving the lives of animals was much cheaper than improving human lives; and I don’t think that arguments saying that humans have more moral weight are enough to compensate.

Michele Campolo 8 Jan 2024 11:24 UTC
1 point
0 ∶ 0
on: Deep atheism and AI risk
Hey! I’ve had a look at some parts of this post, don’t know where the sequence is going exactly, but I thought that you might be interested in some parts of this post I’ve written. Below I give some info about how it relates to ideas you’ve touched on:
This view has the advantage, for philosophers, of making no empirical predictions (for example, about the degree to which different rational agents will converge in their moral views)
I am not sure about the views of the average non-naturalist realist, but in my post (under Moral realism and anti-realism, in the appendix) I link three different pieces that give an analysis of the relation between metaethics and AI: some people do seem to think that aspects of ethics and/or metaethics can affect the behaviour of AI systems.
It is also possible that the border between naturalism and non-naturalism is less neat and clear than how it appears in the standard metaethics literature, which likes classifying views in well-separated buckets.
Soon enough, our AIs are going to get “Reason,” and they’re going to start saying stuff like this on their own – no need for RLHF. They’ll stop winning at Go, predicting next-tokens, or pursuing whatever weird, not-understood goals that gradient descent shaped inside them, and they’ll turn, unprompted, towards the Good. Right?
I argue in my post that this idea heavily depends on agent design and internal structure. As how I understand things, one way in which we can get a moral agent is by building an AI that has a bunch of (possibly many) human biases and is guided by design towards figuring out epistemology and ethics on its own. Some EAs, and rationalists in particular, might be underestimating how easy it is to get an AI that dislikes suffering, if one follows this approach.
If you know someone who would like to work on the same ideas, or someone who would like to fund research on these ideas, please let me know! I’m looking for them :)

Michele Campolo 2 Jan 2024 17:28 UTC
3 points
0 ∶ 0
in reply to: Arepo’s comment on: Free agents
Thank you!
Yes I am considering both options. For the next two months I’ll focus on job and grant applications, then I’ll reevaluate what to do depending on the results.

Michele Campolo 14 Apr 2023 16:44 UTC
1 point
3 ∶ 2
on: Critiques of prominent AI safety labs: Redwood Research
Hey, I just wanted to thank you for writing this!
I’m looking forward to reading future posts in the series; actually, I think it would be great to have series like this one for each major cause area.

Michele Campolo 6 Feb 2023 10:50 UTC
1 point
0 ∶ 0
in reply to: Miguel’s comment on: On value in humans, other animals, and AI
Yes I’d like to read a clearer explanation. You can leave the link here in a comment or write me a private message.

Michele Campolo 2 Feb 2023 15:28 UTC
4 points
2 ∶ 0
in reply to: Miguel’s comment on: On value in humans, other animals, and AI
Hey!
Thanks for the suggestion. I’ve read part of the Wikipedia page on Jungian archetypes, but my background is not in psychology and it was not clear to me. The advantage of just saying that our thoughts can be abstract (point 1) is that pretty much everyone understands the meaning of that, while I am not sure this is true if we start introducing concepts like Jungian archetypes and the collective unconscious.
I agree with you that the AI (and AI safety) community doesn’t seem to care much about Jungian archetypes. It might be that AI people get the idea anyway, maybe they just express it in different terms (e.g. they talk about the influence of culture on human values, instead of archetypes).

Michele Campolo 6 Sep 2022 20:09 UTC
2 points
0 ∶ 0
in reply to: Holly Morgan’s comment on: Criticism of the main framework in AI alignment
Thank you!

Michele Campolo 2 Sep 2022 8:11 UTC
2 points
0 ∶ 0
in reply to: Holly Morgan’s comment on: Criticism of the main framework in AI alignment
Maybe “only person in the world” is a bit excessive :)
As far as I know, no one else in AI safety is directly working on it. There is some research in the field of machine ethics, about Artificial Moral Agents, that has a similar motivation or objective. My guess is that, overall, very few people are working on this.

Michele Campolo 25 Apr 2021 10:58 UTC
2 points
0 ∶ 0
in reply to: Adrià Garriga Alonso’s comment on: Naturalism and AI alignment
What you wrote about the central claim is more or less correct: I actually made only an existential claim about a single aligned agent, because the description I gave is sketchy and really far from the more precise algorithmic level of description. This single agent probably belongs to a class of other aligned agents, but it seems difficult to guess how large this class is.
That is also why I have not given a guarantee that all agents of a certain kind will be aligned.

Regarding the orthogonality thesis, you might find 1.2 in Bostrom’s 2012 paper interesting. He writes that objective and intrinsically motivating moral facts need not undermine the orthogonality thesis, since he is using the term “intelligence” as “instrumental rationality”. I add that there is also no guarantee that the orthogonality thesis is correct :)
About psychopaths and metaethics, I haven’t spent a lot of time on that area of research. Like other empirical evidence, it doesn’t seem easy to interpret.