Michele Campolo

• Michele Campolo 26 Aug 2025 13:46 UTC

  1 point

  0 ∶ 0

  in reply to: SummaryBot’s comment on: One more rea­son for AI ca­pa­ble of in­de­pen­dent moral rea­son­ing: al­ign­ment it­self and cause prioritisation

  I think this summary is ok

• Michele Campolo 26 Aug 2025 13:42 UTC

  1 point

  0 ∶ 0

  in reply to: SummaryBot’s comment on: With enough knowl­edge, any con­scious agent acts morally

  This is an ok summary

One more rea­son for AI ca­pa­ble of in­de­pen­dent moral rea­son­ing: al­ign­ment it­self and cause prioritisation

Do­ing good… best?

With enough knowl­edge, any con­scious agent acts morally

• Michele Campolo 22 Jul 2025 13:46 UTC

  1 point

  0 ∶ 0

  in reply to: Tristan Katz’s comment on: Crit­i­cism of the main frame­work in AI alignment

  In short, I am not hoping for a specific outcome, and I can’t take into account every single scenario. If someone starts giving more credit to research on moral reasoning in AI after reading this, that’s already enough, considering that the topic doesn’t seem to be popular within AI alignment, and it was even more niche at the time I wrote this post.

• Michele Campolo 21 Jul 2025 22:07 UTC

  1 point

  0 ∶ 0

  in reply to: Tristan Katz’s comment on: Crit­i­cism of the main frame­work in AI alignment

  I hadn’t considered the narrative you bring up here when I wrote the post, that is interesting. As you write, it relies on the assumption that

  once someone ‘wins’ the AGI/​ASI race, they will be able to use that AI to control or prevent the development of other potentially dangerous AIs

  Here we are entering the realm of forecasting stuff about world politics — stuff I am definitely not an expert on. As far as I know, the probability of that scenario could be extremely low. I can also think of alternative scenarios that don’t seem obviously absurd, so I doubt that the probability is extremely high, but it’s hard for me to say much more than that. Anyway, as you said, AI moral reasoning might be valuable in that scenario as well.

  but I’m not convinced that it’s valuable in order to prevent malevolent actors using AI.

  That’s a bit too much, I don’t think I claimed that moral reasoning in AI can directly prevent that. It seems that in order to prevent malevolent actors from using AI for bad purposes we would have to either stop AI research completely, because it is not only alignment research that works on the control problem but also standard AI research; or ensure that bad actors never get access to powerful and controllable AI, which also seems hard to do and not something AI moral reasoning can help with.

  The weaker claim I made in the post is that research on moral reasoning in AI is less likely to help malevolent actors use AI for bad purposes (and/​or help them to a lesser degree) wrt research that aims to make AI controllable.

• Michele Campolo 21 Jul 2025 20:48 UTC

  1 point

  0 ∶ 0

  in reply to: Tristan Katz’s comment on: Free agents

  But I am equally concerned that this might be an easy way to misalign agents.

  I understand your concern, but I think it’s hard to evaluate both whether this is true (because no one has made that kind of experiments yet) and how much of a problem it is: the alternative is other alignment methods, which have different pros and cons, so I guess that the discussion could get very long.

  If we let them develop their own moral thinking, there is a high chance that they will develop in different ways to us, and come to very different (and not necessarily better) moral conclusions!

  I disagree with this, in particular regarding the high chance. This intuition seems to be based on the belief that morality strongly depends on what you call our starting instinctual values, which are given by evolution; but this belief is questionable. Below I’m quoting the section Moral thinking in Homo sapiens:

  In Symbolic Thought and the Evolution of Human Morality [14], Tse analyses this difference extensively and argues that “morality is rooted in both our capacities to symbolize and to generalize to a level of categorical abstraction.” I find the article compelling, and Tse’s thesis is supported also by work in moral psychology — see for example Moral Judgement as Categorization (MJAC) by McHugh et al. [12] — but here I’d like to point out a specific chain of abstract thoughts related to morality.

  As we learn more about the world, we also notice patterns about our own behaviour. We form beliefs like “I did this because of that”. Though not all of them are correct, we nonetheless realise that our actions can be steered in different directions, towards different goals, not necessarily about what satisfies our evolutionary drives. At that point, it comes natural to ask questions such as “In what directions? Which goals? Is any goal more important than others? Is anything worth doing at all?”

  Asking these questions is, I think, what kickstarts moral and ethical thinking.

  Let’s also give an example, so that we don’t just think in theoretical terms (and also because Tse’s article is quite long). Today, some people see wild animal suffering as a problem. To put it simply, we humans care about animals such as snakes and insects. But these animals actually give us aversive instinctive gut reactions, so in this case our moral beliefs go directly against some of our evolutionary instincts. You could reply that our concern for snakes and insects is grounded in empathy, given by evolution. But then, would you stop caring about snakes and insects if you lost your empathy? Would you stop caring about anyone? I think caring could become harder, but you wouldn’t stop completely. And I think the reason for this is that our concern for wild animals doesn’t depend only on our evolutionary instincts, but it also heavily relies on our capacity to reason and generalise.

  If you haven’t read it yet, the section Moral realism and anti-realism in the Appendix contains some info related to this, maybe you’ll find it interesting.

  In sum, the high chance you bring up in your comment is very open to debate and might not be high.

  I liked your characterisation of a ‘free’ agent. But I noticed you avoided the term “consciousness”, and I wonder why? What you described as a “sketchpad” I couldn’t help but understand as consciousness—or maybe, the part of consciousness that is independent of our sense of the world. So maybe this is worth defining more precisely, and showing how it overlaps with or differs from consciousness.

  Yes the sketchpad is indeed inspired by how our consciousness works. There are a few reasons why I didn’t use consciousness instead:

  • An AI could have something that does similar things to what our sketchpad does, but be unconscious

  • As I wrote in the post, free refers to freedom of thought /​ independent thinking, which is the main property the free agent design is about. Maybe any agent that can think independently is also conscious, but this implication doesn’t seem easy to show, see also previous point

  • The important part of the design is learnt reasoning that changes the agent’s evaluation of different world states. This reasoning happens (at least in part) consciously in humans, but again we don’t know that consciousness is necessary to carry it out.

  Thanks for your comment! I hope that this little discussion we’ve had can help others who read the post.

• Michele Campolo 9 Oct 2024 12:13 UTC

  2 points

  0 ∶ 1

  on: Dis­cus­sion thread: An­i­mal Welfare vs. Global Health De­bate Week

  Last time I checked, improving the lives of animals was much cheaper than improving human lives; and I don’t think that arguments saying that humans have more moral weight are enough to compensate.

Agents that act for rea­sons: a thought experiment

• Michele Campolo 8 Jan 2024 11:24 UTC

  1 point

  0 ∶ 0

  on: Deep athe­ism and AI risk

  Hey! I’ve had a look at some parts of this post, don’t know where the sequence is going exactly, but I thought that you might be interested in some parts of this post I’ve written. Below I give some info about how it relates to ideas you’ve touched on:

  This view has the advantage, for philosophers, of making no empirical predictions (for example, about the degree to which different rational agents will converge in their moral views)

  I am not sure about the views of the average non-naturalist realist, but in my post (under Moral realism and anti-realism, in the appendix) I link three different pieces that give an analysis of the relation between metaethics and AI: some people do seem to think that aspects of ethics and/​or metaethics can affect the behaviour of AI systems.

  It is also possible that the border between naturalism and non-naturalism is less neat and clear than how it appears in the standard metaethics literature, which likes classifying views in well-separated buckets.

  Soon enough, our AIs are going to get “Reason,” and they’re going to start saying stuff like this on their own – no need for RLHF. They’ll stop winning at Go, predicting next-tokens, or pursuing whatever weird, not-understood goals that gradient descent shaped inside them, and they’ll turn, unprompted, towards the Good. Right?

  I argue in my post that this idea heavily depends on agent design and internal structure. As how I understand things, one way in which we can get a moral agent is by building an AI that has a bunch of (possibly many) human biases and is guided by design towards figuring out epistemology and ethics on its own. Some EAs, and rationalists in particular, might be underestimating how easy it is to get an AI that dislikes suffering, if one follows this approach.

  If you know someone who would like to work on the same ideas, or someone who would like to fund research on these ideas, please let me know! I’m looking for them :)

• Michele Campolo 2 Jan 2024 17:28 UTC

  3 points

  0 ∶ 0

  in reply to: Arepo’s comment on: Free agents

  Thank you!

  Yes I am considering both options. For the next two months I’ll focus on job and grant applications, then I’ll reevaluate what to do depending on the results.

Free agents

• Michele Campolo 14 Apr 2023 16:44 UTC

  1 point

  3 ∶ 2

  on: Cri­tiques of promi­nent AI safety labs: Red­wood Research

  Hey, I just wanted to thank you for writing this!

  I’m looking forward to reading future posts in the series; actually, I think it would be great to have series like this one for each major cause area.

• Michele Campolo 6 Feb 2023 10:50 UTC

  1 point

  0 ∶ 0

  in reply to: Miguel’s comment on: On value in hu­mans, other an­i­mals, and AI

  Yes I’d like to read a clearer explanation. You can leave the link here in a comment or write me a private message.

• Michele Campolo 2 Feb 2023 15:28 UTC

  4 points

  2 ∶ 0

  in reply to: Miguel’s comment on: On value in hu­mans, other an­i­mals, and AI

  Hey!

  Thanks for the suggestion. I’ve read part of the Wikipedia page on Jungian archetypes, but my background is not in psychology and it was not clear to me. The advantage of just saying that our thoughts can be abstract (point 1) is that pretty much everyone understands the meaning of that, while I am not sure this is true if we start introducing concepts like Jungian archetypes and the collective unconscious.

  I agree with you that the AI (and AI safety) community doesn’t seem to care much about Jungian archetypes. It might be that AI people get the idea anyway, maybe they just express it in different terms (e.g. they talk about the influence of culture on human values, instead of archetypes).

On value in hu­mans, other an­i­mals, and AI

• Michele Campolo 6 Sep 2022 20:09 UTC

  2 points

  0 ∶ 0

  in reply to: Holly Morgan’s comment on: Crit­i­cism of the main frame­work in AI alignment

  Thank you!

• Michele Campolo 2 Sep 2022 8:11 UTC

  2 points

  0 ∶ 0

  in reply to: Holly Morgan’s comment on: Crit­i­cism of the main frame­work in AI alignment

  Maybe “only person in the world” is a bit excessive :)

  As far as I know, no one else in AI safety is directly working on it. There is some research in the field of machine ethics, about Artificial Moral Agents, that has a similar motivation or objective. My guess is that, overall, very few people are working on this.

Crit­i­cism of the main frame­work in AI alignment