Paolo Bova

• Paolo Bova 11 May 2026 19:22 UTC

  1 point

  0 ∶ 0

  on: AGI Multi-Agent Align­ment Simulation

  This is super cool work, David and Zoe!
  
  It’s rare to see LLM games that contain this much structure (you have a discrete set of actions which update a world state, and even a bunch of shocks). The other thing I was impressed by is the three different LLM judges. Looking forward to seeing more visualisations.

  I have a few questions.

  • Were any challenges to getting the judges to behave reliably?

  • You mentioned seeing if there were stable ways for players to coordinate on AI alignment in the face of competitive pressure. From your work so far do you have any ideas about hypotheses or interventions that you would want to try?

  • I’m curious as to how the competitive dynamics are captured. Are you drawing upon any models of AI race dynamics? (e.g. Armstrong et al. 2016, Han et al. 2020, Stafford et al. 2022). Also, have you seen the Intelligence Rising paper by Avin et al. 2024? I’m wondering whether you’ve seen behaviours similar to what they’ve seen in their workshops?

Model­ing AI com­pe­ti­tion dy­nam­ics for bet­ter governance

• Paolo Bova 2 Feb 2026 14:01 UTC

  11 points

  0 ∶ 0

  on: The Scal­ing Series Dis­cus­sion Thread: with Toby Ord

  Super cool! Great to see others digging into the costs of Agent performance. I agree that more people should be looking into this.

  I’m particularly interested in predicting the growth of costs for Agentic AI safety evaluations. So I was wondering if you had any takes on this given this recent series. Here are a few more specific questions along those lines for you, Toby

  - Given the cost trends you’ve identified, do you expect the costs of running agents to take up an increasing share of the total costs of AI safety evaluations (including researcher costs)?
  - Which dynamics do you think will drive how the costs of AI safety evaluations change over the next few years?
  - Any thoughts on under what conditions it would be better to elicit the maximum capabilities of models using a few very expensive safety evaluations, or better to prioritise a larger quantity of evaluations that get close to plateau performance (i.e. hitting that sweet spot where their hourly cost /​ performance is lowest, or alternatively their saturation point)? Presumably a mix is a best, but how do we determine what a good mix looks like? What might you recommend to an AI Lab’s Safety/​Preparedness team? I’m thinking about how this might inform evaluation requirements for AI labs.

  Many thanks for the excellent series! You have a knack for finding elegant and intuitive ways to explain the trends from the data. Despite knowing this data well, I feel like I learn something new with every post. Looking forward to the next thing.
  

• Paolo Bova 10 Sep 2022 16:19 UTC

  2 points

  0 ∶ 0

  in reply to: Conor Barnes 🔶’s comment on: In­ter­ac­tively Vi­su­al­iz­ing X-Risk

  Thanks for pushing the fix for Windows. The share buttons work on my device now.

• Paolo Bova 9 Sep 2022 16:05 UTC

  3 points

  0 ∶ 0

  on: Jan Leike: On the wind­fall clause

  Thanks for sharing this critique, Cullen.
  
  I was curious about who would be the firm’s opponent in this scenario, i.e. the actor trying to legally implement the Windfall Clause.
  
  In a world where a Windfall of this order of magnitude is possible, I would anticipate a number of additional actors of somewhat comparable magnitude. I’d also expect states to have more wealth too (even if the AI company didn’t pay tax, an AI advanced enough to generate Windfall profits is likely to grow the economy dramatically). If this were true, I might expect there to be incentives (or the possibility of providing incentives) for sufficiently wealthy states or other actors to use their resources to keep the legal offense-defence ratio more manageable.
  
  That being said, I’m very uncertain about the above. There is certainly precedence for companies to become dramatically richer than some states. Moreover, states benefiting considerably from transformative AI may not necessarily see defending a Windfall Clause as a priority. Nevertheless, I do think there’s merit in thinking carefully about what kind of actors might exist in a world where the Windfall Clause looks like it will soon trigger.

• Paolo Bova 6 Sep 2022 15:33 UTC

  4 points

  0 ∶ 0

  in reply to: Conor Barnes 🔶’s comment on: In­ter­ac­tively Vi­su­al­iz­ing X-Risk

  Great to see the Predict feature. I might have missed this when you first added it, but I’ve seen it now. It looks great and the tool is easy to use! I also like the additional changes you’ve made to make the site more polished. Myself and a friend had some issues when clicking the ‘share’ button which I’ll post as an issue on the Github later.

• Paolo Bova 4 Aug 2022 11:39 UTC

  9 points

  1 ∶ 0

  on: AI ethics: the case for in­clud­ing an­i­mals (my first pub­lished pa­per, Peter Singer’s first on AI)

  I’m very glad to see a paper on this topic. This paper is precisely what the field of AI Ethics has been missing!
  
  Congratulations on the first publication, Fai!
  
  A few highlights from the paper:

  “It is significant that philosophers who disagree strongly with the view that animals have rights or are entitled to equal consideration of interests nevertheless accept that factory farming is indefensible.”

  • In general, the piece appears to do a great job at preempting arguments against animal ethics. Here’s hoping a lot of people see this!

  “Companies that contribute to making the factory farming industry more resilient and better able to resist replacement by less cruel and more sustainable alternatives are acting unethically.”

  • This makes a very clear statement.

  “So, instead of self-driving cars creating a new ethical problem with regard to hitting animals, we will have, when self-driving cars become common, a potential solution to an old ethical problem, and with the new solution, new responsibilities.”

  • This section on self-driving cars is brilliantly practical. Hopefully AI ethics scholars take note as this seems like a rather undaunting and high profile case study.

  “While we appreciate Delphi’s developers’ effort … we are yet to see any efort to make it less speciesist. Until that happens, we agree with Delphi’s developers that Delphi’s output, or outputs from any similar models, should “not be used for advice for humans,” nor should it be used as a model to build ethics in AI.”

  • I agree with this point and I suspect it applies more widely to research advancing AI capabilities.

An­nounc­ing the SPT Model Web App for AI Governance

• Paolo Bova 3 Aug 2022 21:59 UTC

  2 points

  0 ∶ 0

  on: In­for­ma­tion in risky tech­nol­ogy races

  Fantastic summary, Nicholas, Andrew, and Robert. I’m looking forward to reading the paper.
  
  A few quick thoughts on the summary:

  1. It’s reassuring to hear that information hazards are unlikely for lower values of the decisiveness parameter. One relevant follow-up question is how might AI developers form an opinion on what value the decisiveness parameter takes? Is this something we can hope to influence?

  2. It’s not quite as reassuring to hear that framing AI Safety as a group effort might discourage safety investments due to moral hazard. I do find your proposal to share safety knowledge with the leader to be promising. We might also want policymakers to have some way to ensure that those sharing this safety knowledge were well compensated. Doing so might give a preemptive motive for companies to invest in safety, as they might be able to sell it to the leader if they fall behind in the race.

  3. I really like that you caution against updating only on the basis of a model alone. It encourages me to think about how we might empirically test these claims concerning moral hazard and decisiveness.

• Paolo Bova 30 Jul 2022 11:44 UTC

  3 points

  0 ∶ 0

  on: In­ter­ac­tively Vi­su­al­iz­ing X-Risk

  Beautifully made! I love the visuals and my first impressions are that it communicates x-risk in a more hopeful way. The app looks great on mobile too.
  
  Some quick thoughts:
  
  - I anticipated that clicking on a node would either give me a tooltip to explain what that particular node should represent or take me to another page/​section of the site which explained these scenarios in more detail.
  - I initially found it strange that all of the green nodes appear to link to the same prediction about population decline. I vaguely understood that this was a source of evidence for the number of green nodes, but the connection is not very clear. I think the app might benefit from a short explanation of why a user might want to click on these nodes. It might also help if hovering over one node highlighted all nodes which send you to the same place.
  - I feel that the text on the graph is sufficient enough for me to understand the different clusters in the graph. Yet, I wonder if it might look better to use icons to represent these different clusters, and have the longer text appear on hover instead. Of course, I’d keep it as it is if user testing suggested that this change increased confusion.
  - I will cast a vote for being able to input my own data. If I could input my own data, I also think it would be fun to share the resulting graphs.
  - I don’t think I have any ideas for a better title. I do feel that another title should aim to be of a similar length.
  - A few ideas for promoting the app to other EAs. It might be nice to give a talk about the web app, or for someone whose work is closely related to predictions for x-risk to show it off in a talk. Also, perhaps you could reach out to one of the university EA groups to see if they’d be interested in having a visual like this to show in some of their introductory talks.
  
  Lastly, I’d like to congratulate you on launching the site. I’m sure you’ve put in a lot of work to get it to this point, and as a result it looks fantastic!