I wrote something about CICERO, Meta’s new Diplomacy-playing AI. The summary:
CICERO is a new AI developed by Meta AI that achieves good performance at the board game Diplomacy. Diplomacy involves tactical and strategic reasoning as well as natural language communication: players must negotiate, cooperate and occasionally deceive in order to win.
CICERO comprises (1) a strategic model deciding which moves to make on the board and (2) a dialogue model communicating with the other players.
CICERO is honest in the sense that the dialogue model, when it communicates, always tries to communicate the strategy model’s actual intent; however, it can omit information and change its mind in the middle of a conversation, meaning it can behave deceptively or treacherously.
Some who are concerned with risks from advanced AI think the CICERO research project is unusually bad or risky.
It has at least three potentially-concerning aspects:
It may present an advancement in AIs’ strategic and/or tactical capabilities.
It may present an advancement in AIs’ deception and/or persuasion capabilities.
It may be illustrative of cultural issues in AI labs like Meta’s.
My low-confidence take is that (1) and (2) are false because CICERO doesn’t seem to contain any new insights that markedly advance either of these areas of study. Those capabilities are mostly the product of using reinforcement learning to master a game where tactics, strategy, deception and persuasion are useful, and I think there’s nothing surprising or technologically novel about this.
I think, with low confidence, that (3) may be true, but perhaps no more true than of any other AI project of that scale.
Neural networks using reinforcement learning are always (?) trained in simulated worlds. Chess presents a very simple world; Diplomacy, with its negotiation phase, is a substantially more complex world. Scaling up AIs to transformative and/or general heights using the reinforcement learning paradigm may require more complex and/or detailed simulations.
Simulation could be a bottleneck in creating AGI because (1) an accurate enough simulation may already give you the answers you want, (2) an accurate and/or complex enough simulation may be AI-complete and/or (3) extremely costly.
Simulation could also not be a bottleneck because, following Ajeya Cotra’s bio-anchors report, (1) we may get a lot of mileage out of simpler simulated worlds, (2) environments can contain or present problems that are easy to generate and simulate but hard to solve, (3) we may be able to automate simulation and/or (4) people will likely be willing to spend a lot of money on simulation in the future, if that leads to AGI.
CICERO does not seem like an example of a more complex or detailed simulation, since instances of CICERO didn’t actually communicate with one another during self-play. (Generating messages was apparently too computationally expensive.)
The post is written in a personal capacity and doesn’t necessarily reflect the views of my employer (Rethink Priorities).
I wrote something about CICERO, Meta’s new Diplomacy-playing AI. The summary:
CICERO is a new AI developed by Meta AI that achieves good performance at the board game Diplomacy. Diplomacy involves tactical and strategic reasoning as well as natural language communication: players must negotiate, cooperate and occasionally deceive in order to win.
CICERO comprises (1) a strategic model deciding which moves to make on the board and (2) a dialogue model communicating with the other players.
CICERO is honest in the sense that the dialogue model, when it communicates, always tries to communicate the strategy model’s actual intent; however, it can omit information and change its mind in the middle of a conversation, meaning it can behave deceptively or treacherously.
Some who are concerned with risks from advanced AI think the CICERO research project is unusually bad or risky.
It has at least three potentially-concerning aspects:
It may present an advancement in AIs’ strategic and/or tactical capabilities.
It may present an advancement in AIs’ deception and/or persuasion capabilities.
It may be illustrative of cultural issues in AI labs like Meta’s.
My low-confidence take is that (1) and (2) are false because CICERO doesn’t seem to contain any new insights that markedly advance either of these areas of study. Those capabilities are mostly the product of using reinforcement learning to master a game where tactics, strategy, deception and persuasion are useful, and I think there’s nothing surprising or technologically novel about this.
I think, with low confidence, that (3) may be true, but perhaps no more true than of any other AI project of that scale.
Neural networks using reinforcement learning are always (?) trained in simulated worlds. Chess presents a very simple world; Diplomacy, with its negotiation phase, is a substantially more complex world. Scaling up AIs to transformative and/or general heights using the reinforcement learning paradigm may require more complex and/or detailed simulations.
Simulation could be a bottleneck in creating AGI because (1) an accurate enough simulation may already give you the answers you want, (2) an accurate and/or complex enough simulation may be AI-complete and/or (3) extremely costly.
Simulation could also not be a bottleneck because, following Ajeya Cotra’s bio-anchors report, (1) we may get a lot of mileage out of simpler simulated worlds, (2) environments can contain or present problems that are easy to generate and simulate but hard to solve, (3) we may be able to automate simulation and/or (4) people will likely be willing to spend a lot of money on simulation in the future, if that leads to AGI.
CICERO does not seem like an example of a more complex or detailed simulation, since instances of CICERO didn’t actually communicate with one another during self-play. (Generating messages was apparently too computationally expensive.)
The post is written in a personal capacity and doesn’t necessarily reflect the views of my employer (Rethink Priorities).