Really enjoyed this episode, broadly a big fan! Pretty skeptical of the overall alignment strategy, however (though maybe I just don’t get it)
One significant criticism I have is that you mentioned the capabilities tax, but I think the performance hit issue I’m most worried about for truth-seeking oracles vs the current/future paradigm isn’t the performance hit of being worse at capabilities (in the sense of getting true information over the world) without agentic data gathering (though that’s a major worry too), but it being much worse in terms of impact.
Imagine having two superintelligent AI systems in war. One army’s aimbot answers questions like “where should I aim so I can achieve my strategic and military objectives”, the other army’s killbot just has a gun and starts shooting. Sure seems like the second one is structurally advantaged!
This is easiest to see in the military case but there are various analogues like that economically as well. Imagine superhuman business advice bot going against a business that’s staffed top-to-bottom with remote-only superintelligences plus increasing robotics integration.
(The podcast briefly mentioned a scientist agent AI though I’m confused how it can perform similarly on agentic situations with only prediction and not direct RL)
IMO a “Scientist AI” is more promising in a world where we first get a global ban on superintelligent AI, or something else that prevents anyone from building “high-impact AI”. Then AI developers coalesce around “Scientist AI” as a safe approach and develop it carefully.
(I still think a Scientist AI would result in everyone dying, but at least it’s a better starting point)
(I still think a Scientist AI would result in everyone dying, but at least it’s a better starting point)
I think Yoshua’s hope with Scientist AI is that at the weakly superhuman point (and maybe before then) you can ask it questions like “are we doomed if we go with the following alignment strategy” and the Scientist AI is like “88.5%” I think I’d be an optimist if we lived in a world like that (<1.5% misalignment doom conditional upon a fully aligned truth-seeking weakly superhuman AI people use, while more dangerous approaches are tamped down).
I basically agree, although “dangerous approaches are tamped down” is doing most of the work here IMO. By default (i.e. no tamping-down), I expect the situation with a weakly-superhuman Scientist AI to be:
a small number of sane people ask “are we doomed if we go with the following alignment strategy”, and when it says yes, they don’t do it
a lot of people don’t bother to ask at all, they just ask the Scientist AI how to build ASI
a lot of people say “we have to build ASI before the reckless people in group 2”, they build ASI using their best-guess alignment strategy that has an 88.5% chance of failing, and we die with 88.5% probability
(I think Bengio would agree that this is a concern, and would agree that we need global coordination on AI safety to make this work.)
I guess the default for me is that Scientist AI won’t be competitive, so we live in a world with both scientist AI and non-scientist AI. Conditional upon successfully tamping down other approaches enough that Scientist AI gets to the weakly superhuman point while we’re still alive, I’m more optimistic that we can continue to coordinate on doing things safely.
Really enjoyed this episode, broadly a big fan! Pretty skeptical of the overall alignment strategy, however (though maybe I just don’t get it)
One significant criticism I have is that you mentioned the capabilities tax, but I think the performance hit issue I’m most worried about for truth-seeking oracles vs the current/future paradigm isn’t the performance hit of being worse at capabilities (in the sense of getting true information over the world) without agentic data gathering (though that’s a major worry too), but it being much worse in terms of impact.
Imagine having two superintelligent AI systems in war. One army’s aimbot answers questions like “where should I aim so I can achieve my strategic and military objectives”, the other army’s killbot just has a gun and starts shooting. Sure seems like the second one is structurally advantaged!
This is easiest to see in the military case but there are various analogues like that economically as well. Imagine superhuman business advice bot going against a business that’s staffed top-to-bottom with remote-only superintelligences plus increasing robotics integration.
I wrote about that point here, in the second section.
(The podcast briefly mentioned a scientist agent AI though I’m confused how it can perform similarly on agentic situations with only prediction and not direct RL)
IMO a “Scientist AI” is more promising in a world where we first get a global ban on superintelligent AI, or something else that prevents anyone from building “high-impact AI”. Then AI developers coalesce around “Scientist AI” as a safe approach and develop it carefully.
(I still think a Scientist AI would result in everyone dying, but at least it’s a better starting point)
I think Yoshua’s hope with Scientist AI is that at the weakly superhuman point (and maybe before then) you can ask it questions like “are we doomed if we go with the following alignment strategy” and the Scientist AI is like “88.5%” I think I’d be an optimist if we lived in a world like that (<1.5% misalignment doom conditional upon a fully aligned truth-seeking weakly superhuman AI people use, while more dangerous approaches are tamped down).
I basically agree, although “dangerous approaches are tamped down” is doing most of the work here IMO. By default (i.e. no tamping-down), I expect the situation with a weakly-superhuman Scientist AI to be:
a small number of sane people ask “are we doomed if we go with the following alignment strategy”, and when it says yes, they don’t do it
a lot of people don’t bother to ask at all, they just ask the Scientist AI how to build ASI
a lot of people say “we have to build ASI before the reckless people in group 2”, they build ASI using their best-guess alignment strategy that has an 88.5% chance of failing, and we die with 88.5% probability
(I think Bengio would agree that this is a concern, and would agree that we need global coordination on AI safety to make this work.)
I guess the default for me is that Scientist AI won’t be competitive, so we live in a world with both scientist AI and non-scientist AI. Conditional upon successfully tamping down other approaches enough that Scientist AI gets to the weakly superhuman point while we’re still alive, I’m more optimistic that we can continue to coordinate on doing things safely.