I can talk in more detail about the reduction from (capability amplification --> agent foundations) if it’s not clear whether it is possible and it would have an effect on your view.
Yeah, this is still not clear. Suppose we had a solution to agent foundations, I don’t see how that necessarily helps me figure out what to do as H in capability amplification. For example the agent foundations solution could say, use (some approximation of) exhaustive search in the following way, with your utility function as the objective function, but that doesn’t help me because I don’t have a utility function.
When comparing difficulty of two approaches you should presumably compare the difficulty of achieving a fixed goal with one approach or the other.
My point was that HRAD potentially enables the strategy of pushing mainstream AI research away from opaque designs (which are hard to compete with while maintaining alignment, because you don’t understand how they work and you can’t just blindly copy the computation that they do without risking safety), whereas in your approach you always have to worry about “how do I compete with with an AI that doesn’t have an overseer or has an overseer who doesn’t care about safety and just lets the AI use whatever opaque and potentially dangerous technique it wants”.
On the agent foundations side, it seems like plausible approaches involve figuring out how to peer inside the previously-opaque hypotheses, or understanding what characteristic of hypotheses can lead to catastrophic generalization failures and then excluding those from induction.
Oh I see. In my mind the problems with Solomonoff Induction means that it’s probably not the right way to define how induction should be done as an ideal, so we should look for something kind of like Solomonoff Induction but better, not try to patch it by doing additional things on top of it. (Like instead of trying to figure out exactly when CDT would make wrong decisions and add more complexity on top of it to handle those cases, replace it with UDT.)
My point was that HRAD potentially enables the strategy of pushing mainstream AI research away from opaque designs (which are hard to compete with while maintaining alignment, because you don’t understand how they work and you can’t just blindly copy the computation that they do without risking safety), whereas in your approach you always have to worry about “how do I compete with with an AI that doesn’t have an overseer or has an overseer who doesn’t care about safety and just lets the AI use whatever opaque and potentially dangerous technique it wants”.
I think both approaches potentially enable this, but are VERY unlikely to deliver. MIRI seems more bullish that fundamental insights will yield AI that is just plain better (Nate gave me the analogy of Judea Pearl coming up with Causal PGMs as such an insight), whereas Paul just seems optimistic that we can get a somewhat negligible performance hit for safe vs. unsafe AI.
But I don’t think MIRI has given very good arguments for why we might expect this; it would be great if someone can articulate or reference the best available arguments.
I have a very strong intuition that dauntingly large safety-performance trade-offs are extremely likely to persist in practice, thus the only answer to the “how do I compete” question seems to be “be the front-runner”.
Yeah, this is still not clear. Suppose we had a solution to agent foundations, I don’t see how that necessarily helps me figure out what to do as H in capability amplification. For example the agent foundations solution could say, use (some approximation of) exhaustive search in the following way, with your utility function as the objective function, but that doesn’t help me because I don’t have a utility function.
My point was that HRAD potentially enables the strategy of pushing mainstream AI research away from opaque designs (which are hard to compete with while maintaining alignment, because you don’t understand how they work and you can’t just blindly copy the computation that they do without risking safety), whereas in your approach you always have to worry about “how do I compete with with an AI that doesn’t have an overseer or has an overseer who doesn’t care about safety and just lets the AI use whatever opaque and potentially dangerous technique it wants”.
Oh I see. In my mind the problems with Solomonoff Induction means that it’s probably not the right way to define how induction should be done as an ideal, so we should look for something kind of like Solomonoff Induction but better, not try to patch it by doing additional things on top of it. (Like instead of trying to figure out exactly when CDT would make wrong decisions and add more complexity on top of it to handle those cases, replace it with UDT.)
I think both approaches potentially enable this, but are VERY unlikely to deliver. MIRI seems more bullish that fundamental insights will yield AI that is just plain better (Nate gave me the analogy of Judea Pearl coming up with Causal PGMs as such an insight), whereas Paul just seems optimistic that we can get a somewhat negligible performance hit for safe vs. unsafe AI.
But I don’t think MIRI has given very good arguments for why we might expect this; it would be great if someone can articulate or reference the best available arguments.
I have a very strong intuition that dauntingly large safety-performance trade-offs are extremely likely to persist in practice, thus the only answer to the “how do I compete” question seems to be “be the front-runner”.