MIRI’s traditional goal would allow you to break cognition down into steps that we can describe explicitly and implement on transistors, things like “perform a step of logical deduction,” “adjust the probability of this hypothesis,” “do a step of backwards chaining,” etc. This division does not need to be competitive, but it needs to be reasonably close (close enough to obtain a decisive advantage).
Capability amplification requires breaking cognition down into steps that humans can implement. This decomposition does not need to be competitive, but it needs to be efficient enough that it can be implemented during training. Humans can obviously implement more than transistors, the main difference is that in the agent foundations case you need to figure out every response in advance (but then can have a correspondingly greater reason to think that the decomposition will work / will preserve alignment).
I can talk in more detail about the reduction from (capability amplification --> agent foundations) if it’s not clear whether it is possible and it would have an effect on your view.
On competitiveness:
I would prefer be competitive with non-aligned AI, rather than count on forming a singleton, but this isn’t really a requirement of my approach. When comparing difficulty of two approaches you should presumably compare the difficulty of achieving a fixed goal with one approach or the other.
On reliability:
On the agent foundations side, it seems like plausible approaches involve figuring out how to peer inside the previously-opaque hypotheses, or understanding what characteristic of hypotheses can lead to catastrophic generalization failures and then excluding those from induction. Both of these seem likely applicable to ML models, though would depend on how exactly they play out.
On the ML side, I think the other promising approaches involve either adversarial training, ensembling / unanimous votes, which could be applied to the agent foundations problem.
I can talk in more detail about the reduction from (capability amplification --> agent foundations) if it’s not clear whether it is possible and it would have an effect on your view.
Yeah, this is still not clear. Suppose we had a solution to agent foundations, I don’t see how that necessarily helps me figure out what to do as H in capability amplification. For example the agent foundations solution could say, use (some approximation of) exhaustive search in the following way, with your utility function as the objective function, but that doesn’t help me because I don’t have a utility function.
When comparing difficulty of two approaches you should presumably compare the difficulty of achieving a fixed goal with one approach or the other.
My point was that HRAD potentially enables the strategy of pushing mainstream AI research away from opaque designs (which are hard to compete with while maintaining alignment, because you don’t understand how they work and you can’t just blindly copy the computation that they do without risking safety), whereas in your approach you always have to worry about “how do I compete with with an AI that doesn’t have an overseer or has an overseer who doesn’t care about safety and just lets the AI use whatever opaque and potentially dangerous technique it wants”.
On the agent foundations side, it seems like plausible approaches involve figuring out how to peer inside the previously-opaque hypotheses, or understanding what characteristic of hypotheses can lead to catastrophic generalization failures and then excluding those from induction.
Oh I see. In my mind the problems with Solomonoff Induction means that it’s probably not the right way to define how induction should be done as an ideal, so we should look for something kind of like Solomonoff Induction but better, not try to patch it by doing additional things on top of it. (Like instead of trying to figure out exactly when CDT would make wrong decisions and add more complexity on top of it to handle those cases, replace it with UDT.)
My point was that HRAD potentially enables the strategy of pushing mainstream AI research away from opaque designs (which are hard to compete with while maintaining alignment, because you don’t understand how they work and you can’t just blindly copy the computation that they do without risking safety), whereas in your approach you always have to worry about “how do I compete with with an AI that doesn’t have an overseer or has an overseer who doesn’t care about safety and just lets the AI use whatever opaque and potentially dangerous technique it wants”.
I think both approaches potentially enable this, but are VERY unlikely to deliver. MIRI seems more bullish that fundamental insights will yield AI that is just plain better (Nate gave me the analogy of Judea Pearl coming up with Causal PGMs as such an insight), whereas Paul just seems optimistic that we can get a somewhat negligible performance hit for safe vs. unsafe AI.
But I don’t think MIRI has given very good arguments for why we might expect this; it would be great if someone can articulate or reference the best available arguments.
I have a very strong intuition that dauntingly large safety-performance trade-offs are extremely likely to persist in practice, thus the only answer to the “how do I compete” question seems to be “be the front-runner”.
On capability amplification:
MIRI’s traditional goal would allow you to break cognition down into steps that we can describe explicitly and implement on transistors, things like “perform a step of logical deduction,” “adjust the probability of this hypothesis,” “do a step of backwards chaining,” etc. This division does not need to be competitive, but it needs to be reasonably close (close enough to obtain a decisive advantage).
Capability amplification requires breaking cognition down into steps that humans can implement. This decomposition does not need to be competitive, but it needs to be efficient enough that it can be implemented during training. Humans can obviously implement more than transistors, the main difference is that in the agent foundations case you need to figure out every response in advance (but then can have a correspondingly greater reason to think that the decomposition will work / will preserve alignment).
I can talk in more detail about the reduction from (capability amplification --> agent foundations) if it’s not clear whether it is possible and it would have an effect on your view.
On competitiveness:
I would prefer be competitive with non-aligned AI, rather than count on forming a singleton, but this isn’t really a requirement of my approach. When comparing difficulty of two approaches you should presumably compare the difficulty of achieving a fixed goal with one approach or the other.
On reliability:
On the agent foundations side, it seems like plausible approaches involve figuring out how to peer inside the previously-opaque hypotheses, or understanding what characteristic of hypotheses can lead to catastrophic generalization failures and then excluding those from induction. Both of these seem likely applicable to ML models, though would depend on how exactly they play out.
On the ML side, I think the other promising approaches involve either adversarial training, ensembling / unanimous votes, which could be applied to the agent foundations problem.
Yeah, this is still not clear. Suppose we had a solution to agent foundations, I don’t see how that necessarily helps me figure out what to do as H in capability amplification. For example the agent foundations solution could say, use (some approximation of) exhaustive search in the following way, with your utility function as the objective function, but that doesn’t help me because I don’t have a utility function.
My point was that HRAD potentially enables the strategy of pushing mainstream AI research away from opaque designs (which are hard to compete with while maintaining alignment, because you don’t understand how they work and you can’t just blindly copy the computation that they do without risking safety), whereas in your approach you always have to worry about “how do I compete with with an AI that doesn’t have an overseer or has an overseer who doesn’t care about safety and just lets the AI use whatever opaque and potentially dangerous technique it wants”.
Oh I see. In my mind the problems with Solomonoff Induction means that it’s probably not the right way to define how induction should be done as an ideal, so we should look for something kind of like Solomonoff Induction but better, not try to patch it by doing additional things on top of it. (Like instead of trying to figure out exactly when CDT would make wrong decisions and add more complexity on top of it to handle those cases, replace it with UDT.)
I think both approaches potentially enable this, but are VERY unlikely to deliver. MIRI seems more bullish that fundamental insights will yield AI that is just plain better (Nate gave me the analogy of Judea Pearl coming up with Causal PGMs as such an insight), whereas Paul just seems optimistic that we can get a somewhat negligible performance hit for safe vs. unsafe AI.
But I don’t think MIRI has given very good arguments for why we might expect this; it would be great if someone can articulate or reference the best available arguments.
I have a very strong intuition that dauntingly large safety-performance trade-offs are extremely likely to persist in practice, thus the only answer to the “how do I compete” question seems to be “be the front-runner”.