Great post! I agree with your overall assessment that other approaches may be more promising than HRAD.
I’d like to add that this may (in part) depend on our outlook on which AI scenarios are likely. Conditional on MIRI’s view that a hard or unexpected takeoff is likely, HRAD may be more promising (though it’s still unclear). If the takeoff is soft or AI will be more like the economy, then I personally think HRAD is unlikely to be the best way to shape advanced AI.
Conditional on MIRI’s view that a hard or unexpected takeoff is likely, HRAD is more promising (though it’s still unclear).
Do you mean more promising than other technical safety research (e.g. concrete problems, Paul’s directions, MIRI’s non-HRAD research)? If so, I’d be interested in hearing why you think hard / unexpected takeoff differentially favors HRAD.
Do you mean more promising than other technical safety research (e.g. concrete problems, Paul’s directions, MIRI’s non-HRAD research)?
Yeah, and also (differentially) more promising than AI strategy or AI policy work. But I’m not sure how strong the effect is.
If so, I’d be interested in hearing why you think hard / unexpected takeoff differentially favors HRAD.
In a hard / unexpected takeoff scenario, it’s more plausible that we need to get everything more or less exactly right to ensure alignment, and that we have only one shot at it. This might favor HRAD because a less principled approach makes it comparatively unlikely that we get all the fundamentals right when we build the first advanced AI system.
In contrast, if we think there’s no such discontinuity and AI development will be gradual, then AI control may be at least somewhat more similar (but surely not entirely comparable) to how we “align” contemporary software systems. That is, it would be more plausible that we could test advanced AI systems extensively without risking catastrophic failure or that we could iteratively try a variety of safety approaches to see what works best.
It would also be more likely that we’d get warning signs of potential failure modes, so that it’s comparatively more viable to work on concrete problems whenever they arise, or to focus on making the solutions to such problems scalable – which, to my understanding, is a key component of Paul’s approach. In this picture, successful alignment without understanding the theoretical fundamentals is more likely, which makes non-HRAD approaches more promising.
My personal view is that I find a hard and unexpected takeoff unlikely, and accordingly favor other approaches than HRAD, but of course I can’t justify high confidence in this given expert disagreement. Similarly, I’m not highly confident that the above distinction is actually meaningful.
I’d be interested in hearing your thoughts on this!
There’s a strong possibility, even in a soft takeoff, that an unaligned AI would not act in an alarming way until after it achieves a decisive strategic advantage. In that case, the fact that it takes the AI a long time to achieve a decisive strategic advantage wouldn’t do us much good, since we would not pick up an indication that anything was amiss during that period.
Reasons an AI might act in a desirable manner before but not after achieving a decisive strategic advantage:
Prior to achieving a decisive strategic advantage, the AI relies on cooperation with humans to achieve its goals, which provides an incentive not to act in ways that would result in it getting shut down. An AI may be capable of following these incentives well before achieving a decisive strategic advantage.
It may be easier to give an AI a goal system that aligns with human goals in familiar circumstances than it is to give it a goal system that aligns with human goals in all circumstances. An AI with such a goal system would act in ways that align with human goals if it has little optimization power but in ways that are not aligned with human goals if it has sufficiently large optimization power, and it may attain that much optimization power only after achieving a decisive strategic advantage (or before achieving a decisive strategic advantage, but after acquiring the ability to behave deceptively, as in the previous reason).
There’s a strong possibility, even in a soft takeoff, that an unaligned AI would not act in an alarming way until after it achieves a decisive strategic advantage.
That’s assuming that the AI is confident that it will achieve a DSA eventually, and that no competitors will do so first. (In a soft takeoff it seems likely that there will be many AIs, thus many potential competitors.) The worse the AI thinks its chances are of eventually achieving a DSA first, the more rational it becomes for it to risk non-cooperative action at the point when it thinks it has the best chances of success—even if those chances were low. That might help reveal unaligned AIs during a soft takeoff.
Interestingly this suggests that the more AIs there are, the easier it might be to detect unaligned AIs (since every additional competitor decreases any given AI’s odds of getting a DSA first), and it suggests some unintuitive containment strategies such as explicitly explaining to the AI when it would be rational for it to go uncooperative if it was unaligned, to increase the odds of unaligned AIs really risking hostile action early on and being discovered...
Or it could just assume the AI has an unbounded utility function (or bounded very highly). An AI could guess it only has a 1 in 1/B chance of reaching DSA, but that the payoff from reaching this is 100B higher than defecting early. Since there are 100B stars in the galaxy, it seems likely that in a multipolar situation with decent diversity of AIs, some would fulfill this criteria and decide to gamble.
In a hard / unexpected takeoff scenario, it’s more plausible that we need to get everything more or less exactly right to ensure alignment, and that we have only one shot at it. This might favor HRAD because a less principled approach makes it comparatively unlikely that we get all the fundamentals right when we build the first advanced AI system.
FWIW, I’m not ready to cede the “more principled” ground to HRAD at this stage; to me,
it seems like the distinction is more about which aspects of an AI system’s behavior we’re specifying manually, and which aspects we’re setting it up to learn. As far as trying to get everything right the first time, I currently favor a corrigibility kind of approach, as I described in 3c above—I’m worried that trying to solve everything formally ahead of time will actually expose us to more risk.
Great post! I agree with your overall assessment that other approaches may be more promising than HRAD.
I’d like to add that this may (in part) depend on our outlook on which AI scenarios are likely. Conditional on MIRI’s view that a hard or unexpected takeoff is likely, HRAD may be more promising (though it’s still unclear). If the takeoff is soft or AI will be more like the economy, then I personally think HRAD is unlikely to be the best way to shape advanced AI.
(I wrote a related piece on strategic implications of AI scenarios.)
Thanks!
Do you mean more promising than other technical safety research (e.g. concrete problems, Paul’s directions, MIRI’s non-HRAD research)? If so, I’d be interested in hearing why you think hard / unexpected takeoff differentially favors HRAD.
Yeah, and also (differentially) more promising than AI strategy or AI policy work. But I’m not sure how strong the effect is.
In a hard / unexpected takeoff scenario, it’s more plausible that we need to get everything more or less exactly right to ensure alignment, and that we have only one shot at it. This might favor HRAD because a less principled approach makes it comparatively unlikely that we get all the fundamentals right when we build the first advanced AI system.
In contrast, if we think there’s no such discontinuity and AI development will be gradual, then AI control may be at least somewhat more similar (but surely not entirely comparable) to how we “align” contemporary software systems. That is, it would be more plausible that we could test advanced AI systems extensively without risking catastrophic failure or that we could iteratively try a variety of safety approaches to see what works best.
It would also be more likely that we’d get warning signs of potential failure modes, so that it’s comparatively more viable to work on concrete problems whenever they arise, or to focus on making the solutions to such problems scalable – which, to my understanding, is a key component of Paul’s approach. In this picture, successful alignment without understanding the theoretical fundamentals is more likely, which makes non-HRAD approaches more promising.
My personal view is that I find a hard and unexpected takeoff unlikely, and accordingly favor other approaches than HRAD, but of course I can’t justify high confidence in this given expert disagreement. Similarly, I’m not highly confident that the above distinction is actually meaningful.
I’d be interested in hearing your thoughts on this!
There’s a strong possibility, even in a soft takeoff, that an unaligned AI would not act in an alarming way until after it achieves a decisive strategic advantage. In that case, the fact that it takes the AI a long time to achieve a decisive strategic advantage wouldn’t do us much good, since we would not pick up an indication that anything was amiss during that period.
Reasons an AI might act in a desirable manner before but not after achieving a decisive strategic advantage:
Prior to achieving a decisive strategic advantage, the AI relies on cooperation with humans to achieve its goals, which provides an incentive not to act in ways that would result in it getting shut down. An AI may be capable of following these incentives well before achieving a decisive strategic advantage.
It may be easier to give an AI a goal system that aligns with human goals in familiar circumstances than it is to give it a goal system that aligns with human goals in all circumstances. An AI with such a goal system would act in ways that align with human goals if it has little optimization power but in ways that are not aligned with human goals if it has sufficiently large optimization power, and it may attain that much optimization power only after achieving a decisive strategic advantage (or before achieving a decisive strategic advantage, but after acquiring the ability to behave deceptively, as in the previous reason).
That’s assuming that the AI is confident that it will achieve a DSA eventually, and that no competitors will do so first. (In a soft takeoff it seems likely that there will be many AIs, thus many potential competitors.) The worse the AI thinks its chances are of eventually achieving a DSA first, the more rational it becomes for it to risk non-cooperative action at the point when it thinks it has the best chances of success—even if those chances were low. That might help reveal unaligned AIs during a soft takeoff.
Interestingly this suggests that the more AIs there are, the easier it might be to detect unaligned AIs (since every additional competitor decreases any given AI’s odds of getting a DSA first), and it suggests some unintuitive containment strategies such as explicitly explaining to the AI when it would be rational for it to go uncooperative if it was unaligned, to increase the odds of unaligned AIs really risking hostile action early on and being discovered...
Or it could just assume the AI has an unbounded utility function (or bounded very highly). An AI could guess it only has a 1 in 1/B chance of reaching DSA, but that the payoff from reaching this is 100B higher than defecting early. Since there are 100B stars in the galaxy, it seems likely that in a multipolar situation with decent diversity of AIs, some would fulfill this criteria and decide to gamble.
Thanks Tobias.
FWIW, I’m not ready to cede the “more principled” ground to HRAD at this stage; to me, it seems like the distinction is more about which aspects of an AI system’s behavior we’re specifying manually, and which aspects we’re setting it up to learn. As far as trying to get everything right the first time, I currently favor a corrigibility kind of approach, as I described in 3c above—I’m worried that trying to solve everything formally ahead of time will actually expose us to more risk.