I sympathise with/agree with many of your points here (and in general regard AI x-risk), but something about this recent sequence of quick-takes isn’t landing with me in the way some of your other work has. I’ll try and articulate why in some cases, though I apologies if I misread or misunderstand you.
On this post, these two presises/statements raised an eyebrow:
3. Instead, AI agents will compromise, trade, and act within a system of laws indefinitely, in order to achieve their objectives, similar to what humans do now
4. Because this system of laws will descend from our current institutions and legal tradition, it is likely that humans will keep substantial legal rights, potentially retaining lots of wealth from our capital investments and property, even if we become relatively powerless compared to other AI agents in the system
To me, this is just as unsupported as people who are incredibly certain that there will be ‘treacherous turn’. I get this a supposition/alternative hypothesis, but how can you possible hold a premise that a system of laws will persist indefinitely? This sort of reminds me of the Leahy/Bach discussion where Bach just says ‘it’s going to align itself with us if it wants to if it likes us if it loves us”.I kinda want more that if we’re going to build these powerful systems, saying ’trust me bro, it’ll follow our laws and norms and love us back” doesn’t sound very convincing to me. (For clarity, I don’t think this is your position or framing, and I’m not a fan of the classic/Yudkowskian risk position. I want to say I find both perspectives unconvincing)
Secondly, people abide by systems of laws and norms, but we also have many cases of where individuals/parties/groups overturned these norms when they had accumulated enough power and didn’t feel the need to abide by the existing regime. This doesn’t have to look like the traditional DSA model where humanity gets instantly wiped out, but I don’t see why there couldn’t be a future where an AI makes move like Sulla using force to overthrow and depower the opposing factions, or the 18 Brumaire.
I sympathise with/agree with many of your points here (and in general regard AI x-risk), but something about this recent sequence of quick-takes isn’t landing with me in the way some of your other work has. I’ll try and articulate why in some cases, though I apologies if I misread or misunderstand you.
On this post, these two presises/statements raised an eyebrow:
To me, this is just as unsupported as people who are incredibly certain that there will be ‘treacherous turn’. I get this a supposition/alternative hypothesis, but how can you possible hold a premise that a system of laws will persist indefinitely? This sort of reminds me of the Leahy/Bach discussion where Bach just says ‘it’s going to align itself with us if it wants to if it likes us if it loves us”. I kinda want more that if we’re going to build these powerful systems, saying ’trust me bro, it’ll follow our laws and norms and love us back” doesn’t sound very convincing to me. (For clarity, I don’t think this is your position or framing, and I’m not a fan of the classic/Yudkowskian risk position. I want to say I find both perspectives unconvincing)
Secondly, people abide by systems of laws and norms, but we also have many cases of where individuals/parties/groups overturned these norms when they had accumulated enough power and didn’t feel the need to abide by the existing regime. This doesn’t have to look like the traditional DSA model where humanity gets instantly wiped out, but I don’t see why there couldn’t be a future where an AI makes move like Sulla using force to overthrow and depower the opposing factions, or the 18 Brumaire.