Much better epistemics and/or coordination—out of reach now, put potentially obtainable with stronger tech.
Why are these the same category and why are you writing coordination off as impossible? It’s not. We have literally done global nonproliferation treaties before.
This bizarre notion got embedded early in EA that technological feats are possible and solving coordination problems is impossible. It’s actually the opposite—alignment is not tractable and coordination is.
I’m talking about game-changing improvements to our capabilities (mostly via more cognitive labour; not requiring superintelligence)
These are the capacities that we need to help everyone to recognize the situation we’re in and come together to do something about it (and they are partial substitutes: the better everyone’s epistemics are, the less need for a big lift on coordination which has to cover people seeing the world very differently)
I’m not actually making a claim about alignment difficulty—beyond that I do think systems in the vein of those today and the near-successors of those look pretty safe.
I think that getting people to pause AI research would be a bigger lift than any nonproliferation treaties we’ve had in the past (not that such treaties have always been effective!). This isn’t just a military tech, it’s a massively valuable economic tech. Given the incentives, and the importance of having treaties actually followed, I do think this would be a more difficult challenge than any past nonproliferation work. I don’t think that means it’s impossible, but I do think it’s way more likely if something shifts—hence my 1-3.
(Or if you were asking why I say “out of reach now” in the quoted sentence it’s because I’m literally talking about “much better coordination” as a capability; not what could or couldn’t be achieved with a certain level of coordination.)
Why are these the same category and why are you writing coordination off as impossible? It’s not. We have literally done global nonproliferation treaties before.
This bizarre notion got embedded early in EA that technological feats are possible and solving coordination problems is impossible. It’s actually the opposite—alignment is not tractable and coordination is.
These are in the same category because:
I’m talking about game-changing improvements to our capabilities (mostly via more cognitive labour; not requiring superintelligence)
These are the capacities that we need to help everyone to recognize the situation we’re in and come together to do something about it (and they are partial substitutes: the better everyone’s epistemics are, the less need for a big lift on coordination which has to cover people seeing the world very differently)
I’m not actually making a claim about alignment difficulty—beyond that I do think systems in the vein of those today and the near-successors of those look pretty safe.
I think that getting people to pause AI research would be a bigger lift than any nonproliferation treaties we’ve had in the past (not that such treaties have always been effective!). This isn’t just a military tech, it’s a massively valuable economic tech. Given the incentives, and the importance of having treaties actually followed, I do think this would be a more difficult challenge than any past nonproliferation work. I don’t think that means it’s impossible, but I do think it’s way more likely if something shifts—hence my 1-3.
(Or if you were asking why I say “out of reach now” in the quoted sentence it’s because I’m literally talking about “much better coordination” as a capability; not what could or couldn’t be achieved with a certain level of coordination.)