(crossposted from twitter) Main thoughts: 1. Maps pull the territory 2. Beware what maps you summon
Leopold Aschenbrenners series of essays is a fascinating read: there is a ton of locally valid observations and arguments. Lot of the content is the type of stuff mostly discussed in private. Many of the high-level observations are correct.
At the same time, my overall impression is the set of maps sketched pulls toward existential catastrophe, and this is true not only for the ‘this is how things can go wrong’ part, but also for the ‘this is how we solve things’ part.
Leopold is likely aware of the this angle of criticism, and deflects it with ‘this is just realism’ and ‘I don’t wish things were like this, but they most likely are’. I basically don’t buy that claim.
I’m not Jan, but I think (paraphrasing) “Superintelligence will give godlike power and might kill us all. Our solution is that the good guys should race as fast as possible to build the artificial god at breakneck speed first, and then hope to align it with duct tape and prayer” should not, frankly, be your first resort strategy. If this becomes the US’s/China’s natsec community’s first introduction to considerations around superintelligence or AGI or alignment etc, I think it will predictably increase x-risk by making the zero- (actually negative-) sum framing lodged in people’s heads, before they stumble across other considerations.
(crossposted from twitter) Main thoughts:
1. Maps pull the territory
2. Beware what maps you summon
Leopold Aschenbrenners series of essays is a fascinating read: there is a ton of locally valid observations and arguments. Lot of the content is the type of stuff mostly discussed in private. Many of the high-level observations are correct.
At the same time, my overall impression is the set of maps sketched pulls toward existential catastrophe, and this is true not only for the ‘this is how things can go wrong’ part, but also for the ‘this is how we solve things’ part.
Leopold is likely aware of the this angle of criticism, and deflects it with ‘this is just realism’ and ‘I don’t wish things were like this, but they most likely are’. I basically don’t buy that claim.
Can you say more about how you think the solving things part pulls towards x-risk?
I’m not Jan, but I think (paraphrasing) “Superintelligence will give godlike power and might kill us all. Our solution is that the good guys should race as fast as possible to build the artificial god at breakneck speed first, and then hope to align it with duct tape and prayer” should not, frankly, be your first resort strategy. If this becomes the US’s/China’s natsec community’s first introduction to considerations around superintelligence or AGI or alignment etc, I think it will predictably increase x-risk by making the zero- (actually negative-) sum framing lodged in people’s heads, before they stumble across other considerations.