Perhaps one could just bite the bullet and vow to never work on known dangerous tech ever, even if a race is possible?
Maybe the risk of losing to a totalitarian regime due to lack of superweapon advantage is an acceptable cost of lowering FUCKING EXISTENTIAL RISK?
Maybe it’s the pinnacle of human hubris to think that your specific brand of politics is worth gambling the existence of human civilization over.
I mean, the idea that superweapons can alter a major war has never had historical evidence, at all. I honestly think that it’s mostly the pride of scientists fueling that fantasy. You can see this trope in modern fiction. Many times the protagonist in a war story will come up with a gadget or tactic and that leads to winning a war… but any military historian would tell you that such things are unlikely to make a difference in real life. Why do we have this trope? The power-fantasy of the individual! It’s just like the fantasy of being a superhero with superhuman powers, winning wars single-handedly, except dressed up in a futile attempt at realism!
The AI scenario is a bit different if you believe in FOOM, but in that case it’s more likely to be suicidal than a targeted weapon anyway so you should still disavow it.
If there is no FOOM and it progresses more like how Robin Hanson envisions it, then past experience shows us that there’s more to winning a war than simply having a new technology.
Perhaps one could just bite the bullet and vow to never work on known dangerous tech ever, even if a race is possible?
Maybe the risk of losing to a totalitarian regime due to lack of superweapon advantage is an acceptable cost of lowering FUCKING EXISTENTIAL RISK?
Maybe it’s the pinnacle of human hubris to think that your specific brand of politics is worth gambling the existence of human civilization over.
I mean, the idea that superweapons can alter a major war has never had historical evidence, at all. I honestly think that it’s mostly the pride of scientists fueling that fantasy. You can see this trope in modern fiction. Many times the protagonist in a war story will come up with a gadget or tactic and that leads to winning a war… but any military historian would tell you that such things are unlikely to make a difference in real life. Why do we have this trope? The power-fantasy of the individual! It’s just like the fantasy of being a superhero with superhuman powers, winning wars single-handedly, except dressed up in a futile attempt at realism!
The AI scenario is a bit different if you believe in FOOM, but in that case it’s more likely to be suicidal than a targeted weapon anyway so you should still disavow it.
If there is no FOOM and it progresses more like how Robin Hanson envisions it, then past experience shows us that there’s more to winning a war than simply having a new technology.