This is an important point. The difficulty with AGI x-risk is that experimental verification isn’t really possible (short of catastrophic-but-not-existential warning shots, that the most doomy people think are unlikely). Can anyone steelman with any strongly held beliefs that are justified without resort to overwhelming empirical verification? Maybe certain moral beliefs like the Golden Rule? But what about risks? Is there precedent with non-empirically-verified commonly accepted belief in a risk?
Belief in “preventing nuclear war from producing widespread annihilation is important” seems reasonably widespread, and it is supported by empirical evidence that nuclear bombs are possible, even though the claim that nuclear war would be such that it would produce widespread annihilation hasn’t been verified. But of course you can see that such widespread annihilation would be possible, by bombing the most populous cities in order.
Yeah, I thought about nuclear risk, but Hiroshima and Nagasaki seem like good enough evidence for the possibility of widespread annihilation (or even Trinity for that matter). This would only be a good example if there was widespread appreciation for GCR potential from nuclear risk before any nuclear detonations. I don’t think there was? (Especially considering that there was only a few short years (1933 − 1945) from theory to practice with the nuclear chain reaction.)
This is an important point. The difficulty with AGI x-risk is that experimental verification isn’t really possible (short of catastrophic-but-not-existential warning shots, that the most doomy people think are unlikely). Can anyone steelman with any strongly held beliefs that are justified without resort to overwhelming empirical verification? Maybe certain moral beliefs like the Golden Rule? But what about risks? Is there precedent with non-empirically-verified commonly accepted belief in a risk?
Belief in “preventing nuclear war from producing widespread annihilation is important” seems reasonably widespread, and it is supported by empirical evidence that nuclear bombs are possible, even though the claim that nuclear war would be such that it would produce widespread annihilation hasn’t been verified. But of course you can see that such widespread annihilation would be possible, by bombing the most populous cities in order.
Yeah, I thought about nuclear risk, but Hiroshima and Nagasaki seem like good enough evidence for the possibility of widespread annihilation (or even Trinity for that matter). This would only be a good example if there was widespread appreciation for GCR potential from nuclear risk before any nuclear detonations. I don’t think there was? (Especially considering that there was only a few short years (1933 − 1945) from theory to practice with the nuclear chain reaction.)