The claim “Teleportation will be invented next year” is extraordinary. If 20% of physicists believe the claim to be true, we should think there’s a significant chance it will occur. The claim “AI will cause human extinction” is also extraordinary, but, if a significant portion of AI researchers believe it is true, then we should probably think there’s a significant chance (with no additional knowledge) that this will occur. Something can be extraordinary and true. General relativity is extraordinary and true.
Part of what I’m struggling with is that “existential risk” isn’t a claim, it’s a concept. So saying “existential risk is an extraordinary claim” makes no sense. “Teleportation will be invented next year” is a very specific claim I suppose you could call extraordinary. I’m not sure what specific claim you’re evaluating. Maybe it’s the claim that “there is a realistic chance of an existential catastrophe occurring in the next 5 years” or something like that?
I also don’t know what criteria you use to call something “extraordinary” or not. What is it that actually makes general relativity extraordinary?
Are you essentially saying our prior belief in an existential catastrophe occurring should be very low because something so drastic has never happened before, and the evidence for it is not strong enough to shift us to a position where it should be a priority for us? This would be a framing that makes more sense to me.
I think that the statement “There is a one in six chance that humanity will permanently destroy its future potential value” is an extraordinary claim.
What makes something extraordinary is definitely vibes-based and meant as a framing. For me, it is essentially whether the claim involves a vast change in our typical understanding of the world or a vast change in how our behavior should be in context to our world. The claim “There is a second world where people live forever” is an extraordinary claim. The claim “Lifespans in this part of Afghanistan are decreasing” is not an extraordinary claim.
Yeah, your last paragraph is how I think I should have re-framed the article. We have very little precedence for x-risks, given that the ones people worry about all have essentially no historical precedence. Additionally, since humans are the most intelligent, resilient, and creative species, we should expect that the risks that humans face must be truly vast in scale and unable to be defended against.
I generally worry that, because our brains have a proclivity towards believing we’re in the end times, we may waste resources by lowering our epistemic standards when the concept of existential risk is raised.
The claim “Teleportation will be invented next year” is extraordinary. If 20% of physicists believe the claim to be true, we should think there’s a significant chance it will occur. The claim “AI will cause human extinction” is also extraordinary, but, if a significant portion of AI researchers believe it is true, then we should probably think there’s a significant chance (with no additional knowledge) that this will occur. Something can be extraordinary and true. General relativity is extraordinary and true.
Part of what I’m struggling with is that “existential risk” isn’t a claim, it’s a concept. So saying “existential risk is an extraordinary claim” makes no sense. “Teleportation will be invented next year” is a very specific claim I suppose you could call extraordinary. I’m not sure what specific claim you’re evaluating. Maybe it’s the claim that “there is a realistic chance of an existential catastrophe occurring in the next 5 years” or something like that?
I also don’t know what criteria you use to call something “extraordinary” or not. What is it that actually makes general relativity extraordinary?
Are you essentially saying our prior belief in an existential catastrophe occurring should be very low because something so drastic has never happened before, and the evidence for it is not strong enough to shift us to a position where it should be a priority for us? This would be a framing that makes more sense to me.
I think that the statement “There is a one in six chance that humanity will permanently destroy its future potential value” is an extraordinary claim.
What makes something extraordinary is definitely vibes-based and meant as a framing. For me, it is essentially whether the claim involves a vast change in our typical understanding of the world or a vast change in how our behavior should be in context to our world. The claim “There is a second world where people live forever” is an extraordinary claim. The claim “Lifespans in this part of Afghanistan are decreasing” is not an extraordinary claim.
Yeah, your last paragraph is how I think I should have re-framed the article. We have very little precedence for x-risks, given that the ones people worry about all have essentially no historical precedence. Additionally, since humans are the most intelligent, resilient, and creative species, we should expect that the risks that humans face must be truly vast in scale and unable to be defended against.
I generally worry that, because our brains have a proclivity towards believing we’re in the end times, we may waste resources by lowering our epistemic standards when the concept of existential risk is raised.