There are a bunch of increasingly hard questions on the Alignment Test. We need to get enough of the core questions right to avoid the ASI → everyone quickly dies scenario. This is the ‘passing grade’. There are some bonus/extra credit questions that we need to also get right to get an A (a flourishing future).
I think the bonus/extra credit questions are part of the main test—if you don’t get them right everyone still dies, but maybe a bit more slowly.
All the doom flows through the cracks of imperfect alignment/control. And we can asymptote toward, but never reach, existential safety[1].
Of course this applies to all other x-risks too. It’s just that ASI x-risk is very near term and acute (in absolute terms, and relative to all the others), and we aren’t even starting in earnest with the asymptoting yet (and likely won’t if we don’t get a Pause).
I think the bonus/extra credit questions are part of the main test—if you don’t get them right everyone still dies, but maybe a bit more slowly.
All the doom flows through the cracks of imperfect alignment/control. And we can asymptote toward, but never reach, existential safety[1].
Of course this applies to all other x-risks too. It’s just that ASI x-risk is very near term and acute (in absolute terms, and relative to all the others), and we aren’t even starting in earnest with the asymptoting yet (and likely won’t if we don’t get a Pause).