Yes, I think the ’100 years’ criterion isn’t quite what we want. E.g. if there is a catastrophic setback more than 100 years after we build an aligned ASI, thenw e don’t need to rerun the alignment problem. (In practice, perhaps 100 years should be ample time to build good global governance and reduce catastrophic setback risk to near 0, but conceptually we want to clarify this.)
And I agree with Owen that shorter setbacks also seem important. In fact, in a simple binary model we could just define a catastrophic setback to be one that takes you from a society that has built aligned ASI to one where all aligned ASIs are destroyed. ie the key thing is not how many years back you go, but whether you regres back beneath the critical ‘crunch time’ period.
Yes, I think the ’100 years’ criterion isn’t quite what we want. E.g. if there is a catastrophic setback more than 100 years after we build an aligned ASI, thenw e don’t need to rerun the alignment problem. (In practice, perhaps 100 years should be ample time to build good global governance and reduce catastrophic setback risk to near 0, but conceptually we want to clarify this.)
And I agree with Owen that shorter setbacks also seem important. In fact, in a simple binary model we could just define a catastrophic setback to be one that takes you from a society that has built aligned ASI to one where all aligned ASIs are destroyed. ie the key thing is not how many years back you go, but whether you regres back beneath the critical ‘crunch time’ period.