Yeah, a big part of why I left the term vague is because I didn’t want people to get hung up on those details when many AGI catastrophe scenarios are extreme enough to swamp those details. E.g., focusing on whether the astronomical loss threshold is 80% vs. 50% is besides the point if you think AGI failure almost always means losing 98+% of the future’s value.
I might still do it differently if I could re-run the survey, however. It would be nice to have a number, so we could more easily do EV calculations.
I’d be interested in seeing operationalizations at some subset of {1%, 10%, 50%, 90, 99%}.* I can imagine that most safety researchers will give nearly identical answers to all of them, but I can also imagine that large divergences, so decent value of information here.
*Probably can’t do all 5, at least not at once, because of priming effects.
Yeah, a big part of why I left the term vague is because I didn’t want people to get hung up on those details when many AGI catastrophe scenarios are extreme enough to swamp those details. E.g., focusing on whether the astronomical loss threshold is 80% vs. 50% is besides the point if you think AGI failure almost always means losing 98+% of the future’s value.
I might still do it differently if I could re-run the survey, however. It would be nice to have a number, so we could more easily do EV calculations.
I’d be interested in seeing operationalizations at some subset of {1%, 10%, 50%, 90, 99%}.* I can imagine that most safety researchers will give nearly identical answers to all of them, but I can also imagine that large divergences, so decent value of information here.
*Probably can’t do all 5, at least not at once, because of priming effects.