Counterpoint: many people dismissed longtermism as a kind of mathematical blackmail because even miniscule probability events could justify infinite resources to them. The biggest change in moving from longtermism to “holy shit x risk” was emphasizing that the probability is not miniscule.
But I agree with your first point, and think that “p(doom)” should be expanded into “p(doom | agi)” and “p(agi)”.
Or to put more bluntly, the p(doom) estimates tended to rise after the “10^35 future humans so even if p(doom) is really low...” arguments were widely dismissed.
Obviously other stuff happened in the world of AI, and AGI researchers are justified in arguing they simply updated their priors in the light of the rise of emergent behaviour exhibited by LLMs[1] (others obviously always had high and near-term expectations of doom anyway). But Bayes’ Theorem also justifies sceptics updating their p(blackmail) estimates[2]
Priors for regular events like sports, market movements and insurable events are easily converted into money so bold predictions are easily put to the test whether they claim to be based on a robust frequentist model of similar events or inside information or pure powers of observation. But doom in most of the outlined scenarios is a one-off and the incentive structure actually works the opposite way round: people arguing we’re seriously underestimating p(doom) don’t expect to be around if they’re right and are asking for resources now to reduce it. I don’t think it’s an isolated demand for rigour to suggest that a probabilistic claim of this nature bears very little resemblance to a probabilistic claim made where there’s some evidence of some base rate and strong incentive not to be overconfident.
So yeah, I agree, p(doom) isn’t persuasive and I’m not sure decomposing it into p(doom | agi) and p(agi) or equivalents for other x-risk fields puts it on a stronger footing. Understanding how researchers believe a development increases or reduces a source of x-risk is much more convincing argument about their value than incrementing or decrementing an arbitrary-seeming doom number. The “doomsday clock” was an effective rhetorical tool because everyone understood it as asking politicians to reverse course, not because it was accepted as a valid representation of an underlying probability distribution.
[1]though they could also have been justified in updating the other way; [notionally] safety-conscious organisations getting commercially valuable, near-human level outputs from a text transformation matrix arguably gives less reasons to believe anyone would deem giving machines agency worth the effort.
Oh, I agree. Arguments of the form “bad things are theoretically possible, therefore we should worry” are bad and shouldn’t be used. But “bad things are likely” is fine, and seems more likely to reach an average person than “bad things are 50% likely”.
Counterpoint: many people dismissed longtermism as a kind of mathematical blackmail because even miniscule probability events could justify infinite resources to them. The biggest change in moving from longtermism to “holy shit x risk” was emphasizing that the probability is not miniscule.
But I agree with your first point, and think that “p(doom)” should be expanded into “p(doom | agi)” and “p(agi)”.
Or to put more bluntly, the p(doom) estimates tended to rise after the “10^35 future humans so even if p(doom) is really low...” arguments were widely dismissed.
Obviously other stuff happened in the world of AI, and AGI researchers are justified in arguing they simply updated their priors in the light of the rise of emergent behaviour exhibited by LLMs[1] (others obviously always had high and near-term expectations of doom anyway). But Bayes’ Theorem also justifies sceptics updating their p(blackmail) estimates[2]
Priors for regular events like sports, market movements and insurable events are easily converted into money so bold predictions are easily put to the test whether they claim to be based on a robust frequentist model of similar events or inside information or pure powers of observation. But doom in most of the outlined scenarios is a one-off and the incentive structure actually works the opposite way round: people arguing we’re seriously underestimating p(doom) don’t expect to be around if they’re right and are asking for resources now to reduce it. I don’t think it’s an isolated demand for rigour to suggest that a probabilistic claim of this nature bears very little resemblance to a probabilistic claim made where there’s some evidence of some base rate and strong incentive not to be overconfident.
So yeah, I agree, p(doom) isn’t persuasive and I’m not sure decomposing it into p(doom | agi) and p(agi) or equivalents for other x-risk fields puts it on a stronger footing. Understanding how researchers believe a development increases or reduces a source of x-risk is much more convincing argument about their value than incrementing or decrementing an arbitrary-seeming doom number. The “doomsday clock” was an effective rhetorical tool because everyone understood it as asking politicians to reverse course, not because it was accepted as a valid representation of an underlying probability distribution.
[1]though they could also have been justified in updating the other way; [notionally] safety-conscious organisations getting commercially valuable, near-human level outputs from a text transformation matrix arguably gives less reasons to believe anyone would deem giving machines agency worth the effort.
[2]also in either direction.
Oh, I agree. Arguments of the form “bad things are theoretically possible, therefore we should worry” are bad and shouldn’t be used. But “bad things are likely” is fine, and seems more likely to reach an average person than “bad things are 50% likely”.