The substantive complaint was that they [ALLFED] did an invalid calculation when calculating the annual probability of nuclear war. They did a survey to establish a range of probabilities, then they averaged them. One could argue about what kinds of ‘average them’ moves work for the first year, but over time the lack of a nuclear war is Bayesian evidence in favor of lower probabilities and against higher probabilities. It’s incorrect to not adjust for this, and the complaint was not merely the error, but that the error was pointed out and not corrected.
Tl; dr: ALLFED appreciates the feedback. We disagree that it was a mistake—there were smart people on both sides of this issue. Good epistemics are very important to ALLFED.
Full version:
Zvi is investigating the issue. I won’t name names, but suffice it to say, there were smart people disagreeing on this issue. We have been citing the fault tree analysis of the probability of nuclear war, which we think is the most rigorous study because it uses actual data. Someone did suggest that we should update the probability estimate based on the fact that nuclear war has not yet occurred (excluding World War II). Taking a look at the paper itself (see the top of page 9 and equation (5) on that page), for conditional probabilities of occurrence for which effectively zero historical occurrences have been observed out of n total cases when it could have occurred, the probability in the model was updated according to a Bayesian posterior distribution with a uniform prior and binomial likelihood function. Historical occurrences updated in this way were A) the conditional probability that Threat Assessment Conference (TAC)-level attack indicators will be promoted to a Missile Attack Conference (MAC), and (B) the conditional probability of leaders’ decision to launch in response to mistaken MAC-level indicators of being under attack. Based on this methodology, it would be double-counting to update their final distribution further based on the historical absence of accidental nuclear launches over the last 76 years.
But what we do agree on is that if one starts with a high prior, one should update. And that’s what was done by one of our coauthors for his model of the probability of nuclear war, and he got similar results to the fault tree analysis. Furthermore, the fault tree analysis was only for inadvertent nuclear war (one side thinking they are being attacked, and then “retaliating”). However, there are other mechanisms for nuclear war, including intentional attack, and accidental detonation of a nuclear weapon and escalation from there. Furthermore, though many people consider nuclear winter only possible for a US-Russia nuclear war, now that China has a greater purchasing power parity than the US, we think there is comparable combustible material there. So the possibility of US-China nuclear war or Russia-China nuclear war further increases probabilities. So even if there should be some updating downward on the inadvertent US-Russia nuclear war, I think the fault tree analysis still provides a reasonable estimate. I also explained this on my first 80k podcast.
Also, we say in the paper, “Considering uncertainty represented within our models, our result is robust: reverting the conclusion required simultaneously changing the 3-5 most important parameters to the pessimistic ends.” So as Zvi has recognized, even if one thinks the probability of nuclear war should be significantly lower, the overall conclusion doesn’t change. We have encouraged people to put their own estimates in.
Again, we really appreciate the feedback. Good epistemics are very important to us. We are trying to reach the truth. We want to have maximum positive impact on the world, so that’s why we spend a significant amount of time on prioritization.
Zvi has now put a postscript in the ALLFED section above. We have updated the inadvertent nuclear war fault tree model result based on no nuclear war since the data stopped coming in, and also reduced the annual probability of nuclear war further going forward. And then, so as to not over claim on cost effectiveness, we did not include a correction for non-inadvertent US/Russia nuclear war nor conflict with China. Resilient foods are still highly competitive with AGI safety according to the revised model.
Tl; dr: ALLFED appreciates the feedback. We disagree that it was a mistake—there were smart people on both sides of this issue. Good epistemics are very important to ALLFED.
Full version:
Zvi is investigating the issue. I won’t name names, but suffice it to say, there were smart people disagreeing on this issue. We have been citing the fault tree analysis of the probability of nuclear war, which we think is the most rigorous study because it uses actual data. Someone did suggest that we should update the probability estimate based on the fact that nuclear war has not yet occurred (excluding World War II). Taking a look at the paper itself (see the top of page 9 and equation (5) on that page), for conditional probabilities of occurrence for which effectively zero historical occurrences have been observed out of n total cases when it could have occurred, the probability in the model was updated according to a Bayesian posterior distribution with a uniform prior and binomial likelihood function. Historical occurrences updated in this way were A) the conditional probability that Threat Assessment Conference (TAC)-level attack indicators will be promoted to a Missile Attack Conference (MAC), and (B) the conditional probability of leaders’ decision to launch in response to mistaken MAC-level indicators of being under attack. Based on this methodology, it would be double-counting to update their final distribution further based on the historical absence of accidental nuclear launches over the last 76 years.
But what we do agree on is that if one starts with a high prior, one should update. And that’s what was done by one of our coauthors for his model of the probability of nuclear war, and he got similar results to the fault tree analysis. Furthermore, the fault tree analysis was only for inadvertent nuclear war (one side thinking they are being attacked, and then “retaliating”). However, there are other mechanisms for nuclear war, including intentional attack, and accidental detonation of a nuclear weapon and escalation from there. Furthermore, though many people consider nuclear winter only possible for a US-Russia nuclear war, now that China has a greater purchasing power parity than the US, we think there is comparable combustible material there. So the possibility of US-China nuclear war or Russia-China nuclear war further increases probabilities. So even if there should be some updating downward on the inadvertent US-Russia nuclear war, I think the fault tree analysis still provides a reasonable estimate. I also explained this on my first 80k podcast.
Also, we say in the paper, “Considering uncertainty represented within our models, our result is robust: reverting the conclusion required simultaneously changing the 3-5 most important parameters to the pessimistic ends.” So as Zvi has recognized, even if one thinks the probability of nuclear war should be significantly lower, the overall conclusion doesn’t change. We have encouraged people to put their own estimates in.
Again, we really appreciate the feedback. Good epistemics are very important to us. We are trying to reach the truth. We want to have maximum positive impact on the world, so that’s why we spend a significant amount of time on prioritization.
Zvi has now put a postscript in the ALLFED section above. We have updated the inadvertent nuclear war fault tree model result based on no nuclear war since the data stopped coming in, and also reduced the annual probability of nuclear war further going forward. And then, so as to not over claim on cost effectiveness, we did not include a correction for non-inadvertent US/Russia nuclear war nor conflict with China. Resilient foods are still highly competitive with AGI safety according to the revised model.