Thanks for highlighting these links! I’ve now read them and found them interesting. I have a few somewhat vague/broad questions I’d be interested to hear your reactions to:
Do you have thoughts on how impactful the Treaty on the Prohibition of Nuclear Weapons has been or will be?
Do you have thoughts on how impactful treaties intended to reduce existential risk from AI could be, and what the best form for them to take is?
These treaties could focus on things like reducing the chance of corner-cutting/race dynamics, incentivising/requiring better alignment/safety methods, reducing risks of misuse, etc.
Do you have thoughts on what sorts of questions it’d be useful to have forecasts about, in order to inform answers to those questions?
These could be proxies we’ll get answers about in the next couple years, or further afield and more directly important things.
I ask because I may look into one or both of the first two of those questions later this year, and will probably be involved in setting up a batch of nuclear-risk-related forecasting questions for the site Metaculus soon.
But as I say, those questions are somewhat vague/broad, and I already know of other sources, contacts, etc. - so please don’t feel compelled to respond unless you find those questions interesting or already have pre-loaded thoughts.
One piece of info related to the NPT that might be helpful: The NPT does contain an article (article VI) in which the the P5 (the 5 current permanent members of the UN Security Council, who at the time the NPT was made were the only countries who had successfully tested nuclear weapons) as well as all of the parties agree to participate in good-faith negotiations to pursue nuclear disarmament, but it does not specify a time-table and the language is deliberately vague. I think the NPT has done a good job of doing what its main goal is and what its name implies: limiting nuclear proliferation. It has clearly not been able to get existing nuclear powers to get rid of all of their nuclear weapons (although it’s hard to know if the NPT has not played a significant role in reducing the arsenals of the nuclear powers relative to a counterfactual world in which there was no NPT… perhaps in a counterfactual world without the NPT, the US and USSR would not feel compelled to engage in the SALT and START negotiations without officially committing in Article VI of the NPT). Luisa has done a lot more thinking about the Nuclear Ban treaty, so I’ll defer to her on that.
Jeffrey Ohl is going to look into this question as part of a summer project for the Stanford Existential Risk Initiative, and I’ll be mentoring him. We literally just started talking about the project last week, so more to come on that! For now, I’ll say the characteristics of the technology itself that you are trying to regulate (e.g. fissile material vs. inputs to the creating of AGI for instance) is very important in terms of how a successful treaty could be constructed. This aspect is important in terms of the mechanisms that must be put in place to verify compliance with the treaty. I co-wrote an article with Chris Bakerlee on engineered pathogens for Vox a few years ago that discusses, among other things, some of the challenges around regulating biotechnology that make verifying compliance with the Biological Weapons Convention difficult https://www.vox.com/future-perfect/2018/12/6/18127430/superbugs-biotech-pathogens-biorisk-pandemic
What time range are you looking at when it comes to forecasts, and what sort of things do you have in mind? I recall when I read Superforecasting a few years ago that forecasts aren’t particularly reliable beyond a few years even for Superforecasters (though correct me if I’m wrong/maybe views on that are different now than they were then?). These treaties operate on pretty long time-scales… e.g. the NPT was conceived of in the mid 1960s, it was signed in 1968, it went into force in 1970, and then countries joined over the course of a few decades. https://en.wikipedia.org/wiki/List_of_parties_to_the_Treaty_on_the_Non-Proliferation_of_Nuclear_Weapons
I’ll be quite interested to learn more about Jeffrey’s project once it’s further along. I might reach out to you or Jeffrey in a few weeks about that.
Regarding the forecasts, we can have any time range and any topic. I already have a bunch of ideas, but just wanted to see if anything bubbled to mind for you independently which I could add to my list. (It’s ok if not!)
I recall when I read Superforecasting a few years ago that forecasts aren’t particularly reliable beyond a few years even for Superforecasters (though correct me if I’m wrong/maybe views on that are different now than they were then?).
I guess this might depend what you mean by “particularly reliable”.
My understanding is that there’s basically just no good evidence either way regarding how accurate and calibrated forecasts are over long time-scales (at least if we restrict ourselves to relevant kinds of forecasts, e.g. ones made by people who seem to have been genuinely trying rather than just making claims for rhetorical/political effect). But there’s a little evidence (from Tetlock) to suggest that accuracy may decline relatively slowly after the first year or so. See in particular the great post How Feasible Is Long-range Forecasting?, footnote 17 there, and posts tagged long-range forecasting. Here’s the summary of that post:
How accurate do long-range (≥10yr) forecasts tend to be, and how much should we rely on them?
As an initial exploration of this question, I sought to study the track record of long-range forecasting exercises from the past. Unfortunately, my key finding so far is that it is difficult to learn much of value from those exercises, for the following reasons:
Long-range forecasts are often stated too imprecisely to be judged for accuracy. [More]
Even if a forecast is stated precisely, it might be difficult to find the information needed to check the forecast for accuracy. [More]
Degrees of confidence for long-range forecasts are rarely quantified. [More]
In most cases, no comparison to a “baseline method” or “null model” is possible, which makes it difficult to assess how easy or difficult the original forecasts were. [More]
Incentives for forecaster accuracy are usually unclear or weak. [More]
Very few studies have been designed so as to allow confident inference about which factors contributed to forecasting accuracy. [More]
It’s difficult to know how comparable past forecasting exercises are to the forecasting we [at Open Phil] do for grantmaking purposes, e.g. because the forecasts we make are of a different type, and because the forecasting training and methods we use are different. [More]
We plan to continue to make long-range quantified forecasts about our work so that, in the long run, we might learn something about the feasibility of long-range forecasting, at least for our own case. [More]
Thanks for highlighting these links! I’ve now read them and found them interesting. I have a few somewhat vague/broad questions I’d be interested to hear your reactions to:
Do you have thoughts on how impactful the Treaty on the Prohibition of Nuclear Weapons has been or will be?
(See also one partial analysis on that here: Will the Treaty on the Prohibition of Nuclear Weapons affect nuclear deproliferation through legal channels?)
Do you have thoughts on how impactful treaties intended to reduce existential risk from AI could be, and what the best form for them to take is?
These treaties could focus on things like reducing the chance of corner-cutting/race dynamics, incentivising/requiring better alignment/safety methods, reducing risks of misuse, etc.
Do you have thoughts on what sorts of questions it’d be useful to have forecasts about, in order to inform answers to those questions?
These could be proxies we’ll get answers about in the next couple years, or further afield and more directly important things.
I ask because I may look into one or both of the first two of those questions later this year, and will probably be involved in setting up a batch of nuclear-risk-related forecasting questions for the site Metaculus soon.
But as I say, those questions are somewhat vague/broad, and I already know of other sources, contacts, etc. - so please don’t feel compelled to respond unless you find those questions interesting or already have pre-loaded thoughts.
One piece of info related to the NPT that might be helpful: The NPT does contain an article (article VI) in which the the P5 (the 5 current permanent members of the UN Security Council, who at the time the NPT was made were the only countries who had successfully tested nuclear weapons) as well as all of the parties agree to participate in good-faith negotiations to pursue nuclear disarmament, but it does not specify a time-table and the language is deliberately vague. I think the NPT has done a good job of doing what its main goal is and what its name implies: limiting nuclear proliferation. It has clearly not been able to get existing nuclear powers to get rid of all of their nuclear weapons (although it’s hard to know if the NPT has not played a significant role in reducing the arsenals of the nuclear powers relative to a counterfactual world in which there was no NPT… perhaps in a counterfactual world without the NPT, the US and USSR would not feel compelled to engage in the SALT and START negotiations without officially committing in Article VI of the NPT). Luisa has done a lot more thinking about the Nuclear Ban treaty, so I’ll defer to her on that.
Jeffrey Ohl is going to look into this question as part of a summer project for the Stanford Existential Risk Initiative, and I’ll be mentoring him. We literally just started talking about the project last week, so more to come on that! For now, I’ll say the characteristics of the technology itself that you are trying to regulate (e.g. fissile material vs. inputs to the creating of AGI for instance) is very important in terms of how a successful treaty could be constructed. This aspect is important in terms of the mechanisms that must be put in place to verify compliance with the treaty. I co-wrote an article with Chris Bakerlee on engineered pathogens for Vox a few years ago that discusses, among other things, some of the challenges around regulating biotechnology that make verifying compliance with the Biological Weapons Convention difficult https://www.vox.com/future-perfect/2018/12/6/18127430/superbugs-biotech-pathogens-biorisk-pandemic
What time range are you looking at when it comes to forecasts, and what sort of things do you have in mind? I recall when I read Superforecasting a few years ago that forecasts aren’t particularly reliable beyond a few years even for Superforecasters (though correct me if I’m wrong/maybe views on that are different now than they were then?). These treaties operate on pretty long time-scales… e.g. the NPT was conceived of in the mid 1960s, it was signed in 1968, it went into force in 1970, and then countries joined over the course of a few decades. https://en.wikipedia.org/wiki/List_of_parties_to_the_Treaty_on_the_Non-Proliferation_of_Nuclear_Weapons
Thanks for this response!
I’ll be quite interested to learn more about Jeffrey’s project once it’s further along. I might reach out to you or Jeffrey in a few weeks about that.
Regarding the forecasts, we can have any time range and any topic. I already have a bunch of ideas, but just wanted to see if anything bubbled to mind for you independently which I could add to my list. (It’s ok if not!)
I guess this might depend what you mean by “particularly reliable”.
My understanding is that there’s basically just no good evidence either way regarding how accurate and calibrated forecasts are over long time-scales (at least if we restrict ourselves to relevant kinds of forecasts, e.g. ones made by people who seem to have been genuinely trying rather than just making claims for rhetorical/political effect). But there’s a little evidence (from Tetlock) to suggest that accuracy may decline relatively slowly after the first year or so. See in particular the great post How Feasible Is Long-range Forecasting?, footnote 17 there, and posts tagged long-range forecasting. Here’s the summary of that post: