MichaelA🔸 comments on The NPT: Learning from a Longtermist Success [Links!]

[MichaelA🔸]

Thanks for this response!

I’ll be quite interested to learn more about Jeffrey’s project once it’s further along. I might reach out to you or Jeffrey in a few weeks about that.

Regarding the forecasts, we can have any time range and any topic. I already have a bunch of ideas, but just wanted to see if anything bubbled to mind for you independently which I could add to my list. (It’s ok if not!)

I recall when I read Superforecasting a few years ago that forecasts aren’t particularly reliable beyond a few years even for Superforecasters (though correct me if I’m wrong/​maybe views on that are different now than they were then?).

I guess this might depend what you mean by “particularly reliable”.

My understanding is that there’s basically just no good evidence either way regarding how accurate and calibrated forecasts are over long time-scales (at least if we restrict ourselves to relevant kinds of forecasts, e.g. ones made by people who seem to have been genuinely trying rather than just making claims for rhetorical/​political effect). But there’s a little evidence (from Tetlock) to suggest that accuracy may decline relatively slowly after the first year or so. See in particular the great post How Feasible Is Long-range Forecasting?, footnote 17 there, and posts tagged long-range forecasting. Here’s the summary of that post:

How accurate do long-range (≥10yr) forecasts tend to be, and how much should we rely on them?

As an initial exploration of this question, I sought to study the track record of long-range forecasting exercises from the past. Unfortunately, my key finding so far is that it is difficult to learn much of value from those exercises, for the following reasons:

1. Long-range forecasts are often stated too imprecisely to be judged for accuracy. [More]

2. Even if a forecast is stated precisely, it might be difficult to find the information needed to check the forecast for accuracy. [More]

3. Degrees of confidence for long-range forecasts are rarely quantified. [More]

4. In most cases, no comparison to a “baseline method” or “null model” is possible, which makes it difficult to assess how easy or difficult the original forecasts were. [More]

5. Incentives for forecaster accuracy are usually unclear or weak. [More]

6. Very few studies have been designed so as to allow confident inference about which factors contributed to forecasting accuracy. [More]

7. It’s difficult to know how comparable past forecasting exercises are to the forecasting we [at Open Phil] do for grantmaking purposes, e.g. because the forecasts we make are of a different type, and because the forecasting training and methods we use are different. [More]

We plan to continue to make long-range quantified forecasts about our work so that, in the long run, we might learn something about the feasibility of long-range forecasting, at least for our own case. [More]