I work at Founders Pledge, which has made many forecasting-related grants, some of them quite recently. Like Marcus, I’ve been fairly successful at forecasting — I am a so-called superforecaster a — but have a fair amount of skepticism. My views here are personal ones, not FP’s.
I have some agreements and disagreements with this post. The main point of agreement I have is with Marcus’ “vibe” here: I think forecasting’s apparent status and prominence among EAs outstrip either its prima facie promisingness or the to-date empirical support for its use.
I’m not sure that I agree that too much has been spent on forecasting, and I definitely don’t agree that enough time has passed that we’d know by now whether this work has been useful. We’re talking about a very short period of time here.
I think we’re at risk of conflating a bunch of different kinds of forecasting work:
Investments in calibration: Funding new techniques or experiments in more effective forecasting
Investments in diffusion: Broadly, attempts to “make forecasting a thing” by supporting e.g. new platforms
Investments in capacity: Attempts to propagate or institutionalize formal forecasting at influential institutions
Investments in public goods: Supporting institutions that do good forecasting and which the broader EA ecosystem finds useful.
I hope it’s clear that these are very different kinds of effort and should be considered differently promising. One fairly strongly held view I have is that further investments in precise calibration are probably not worthwhile: as far as I know, there are no consequential institutions that are able to usefully differentiate between the ways they’d respond to a 63% forecast vs a 65% one.
Finance, of course, could benefit from such an edge. But here’s where I find Marcus’ vibe most compelling: if this were really so useful at the moment, then good human forecasters would be much better-paid.
At this point, I think it’s critical to draw another distinction. In funding forecasting work, effective giving orgs are essentially trying to purchase an outcome. I think the best case for forecasting work is that we’re not trying to purchase well-calibrated forecasts but rather institutional forms that generate well-calibrated forecasts.
I investigated FP’s most recent large grant in this space, which seeded a forecasting practice at an international security-focused think tank. Almost everyone I spoke to for that investigation viewed good forecasts as something like an incidental side-effect of the process required to generate them: generating useful questions, formally surfacing critical disagreements, identifying critical paths, decomposing reasoning, generating anchors that can be updated as events progress, making individuals’ judgment intercomparable.
As forecasting proponents have been arguing for years, judgmental forecasting is something like “institutionalized good judgment” — Brier scores are sort of like an OKR for org epistemics. And if you talk to people at the kinds of institutions where EAs are enthusiastic to see forecasting implemented, you’ll find either (a) an eagerness to see these kinds of norms and guardrails put in place or (b) an epistemic posture that makes the need for these guardrails self-evident.
My overall feeling here is one of sympathy for Marcus’ view: I think there is a there there, but I agree that EAs’ native enthusiasm for this kind of work has outrun our rigorous thinking about its usefulness, and I think we could probably use more discipline in that regard.
I work at Founders Pledge, which has made many forecasting-related grants, some of them quite recently. Like Marcus, I’ve been fairly successful at forecasting — I am a so-called superforecaster a — but have a fair amount of skepticism. My views here are personal ones, not FP’s.
I have some agreements and disagreements with this post. The main point of agreement I have is with Marcus’ “vibe” here: I think forecasting’s apparent status and prominence among EAs outstrip either its prima facie promisingness or the to-date empirical support for its use.
I’m not sure that I agree that too much has been spent on forecasting, and I definitely don’t agree that enough time has passed that we’d know by now whether this work has been useful. We’re talking about a very short period of time here.
I think we’re at risk of conflating a bunch of different kinds of forecasting work:
Investments in calibration: Funding new techniques or experiments in more effective forecasting
Investments in diffusion: Broadly, attempts to “make forecasting a thing” by supporting e.g. new platforms
Investments in capacity: Attempts to propagate or institutionalize formal forecasting at influential institutions
Investments in public goods: Supporting institutions that do good forecasting and which the broader EA ecosystem finds useful.
I hope it’s clear that these are very different kinds of effort and should be considered differently promising. One fairly strongly held view I have is that further investments in precise calibration are probably not worthwhile: as far as I know, there are no consequential institutions that are able to usefully differentiate between the ways they’d respond to a 63% forecast vs a 65% one.
Finance, of course, could benefit from such an edge. But here’s where I find Marcus’ vibe most compelling: if this were really so useful at the moment, then good human forecasters would be much better-paid.
At this point, I think it’s critical to draw another distinction. In funding forecasting work, effective giving orgs are essentially trying to purchase an outcome. I think the best case for forecasting work is that we’re not trying to purchase well-calibrated forecasts but rather institutional forms that generate well-calibrated forecasts.
I investigated FP’s most recent large grant in this space, which seeded a forecasting practice at an international security-focused think tank. Almost everyone I spoke to for that investigation viewed good forecasts as something like an incidental side-effect of the process required to generate them: generating useful questions, formally surfacing critical disagreements, identifying critical paths, decomposing reasoning, generating anchors that can be updated as events progress, making individuals’ judgment intercomparable.
As forecasting proponents have been arguing for years, judgmental forecasting is something like “institutionalized good judgment” — Brier scores are sort of like an OKR for org epistemics. And if you talk to people at the kinds of institutions where EAs are enthusiastic to see forecasting implemented, you’ll find either (a) an eagerness to see these kinds of norms and guardrails put in place or (b) an epistemic posture that makes the need for these guardrails self-evident.
My overall feeling here is one of sympathy for Marcus’ view: I think there is a there there, but I agree that EAs’ native enthusiasm for this kind of work has outrun our rigorous thinking about its usefulness, and I think we could probably use more discipline in that regard.
I agree that there is a lot of stuff being conflated in “forecasting”. I suppose, I want to single out Prediction Markets and Judgemental Forecasting.