For context: I founded Inkling Markets in 2006 out of Y Combinator running internal prediction markets for companies and governments, and then Cultivate Labs in 2014, which has participated in some of the projects this community has funded. So I’ve watched this play out for 20 years. Before IARPA ACE and Tetlock’s Superforecasters, before FRI, before most of what’s mentioned in this thread existed.
Two things I think this debate is missing:
1) On whether the OP/CG money was wasted. Several commenters imply specific grants were boondoggles and this was recently mentioned in Nuño’s forecasting newsletter as well. The stated goal of much of that funding was to influence decision-making inside governments, particularly the US Government.
Anyone who has actually tried this knows it’s an extremely expensive and difficult endeavor. For example, just getting in the door to talk to the people who have budgets to spend requires former senior officials on your team to make introductions. These are some of the highest demand people in Washington because of their networks. Then if you get through the door and eventually get to yes, the procurement and contracting takes months or doesn’t end up even being possible because of incompatible contract “vehicles.” An $8M grant sounds like a lot until you price out what it actually costs to embed forecasting infrastructure into a federal agency’s workflow. You can fairly argue the bets didn’t pay off, or that an aspect of the strategy was wrong. But I can assure you the money wasn’t being burned on team dinners.
2) The deeper problem is one almost no one is funding. This morning while writing this post, I pulled the data. Inkling, and then Cultivate Labs, has worked with 150+ companies, government agencies, and think tanks. Not a single engagement has been canceled because the software was inadequate, our team sucked, or the predictions weren’t accurate enough. Not one. They were canceled because we couldn’t get enough employees to participate, or because the predictions, accurate ones, weren’t getting used. Senior leaders didn’t want to listen and outputs weren’t integrated into decision processes. In those instances, there was limited or no demand signal, so why should an employee/forecaster care either?
This wasn’t a UX problem, a mechanism problem, or a “need for better epistemics” problem. It was culture and politics and remains so to this day. Management has one eye on company performance and one eye on their own position. If a forecast supports what they want to do, great. If it doesn’t, it’s dead to them. I’ve watched organizations ignore strikingly accurate forecasts about product demand, product launch timelines, budget overshoot, future customer service failures, competitor behavior, results on future employee satisfaction surveys, not because the forecasts were not calibrated or the Brier score was poor, but because acting on them would have required someone to a) admit they were wrong, b) surrender authority to a process that didn’t flatter them or their leadership, or c) take away a modicum of control over decision making.
So when Marcus says he can’t point to “this forecasting happened, and now we have made demonstrably better decisions,” he’s mostly right, but the diagnosis is wrong. The forecasts existed in 150+ organizations that I personally know of, and I’m sure there are more. Many of them some of the largest, most profitable companies in the world. The decisions didn’t change because organizations aren’t built to be changed by forecasts, even today.
What I’ve increasingly come to believe, and what has shaped the work my team has been pivoting to the past couple years now, is that forecasting is most useful when it isn’t the deliverable itself. Now we’re embedding live forecasts inside a broader analytic framing and decisionmakers are engaging with it because it matches how they already think. A 63% probability on a narrow question doesn’t really survive contact with anyone in leadership. But the same forecast as context inside a broader narrative does, because senior people don’t navigate the world at the granularity of point estimates. They navigate through a small set of plausible futures with a few pivotal factors in mind, and decision points they feel they need to make. I’m not claiming forecasting as context vs. deliverable solves the incentive problem. It doesn’t fully. But if done correctly and with the involvement of others, it raises the political cost and visibility of simply ignoring forecasts, and instead brings needed rigor to a decision making process—which I think was the point of believing in forecasting in the first place.
Ultimately, I think grant-makers have been over-indexing on the easier problem unfortunately. Funding more accuracy, more platforms, more benchmarks, more tournaments...That work is tractable, measurable, and produces clean deliverables, which is part of why I assume it keeps getting funded. But that should be the domain of academia. The harder problem, and the much more important problem IMO for the practitioners who are actually trying to affect change, is the last mile: how do forecasts get embedded into decision processes that have every cultural and political incentive to ignore them? How do we create alternative management models and pre-commitment mechanisms where leaders agree in advance how a forecast will move a decision? Where are the paid studies of the rare organizations that did truly integrate forecasting and what made them different? Where is the research on forecast-as-context vs. forecast-as-deliverable and how to optimize such a system?
For context: I founded Inkling Markets in 2006 out of Y Combinator running internal prediction markets for companies and governments, and then Cultivate Labs in 2014, which has participated in some of the projects this community has funded. So I’ve watched this play out for 20 years. Before IARPA ACE and Tetlock’s Superforecasters, before FRI, before most of what’s mentioned in this thread existed.
Two things I think this debate is missing:
1) On whether the OP/CG money was wasted. Several commenters imply specific grants were boondoggles and this was recently mentioned in Nuño’s forecasting newsletter as well. The stated goal of much of that funding was to influence decision-making inside governments, particularly the US Government.
Anyone who has actually tried this knows it’s an extremely expensive and difficult endeavor. For example, just getting in the door to talk to the people who have budgets to spend requires former senior officials on your team to make introductions. These are some of the highest demand people in Washington because of their networks. Then if you get through the door and eventually get to yes, the procurement and contracting takes months or doesn’t end up even being possible because of incompatible contract “vehicles.” An $8M grant sounds like a lot until you price out what it actually costs to embed forecasting infrastructure into a federal agency’s workflow. You can fairly argue the bets didn’t pay off, or that an aspect of the strategy was wrong. But I can assure you the money wasn’t being burned on team dinners.
2) The deeper problem is one almost no one is funding. This morning while writing this post, I pulled the data. Inkling, and then Cultivate Labs, has worked with 150+ companies, government agencies, and think tanks. Not a single engagement has been canceled because the software was inadequate, our team sucked, or the predictions weren’t accurate enough. Not one. They were canceled because we couldn’t get enough employees to participate, or because the predictions, accurate ones, weren’t getting used. Senior leaders didn’t want to listen and outputs weren’t integrated into decision processes. In those instances, there was limited or no demand signal, so why should an employee/forecaster care either?
This wasn’t a UX problem, a mechanism problem, or a “need for better epistemics” problem. It was culture and politics and remains so to this day. Management has one eye on company performance and one eye on their own position. If a forecast supports what they want to do, great. If it doesn’t, it’s dead to them. I’ve watched organizations ignore strikingly accurate forecasts about product demand, product launch timelines, budget overshoot, future customer service failures, competitor behavior, results on future employee satisfaction surveys, not because the forecasts were not calibrated or the Brier score was poor, but because acting on them would have required someone to a) admit they were wrong, b) surrender authority to a process that didn’t flatter them or their leadership, or c) take away a modicum of control over decision making.
So when Marcus says he can’t point to “this forecasting happened, and now we have made demonstrably better decisions,” he’s mostly right, but the diagnosis is wrong. The forecasts existed in 150+ organizations that I personally know of, and I’m sure there are more. Many of them some of the largest, most profitable companies in the world. The decisions didn’t change because organizations aren’t built to be changed by forecasts, even today.
What I’ve increasingly come to believe, and what has shaped the work my team has been pivoting to the past couple years now, is that forecasting is most useful when it isn’t the deliverable itself. Now we’re embedding live forecasts inside a broader analytic framing and decisionmakers are engaging with it because it matches how they already think. A 63% probability on a narrow question doesn’t really survive contact with anyone in leadership. But the same forecast as context inside a broader narrative does, because senior people don’t navigate the world at the granularity of point estimates. They navigate through a small set of plausible futures with a few pivotal factors in mind, and decision points they feel they need to make. I’m not claiming forecasting as context vs. deliverable solves the incentive problem. It doesn’t fully. But if done correctly and with the involvement of others, it raises the political cost and visibility of simply ignoring forecasts, and instead brings needed rigor to a decision making process—which I think was the point of believing in forecasting in the first place.
Ultimately, I think grant-makers have been over-indexing on the easier problem unfortunately. Funding more accuracy, more platforms, more benchmarks, more tournaments...That work is tractable, measurable, and produces clean deliverables, which is part of why I assume it keeps getting funded. But that should be the domain of academia. The harder problem, and the much more important problem IMO for the practitioners who are actually trying to affect change, is the last mile: how do forecasts get embedded into decision processes that have every cultural and political incentive to ignore them? How do we create alternative management models and pre-commitment mechanisms where leaders agree in advance how a forecast will move a decision? Where are the paid studies of the rare organizations that did truly integrate forecasting and what made them different? Where is the research on forecast-as-context vs. forecast-as-deliverable and how to optimize such a system?
That’s where the marginal dollar should go.