The wisdom of crowds effect kicks in with very few forecasts. In the working paper I cite elsewhere in the comments, even 5 forecasts gets you pretty far along into the WoC effect, and 10 even more so. This is for asking people what they think, not prediction markets—the latter should, theoretically, require more forecasts, since seeing the implicit beliefs of others through the market price could lead to herding etc. But the wisdom of crowds effect kicking in for very small N is well established in the literature.
I have a different takeaway as you, though, that we only know about this effect—or about the biases people have and how to adjust their forecasts—because of work on forecasting. I don’t know how we’d know this stylized fact without work on it. For the wisdom of the crowds effect specifically, perhaps you could stop funding early since that one is well known, but it’s sufficiently surprising to most people that there could be value in showing it for more domains, and it is really just one example of what we learn more generally from research on forecasting—and these other results on how to optimally weigh forecasts can shrink error much more even after taking the wisdom of the crowds effect into consideration. (In our work, WoC gets you a ~60% reduction in the MSE, but other small adjustments lead to an improvement of an additional ~60% reduction in error compared to the WoC estimate, and those aren’t even all the improvements we can make.)
Today I would never run an experiment without using forecasts to help with power calculations. And there is very recent work I’d use to adjust those forecasts, and we’re collectively not near the optimum in terms of learning what we can learn to make more accurate forecasts or integrating them into workflows. As I said elsewhere in the comments, the claims in the OP are far too strong. Even your asking a few experts—that’s something that could be improved on and integrated into workflows and is part of the titular “forecasting”. (It reads to me kind of like: don’t do forecasting, do this other thing which is itself forecasting and is informed by and improved upon by… forecasting.)
A more defensible claim imo would be that there are some projects that are self-supporting and those should not be funded, or that in some but not all cases if the market doesn’t pay for it then it’s not valuable (abstracting from coordination failures and other market failures, or the externalities of basic research).
I think “your mileage may vary” quite a lot in this. In the context of the social science prediction market, you tend to be asking people who have expertise and familiarity with the methods and context, and sometimes more experienced than the people posting the questions.
On the other hand, if you post detailed technical questions on a mainstream prediction market or even on Metaculus, I expect / have the sense that you don’t get much of this’ wisdom of the crowds dividend’.
The wisdom of crowds effect kicks in with very few forecasts. In the working paper I cite elsewhere in the comments, even 5 forecasts gets you pretty far along into the WoC effect, and 10 even more so. This is for asking people what they think, not prediction markets—the latter should, theoretically, require more forecasts, since seeing the implicit beliefs of others through the market price could lead to herding etc. But the wisdom of crowds effect kicking in for very small N is well established in the literature.
I have a different takeaway as you, though, that we only know about this effect—or about the biases people have and how to adjust their forecasts—because of work on forecasting. I don’t know how we’d know this stylized fact without work on it. For the wisdom of the crowds effect specifically, perhaps you could stop funding early since that one is well known, but it’s sufficiently surprising to most people that there could be value in showing it for more domains, and it is really just one example of what we learn more generally from research on forecasting—and these other results on how to optimally weigh forecasts can shrink error much more even after taking the wisdom of the crowds effect into consideration. (In our work, WoC gets you a ~60% reduction in the MSE, but other small adjustments lead to an improvement of an additional ~60% reduction in error compared to the WoC estimate, and those aren’t even all the improvements we can make.)
Today I would never run an experiment without using forecasts to help with power calculations. And there is very recent work I’d use to adjust those forecasts, and we’re collectively not near the optimum in terms of learning what we can learn to make more accurate forecasts or integrating them into workflows. As I said elsewhere in the comments, the claims in the OP are far too strong. Even your asking a few experts—that’s something that could be improved on and integrated into workflows and is part of the titular “forecasting”. (It reads to me kind of like: don’t do forecasting, do this other thing which is itself forecasting and is informed by and improved upon by… forecasting.)
A more defensible claim imo would be that there are some projects that are self-supporting and those should not be funded, or that in some but not all cases if the market doesn’t pay for it then it’s not valuable (abstracting from coordination failures and other market failures, or the externalities of basic research).
I think “your mileage may vary” quite a lot in this. In the context of the social science prediction market, you tend to be asking people who have expertise and familiarity with the methods and context, and sometimes more experienced than the people posting the questions.
On the other hand, if you post detailed technical questions on a mainstream prediction market or even on Metaculus, I expect / have the sense that you don’t get much of this’ wisdom of the crowds dividend’.