I always appreciate your newsletter, and agree with your grim assessment of prediction markets’ long-suffering history. Here is what I am left wondering after reading this edition:
Okay, so the USA has mostly dropped the ball on this for forty years. But what about every other country? China seems pretty ambitious and willing to make things happen in order to secure their place on the world stage—where is the CCP-subsidized market hive-mind driving all the crucial central planning decisions? Well, maybe a prediction market doesn’t play well with wanting to exert lots of top-down control and suppress free speech. Okay, what about countries in Europe? What about Taiwan or Singapore? Nobody has yet achieved some kind of Hansonian utopia, so what is the limiting factor?
Maybe there is a strong correlation between different countries’ policy decisions caused by a ‘global elite culture’ that picks similar policies everywhere, and occasionally screws up by all deciding to ignore the potential of nuclear power, all choosing the same solutions to covid-19, etc. This is Hanson’s idea; I find it a bit conspiratorial compared to my framing in the next point. (But if it’s true, what are we to do? Perhaps try to build up EA/rationalism as a movement until we can influence global elite culture for the better? Or maybe become advisers to potential outliers from global elite culture, like the Saudi Arabian monarchy or something? Or somehow construct whole alternative institutions using stuff like crypto and charter cities, “Atlas Shrugged” style?)
Maybe it’s less about ‘elite’ culture and more about universal human biases. Perhaps prediction markets are abhorrent to normal folks insofar as they offend traditional status hierarchies (by holding people too closely to their word and showing the hypocrisy of leaders, or something), and that causes them to be rejected wherever they are tried. That might sound nutty, but personally I think a big part of why healthcare systems are so complicated and expensive is because so many reform ideas that look great on paper run into universal deeply-engrained cultural preferences—for instance, the taboo tradeoff between money & lives prevents healthcare systems from making price information too obvious or making inequalities in care quality too visible. Similarly, the human desire to show that we care causes us to overspend late in life when help would not do much good, and underspend on some cheap preventative things (like doing more to push people towards better exercise/diet/etc). If this is true for prediction markets, what do we do? Maybe it suggests that we should start building the prediction-market future starting from stock markets (via services like Kalshi), since stock markets are (grudgingly!) tolerated by human culture, rather than trying to persuade governments or corporations to adopt them (where it is too easy for a leader to veto it based on their gut opposition).
Maybe it’s misleading to frame the question this way, as “why is EVERY COUNTRY failing in the SAME WAY”, because most prediction market advocates have all been inside the USA/anglosphere, so other countries haven’t really had a fair shot at being persuaded? (In this case, maybe all that’s necessary is to fund some prediction-market advocacy groups in Taiwan, Singapore, India, Dubai, South Korea, and other diverse locations until somebody finally takes the offer! Then, once one country is doing it, that will make it easier for the innovation to spread elsewhere.)
Maybe there’s no special explanation, and every nation is just failing for its own distinct reason, just because governments aren’t that competent and reform is hard and the possibility space of failure is much larger than the small target of success. Countries make dumb decisions all the time and are constantly leaving large amounts of potential economic growth on the table, just because life is tough and it’s hard to make good decisions. Null hypothesis! In which case we just need to try harder and then our dreams might come true (even here in the USA, despite the grim history of defeats). This hypothesis gets stronger when you consider that the existing community of prediction-market advocacy is quite small and there is not much funding in it.
Maybe prediction markets are somehow not ready for prime-time, lacking some crucial feature that would speed adoption and retrospectively seem like an obvious part of the complete package? (In this case, the challenge is to figure out what feature to add or how to otherwise tweak the design to get product/market fit, and then it’s off to the races. Hanson has often identified “the difficulty of finding a customer willing to pay for info” as the most difficult piece of the puzzle. Hence the appeal of corporate prediction markets more usable, or of persuading governments to subsidize prediction markets on topics of public interest. This reasoning also lines up with your complaint about how, without a customer willing to subsidize the market, markets gravitate towards covering meaningless emotional topics like sports, where people are irrational.) Maybe we need to make prediction markets easier to use so they can be adopted more easily by corporate customers (my “gitlab of prediction markets” idea), or maybe we need to increase liquidity of markets by lowering fees and doing things like parking the invested funds in the S&P 500 so that prediction stops being a zero-sum game. Maybe we just need to figure out better and better ways around the regulations banning prediction markets. Et cetera.
Of course I would be eager to hear your thoughts on what the key limiting factor(s) might be.
In your last newsletter, you remarked, “It’s kind of interesting how $40k [given away by Astral Codex Ten Grants] feels like a significant quantity of all the funding there is for small experiments in the forecasting space. This is probably suboptimal.” What prediction-market experiments would you be most interested to see run?
Do you think that advocacy/lobbying for prediction markets (perhaps in other countries as mentioned) is a worthwhile endeavor? Or do you think that would be less effective than experimenting with different market designs?
Robin Hanson wanted to run a “Fire the CEO” conditional prediction market about companies’ stock price if the CEO did / did not resign by the end of the quarter. I guess the plan would be to initially subsidize the market yourself, then prove the worth of the idea, then run a service where companies pay you to be included among your many company-specific markets. Do you think this is a promising plan? Is money the biggest roadblock, or would it be illegal to offer this service in the USA / without the companies’ permission / etc? (Could we just run it in the UK or something?)
FTX is a huge crypto exchange, they’ve already done prediction markets on the past for presidential elections, and they’re apparently EA-aligned. Could we ask them if they’d please run other prediction markets that we think would be useful, such as a conditional prediction market about the stock market’s performance conditional on a presidential election outcome, or a prediction market about some high-minded EA-relevant topic?
There is kind of a difference between “studying forecasting to benefit EA” (which seems to describe many of your projects at Quantified Uncertainty Research Institute, such as creating software to help grantmakers estimate impact), versus “prediction markets as an EA cause area” (under the heading of “improving institutional decision-making”, progress studies, and improving “civilizational adequacy”). How do you feel about the relationship between the two, and which side do you tend to feel is the better place to focus effort in terms of impact?
This is a lot of questions, so no pressure to respond to everything—I mostly intend this as food for thought. Also, I wrote this post casually, but let me know if you think it would be good to rework as a top-level post (ie, if you think these are good questions that people should be thinking more about).
Seconding Nuño’s assessment that this comment is awesome. While waiting for his response I’ll butt in with some quick off-the-cuff takes of my own.
On why no countries use prediction markets / forecasting to make crucial decisions:
My first reaction is “idk, but your comment already provides a really great breakdown of options that I would be excited to be turned into a top-level post.”
More work put into creating impactful questions, e.g. via identifying forecastable cruxes in key EA debates and integrating with ongoing EA-aligned research.
Better incentives for deep, collaborative predictions on these impactful questions.
The question you ask about “studying forecasting to benefit EA” vs. “prediction markets as an EA cause area” is also important. I’m inclined to favor interventions closer to “studying forecasting to benefit EA” at present (though I might frame it more as “improve EA’s wisdom/decision-making via various means including forecasting”, h/t QURI/Ozzie for influence here) because I feel we’re a relatively young and growing movement with a lot of resources (money + people) to use and not much clarity on how to do it best. Once we get better at this ourselves, e.g. forecasting platforms and prediction markets have clearly substantially improved important EA decisions, I’d feel it’s more time for “prediction markets/forecasting platforms improving non-EA’s decision-making” to be an EA cause area. I’m open to changing my mind on this e.g. if I see more evidence of forecasting having already improved important EA decisions. https://forum.effectivealtruism.org/posts/Ds2PCjKgztXtQrqAF/disentangling-improving-institutional-decision-making-2 and https://forum.effectivealtruism.org/posts/YpaQcARgLHFNBgyGa/prioritization-research-for-advancing-wisdom-and are good pointers on this overall question as well.
Hypotheses for the global lack of adoption of prediction markets/probabilistic methods
From the hypothesis you outline, the ones that sound the most plausible, or like they hit more the nail in the head, are:
Null hypothesis! Governments aren’t that competent. I have some thoughts on how “strong optimizers”, e.g., a Machiavelli, a Bismark, just aren’t that common, and are becoming less common.
We can see this happening in Britain, where Dominic Cummings pushed for prediction markets/forecasting tournaments for governmental decision-making, and this got translated into a totally inoffensive forecasting platform with totally milquetoast questions which don’t affect decisions.
Prediction markets not being ready for prime-time
But, to some extent, all your hypothesis have something to it. It’s also not clear how one would go about differentiating between the different hypotheses. “Good judgment”, sure, but we still don’t really have the tools for thought to be reliably able to distinguish between these kinds of hypothesis, and I dislike punditry.
Prediction market experiments and other cool things
In your last newsletter, you remarked, “It’s kind of interesting how $40k [given away by Astral Codex Ten Grants] feels like a significant quantity of all the funding there is for small experiments in the forecasting space. This is probably suboptimal.” What prediction-market experiments would you be most interested to see run?
More on this to come in the next edition of the newsletter!
Yes, but I haven’t really done the math to compare to other interventions
Ditto. Also, last time I checked I think Hanson was still excited about it. I guess I’d be more excited about, e.g., prediction markets on topics of great importance to OpenPhil/EA, but that might be a bit myopic (?)
Yes, I just asked that yesterday to their head of engineering, and he seemed pretty receptive. No stock markets, though, and they still have to get their respective licenses at least on the US.
I think that description of QURI is too much of a simplification, I’d make some emphasis on software that is scalable (guesstimate, metaforecast), and on research that is more exploratory. That said, I’m betting heavily on the QURI side of things, though I do keep an eye on forecasting/prediction markets.
More Wagner
I wrote this post casually, but let me know if you think it would be good to rework as a top-level post
Yes, but I’d be more interested in getting something more comprehensive/structured; these points seem pretty scattered. To be clear, I do greatly prefer scattered thoughts over nothing.
There is (generally) a disconnect between decision makers and forecasting platforms
Spot forecasts are not especially useful on their own
There are some good examples of decision makers at least looking at markets
Re 1: the disconnect between decision makers and forecasting platforms. I think the problem comes in two directions.
Decision makers don’t value the forecasts as much as they would cost to create (even if the value they would provide would be huge)
The incentives to make the forecasts are usually orthogonal to the people using them. (Prediction markets seem to most naturally arise from investing and gambling) which isn’t necessarily a strong enough incentive in and of itself. My understanding of the gambling industry is that most people are interested in short-term, volatile markets. (The boom in in-play sport vs pre-match odds; or Polymarket’s success in markets which are generally <1 month). Investing is a little different (and as I’ll say latter) I think people do take those forecasts fairly seriously.
Re 2: someone saying “X has a y% chance of happening” is not (usually) especially valuable to a decision maker. (Especially since the market is already accounting for what it expects the decision maker to do). Models (even fairly poor ones) often have more use to a decision maker, since they can see how their decision might affect the outcome. [Yes, there are ideas like counterfactual markets, but none of those ideas can really capture the full space of possibilities and will also just fragment liquidity]. The best you can really do is extract a model statistically (when indicator goes up, forecast goes down, so indicator might be saying something about event).
Re 3: It would take a while for me to summarise the evidence here, but I think there’s a pretty strong case that central banks (eg the Federal Reserve in the US) are increasingly looking at market indicators when setting monetary policy. I think CEOs and other decision makers in business look at market prices as indicators when deciding direction of their companes. (Although it’s hard to fully describe this as a prediction market as much as “looking at the competition” I think with some time I could articulate what I mean)
I always appreciate your newsletter, and agree with your grim assessment of prediction markets’ long-suffering history. Here is what I am left wondering after reading this edition:
Okay, so the USA has mostly dropped the ball on this for forty years. But what about every other country? China seems pretty ambitious and willing to make things happen in order to secure their place on the world stage—where is the CCP-subsidized market hive-mind driving all the crucial central planning decisions? Well, maybe a prediction market doesn’t play well with wanting to exert lots of top-down control and suppress free speech. Okay, what about countries in Europe? What about Taiwan or Singapore? Nobody has yet achieved some kind of Hansonian utopia, so what is the limiting factor?
Maybe there is a strong correlation between different countries’ policy decisions caused by a ‘global elite culture’ that picks similar policies everywhere, and occasionally screws up by all deciding to ignore the potential of nuclear power, all choosing the same solutions to covid-19, etc. This is Hanson’s idea; I find it a bit conspiratorial compared to my framing in the next point. (But if it’s true, what are we to do? Perhaps try to build up EA/rationalism as a movement until we can influence global elite culture for the better? Or maybe become advisers to potential outliers from global elite culture, like the Saudi Arabian monarchy or something? Or somehow construct whole alternative institutions using stuff like crypto and charter cities, “Atlas Shrugged” style?)
Maybe it’s less about ‘elite’ culture and more about universal human biases. Perhaps prediction markets are abhorrent to normal folks insofar as they offend traditional status hierarchies (by holding people too closely to their word and showing the hypocrisy of leaders, or something), and that causes them to be rejected wherever they are tried. That might sound nutty, but personally I think a big part of why healthcare systems are so complicated and expensive is because so many reform ideas that look great on paper run into universal deeply-engrained cultural preferences—for instance, the taboo tradeoff between money & lives prevents healthcare systems from making price information too obvious or making inequalities in care quality too visible. Similarly, the human desire to show that we care causes us to overspend late in life when help would not do much good, and underspend on some cheap preventative things (like doing more to push people towards better exercise/diet/etc). If this is true for prediction markets, what do we do? Maybe it suggests that we should start building the prediction-market future starting from stock markets (via services like Kalshi), since stock markets are (grudgingly!) tolerated by human culture, rather than trying to persuade governments or corporations to adopt them (where it is too easy for a leader to veto it based on their gut opposition).
Maybe it’s misleading to frame the question this way, as “why is EVERY COUNTRY failing in the SAME WAY”, because most prediction market advocates have all been inside the USA/anglosphere, so other countries haven’t really had a fair shot at being persuaded? (In this case, maybe all that’s necessary is to fund some prediction-market advocacy groups in Taiwan, Singapore, India, Dubai, South Korea, and other diverse locations until somebody finally takes the offer! Then, once one country is doing it, that will make it easier for the innovation to spread elsewhere.)
Maybe there’s no special explanation, and every nation is just failing for its own distinct reason, just because governments aren’t that competent and reform is hard and the possibility space of failure is much larger than the small target of success. Countries make dumb decisions all the time and are constantly leaving large amounts of potential economic growth on the table, just because life is tough and it’s hard to make good decisions. Null hypothesis! In which case we just need to try harder and then our dreams might come true (even here in the USA, despite the grim history of defeats). This hypothesis gets stronger when you consider that the existing community of prediction-market advocacy is quite small and there is not much funding in it.
Maybe prediction markets are somehow not ready for prime-time, lacking some crucial feature that would speed adoption and retrospectively seem like an obvious part of the complete package? (In this case, the challenge is to figure out what feature to add or how to otherwise tweak the design to get product/market fit, and then it’s off to the races. Hanson has often identified “the difficulty of finding a customer willing to pay for info” as the most difficult piece of the puzzle. Hence the appeal of corporate prediction markets more usable, or of persuading governments to subsidize prediction markets on topics of public interest. This reasoning also lines up with your complaint about how, without a customer willing to subsidize the market, markets gravitate towards covering meaningless emotional topics like sports, where people are irrational.) Maybe we need to make prediction markets easier to use so they can be adopted more easily by corporate customers (my “gitlab of prediction markets” idea), or maybe we need to increase liquidity of markets by lowering fees and doing things like parking the invested funds in the S&P 500 so that prediction stops being a zero-sum game. Maybe we just need to figure out better and better ways around the regulations banning prediction markets. Et cetera.
Of course I would be eager to hear your thoughts on what the key limiting factor(s) might be.
In your last newsletter, you remarked, “It’s kind of interesting how $40k [given away by Astral Codex Ten Grants] feels like a significant quantity of all the funding there is for small experiments in the forecasting space. This is probably suboptimal.” What prediction-market experiments would you be most interested to see run?
Do you think that advocacy/lobbying for prediction markets (perhaps in other countries as mentioned) is a worthwhile endeavor? Or do you think that would be less effective than experimenting with different market designs?
Robin Hanson wanted to run a “Fire the CEO” conditional prediction market about companies’ stock price if the CEO did / did not resign by the end of the quarter. I guess the plan would be to initially subsidize the market yourself, then prove the worth of the idea, then run a service where companies pay you to be included among your many company-specific markets. Do you think this is a promising plan? Is money the biggest roadblock, or would it be illegal to offer this service in the USA / without the companies’ permission / etc? (Could we just run it in the UK or something?)
FTX is a huge crypto exchange, they’ve already done prediction markets on the past for presidential elections, and they’re apparently EA-aligned. Could we ask them if they’d please run other prediction markets that we think would be useful, such as a conditional prediction market about the stock market’s performance conditional on a presidential election outcome, or a prediction market about some high-minded EA-relevant topic?
There is kind of a difference between “studying forecasting to benefit EA” (which seems to describe many of your projects at Quantified Uncertainty Research Institute, such as creating software to help grantmakers estimate impact), versus “prediction markets as an EA cause area” (under the heading of “improving institutional decision-making”, progress studies, and improving “civilizational adequacy”). How do you feel about the relationship between the two, and which side do you tend to feel is the better place to focus effort in terms of impact?
This is a lot of questions, so no pressure to respond to everything—I mostly intend this as food for thought. Also, I wrote this post casually, but let me know if you think it would be good to rework as a top-level post (ie, if you think these are good questions that people should be thinking more about).
Seconding Nuño’s assessment that this comment is awesome. While waiting for his response I’ll butt in with some quick off-the-cuff takes of my own.
On why no countries use prediction markets / forecasting to make crucial decisions:
My first reaction is “idk, but your comment already provides a really great breakdown of options that I would be excited to be turned into a top-level post.”
If I had to guess I think it’s some combination of universal human biases and fundamental issues with the value of prediction markets at present. On human biases, it seems like many people have a distaste for markets on important topics and quantified forecasting is an unnatural-feeling activity for many people to partake in. On fundamental issues, I’d refer to https://forum.effectivealtruism.org/posts/dQhjwHA7LhfE8YpYF/prediction-markets-in-the-corporate-setting which you mentioned and https://forum.effectivealtruism.org/posts/E4QnGsXLEEcNysADT/issues-with-futarchy for ideas.
On what things I’d like to see done overall, I’d point to the solution ideas and conclusion section of my post https://forum.effectivealtruism.org/posts/S2vfrZsFHn7Wy4ocm/bottlenecks-to-more-impactful-crowd-forecasting-2. In particular:
More work put into creating impactful questions, e.g. via identifying forecastable cruxes in key EA debates and integrating with ongoing EA-aligned research.
Better incentives for deep, collaborative predictions on these impactful questions.
The question you ask about “studying forecasting to benefit EA” vs. “prediction markets as an EA cause area” is also important. I’m inclined to favor interventions closer to “studying forecasting to benefit EA” at present (though I might frame it more as “improve EA’s wisdom/decision-making via various means including forecasting”, h/t QURI/Ozzie for influence here) because I feel we’re a relatively young and growing movement with a lot of resources (money + people) to use and not much clarity on how to do it best. Once we get better at this ourselves, e.g. forecasting platforms and prediction markets have clearly substantially improved important EA decisions, I’d feel it’s more time for “prediction markets/forecasting platforms improving non-EA’s decision-making” to be an EA cause area. I’m open to changing my mind on this e.g. if I see more evidence of forecasting having already improved important EA decisions. https://forum.effectivealtruism.org/posts/Ds2PCjKgztXtQrqAF/disentangling-improving-institutional-decision-making-2 and https://forum.effectivealtruism.org/posts/YpaQcARgLHFNBgyGa/prioritization-research-for-advancing-wisdom-and are good pointers on this overall question as well.
Hypotheses for the global lack of adoption of prediction markets/probabilistic methods
From the hypothesis you outline, the ones that sound the most plausible, or like they hit more the nail in the head, are:
Null hypothesis! Governments aren’t that competent. I have some thoughts on how “strong optimizers”, e.g., a Machiavelli, a Bismark, just aren’t that common, and are becoming less common.
We can see this happening in Britain, where Dominic Cummings pushed for prediction markets/forecasting tournaments for governmental decision-making, and this got translated into a totally inoffensive forecasting platform with totally milquetoast questions which don’t affect decisions.
Prediction markets not being ready for prime-time
But, to some extent, all your hypothesis have something to it. It’s also not clear how one would go about differentiating between the different hypotheses. “Good judgment”, sure, but we still don’t really have the tools for thought to be reliably able to distinguish between these kinds of hypothesis, and I dislike punditry.
Prediction market experiments and other cool things
More on this to come in the next edition of the newsletter!
Yes, but I haven’t really done the math to compare to other interventions
Ditto. Also, last time I checked I think Hanson was still excited about it. I guess I’d be more excited about, e.g., prediction markets on topics of great importance to OpenPhil/EA, but that might be a bit myopic (?)
Yes, I just asked that yesterday to their head of engineering, and he seemed pretty receptive. No stock markets, though, and they still have to get their respective licenses at least on the US.
I think that description of QURI is too much of a simplification, I’d make some emphasis on software that is scalable (guesstimate, metaforecast), and on research that is more exploratory. That said, I’m betting heavily on the QURI side of things, though I do keep an eye on forecasting/prediction markets.
More Wagner
Yes, but I’d be more interested in getting something more comprehensive/structured; these points seem pretty scattered. To be clear, I do greatly prefer scattered thoughts over nothing.
This comment is glorious. I’ll take some time to answer, though.
My general take on this space is:
There is (generally) a disconnect between decision makers and forecasting platforms
Spot forecasts are not especially useful on their own
There are some good examples of decision makers at least looking at markets
Re 1: the disconnect between decision makers and forecasting platforms. I think the problem comes in two directions.
Decision makers don’t value the forecasts as much as they would cost to create (even if the value they would provide would be huge)
The incentives to make the forecasts are usually orthogonal to the people using them. (Prediction markets seem to most naturally arise from investing and gambling) which isn’t necessarily a strong enough incentive in and of itself. My understanding of the gambling industry is that most people are interested in short-term, volatile markets. (The boom in in-play sport vs pre-match odds; or Polymarket’s success in markets which are generally <1 month). Investing is a little different (and as I’ll say latter) I think people do take those forecasts fairly seriously.
Re 2: someone saying “X has a y% chance of happening” is not (usually) especially valuable to a decision maker. (Especially since the market is already accounting for what it expects the decision maker to do). Models (even fairly poor ones) often have more use to a decision maker, since they can see how their decision might affect the outcome. [Yes, there are ideas like counterfactual markets, but none of those ideas can really capture the full space of possibilities and will also just fragment liquidity]. The best you can really do is extract a model statistically (when indicator goes up, forecast goes down, so indicator might be saying something about event).
Re 3: It would take a while for me to summarise the evidence here, but I think there’s a pretty strong case that central banks (eg the Federal Reserve in the US) are increasingly looking at market indicators when setting monetary policy. I think CEOs and other decision makers in business look at market prices as indicators when deciding direction of their companes. (Although it’s hard to fully describe this as a prediction market as much as “looking at the competition” I think with some time I could articulate what I mean)