I currently have an around 400-day streak on Manifold Markets (though lately I only spend a minute or two a day on it) and have no particular vendetta against it. I also use Metaculus. I’m reasonably well-ranked on both but have not been paid by either platform, ignoring a few Manifold donations. I have not attended any Manifest. I think Manifold has value as a weird form of social media, but I think it’s important to be clear that this is what it is, and not a manifestation of collective EA or rationalist consciousness, or an effective attempt to improve the world in its current form.
Overview of Manifold
Manifold is a prediction market website where people can put virtual money (called “mana”) into bets on outcomes. There are several key features of this: 1. You’re rewarded with virtual money both for participating and for predicting well, though you can also pay to get more. 2. You can spend this mana to ask questions, which you will generally vet and resolve yourself (allowing many more questions than on comparable sites). Moderators can reverse unjustified decisions but it’s usually self-governed. Until recently, you could also donate mana to real charities, though recently this stopped; now only a few “prize” questions provide a more exclusive currency that can be donated, and most questions produce unredeemable mana.
How might it claim to improve the world?
There are two ways in which Manifold could be improving the world. It could either make good predictions (which would be intrinsically valuable for improving policy or making wealth) or it could donate money to charities. Until recently, the latter looked quite reasonable: the company appeared to be rewarding predictive power with the ability to donate money to charities. The counterfactuality of these donations is questionable, however, since the money for it came from EA-aligned grants, and most of it goes to very mainstream EA charities. It has a revenue stream from people buying mana, but this is less than $10k/month, some of which isn’t really revenue (since it will ultimately be converted to donations), and presumably this doesn’t cover the staff costs. The founders appear to believe that eventually they will get paid enough money to run markets for other organisations, in which case the donations would be counterfactual. But this relies on the markets producing good predictions.
Sadly, Manifold does not produce particularly good predictions. In last year’s ACX contest, it performed worse than simply averaging predictions from the same number of people who took part in each market. Their calibration, while good by human standards, has a clear systematic bias towards predicting things will happen when they don’t (Yes bias). By contrast, rival firm Metaculus has no easily-corrected bias and seems to perform better at making predictions on the same questions (including in the ACX contest). Metaculus’ self-measured Brier score is 0.111, compared to Manifold’s 0.168 (lower is better, and this is quite a lot lower, though they are not answering all the same questions). Metaculus doesn’t publish the number of monthly active users like Manifold does, but the number of site visits they receive are comparable (slightlyhigher for Metaculus by one measure, lower by another), so it doesn’t seem like the prediction difference can be explained by user numbers alone.
Can the predictive power be improved?
Some of the problems with Manifold, like the systematic Yes bias, can be algorithmically fixed by potential users. Others are more intrinsic to the medium. Many questions resolve based on extensive discussions about exactly how to categorise reality, meaning that subtle clarifications by the author can result in huge swings in probability. The market mechanism of Manifold produces an information premium that incentivises people to act quickly on information, meaning that questions also swing wildly based on rumours.
The idea behind giving people mana for predictions is that wealth should accumulate with predictive prowess, resulting in people who have historically predicted better being able to weight their opinions more strongly. However in practice Manifold has many avenues for people incapable of making correct predictions to get Mana. It:
1. Wants bad predictors to pay money to keep playing
2. Hands out mana for asking engaging questions (which are not at all the same as useful questions)
3. Has many weird meta markets where people with lots of money can typically make more mana without predicting anything other than the spending of the superwealthy (“whalebait”)
4. Allows personal markets, where people can earn mana by doing tasks they set for themselves. This is an officially endorsed form of insider trading, insider trading being generally accepted.
5. Lets people gamble, go into negative equity, then burn their accounts and make new ones if bets go badly. The use of “puppet accounts” to manipulate markets in this or more complex ways is actively fought but still happens, and several of the all-time highest earners have transpired to use them to generate wealth.
6. Tends to leave markets open until after the answer to a question is publicly known and the answer is posted in the comments, so as well as large mana rewards for reading the news fast, simply reacting to posts on the site itself can produce a reliable profit
The first of these is entirely structural to their business model. While in principle the others could be “fixed” (if you consider them bugs) and some have lately become smaller (e.g. you can no longer bet things outside the 1-99% range, reducing profit from publicly known events), several have proven rather robust. This is quite apart from insider trading (which arguably makes the markets more accurate, even if it rewards people unfairly) and pump-and-dump schemes, which are hard to fix in any market system.
Framed as a fun pastime, having tricks to make mana without making predictions is fine, and this could drive up engagement. But these are all signs that the accuracy of the predictions (as opposed to news-reading) is not a highly ranked goal of the site. And it’s important to highlight the tradeoff between attracting idiots who will pay to play, as on gambling sites, and attracting companies who will pay to get good results. The assumption that idiots are distributed across opinions and so their biases will cancel each other out is ill-founded, particularly if a site has poor diversity.
By comparison, Metaculus simply applies an algorithm to weight people’s opinions based on past performance. There are obviously systematic differences between who uses which website, but a study indicated that under ideal conditions, a non-market approach to aggregating opinions was more successful than a prediction market. Prediction markets may be better than just doing a poll (for short-term predictions – Manifold can argue for its calibration over longer times via its loan system, which is plausible but this hasn’t been demonstrated), but worse than measures that can actually look at who is saying what. This would indicate that even without the quirky mana sources and time-dynamics, we wouldn’t expect Manifold to produce the best predictions. You can also see market failures explicitly in a handful of long-term markets (often about AI extinction risk, which cannot possibly pay out to humans) where a small number of very rich people have pegged the value of the market at the levels that they want it, in spite of this being essentially guaranteed to lose mana. Arguably this is the opposite criticism to the free-market criticisms—humans being willing to lose currency to do what they perceive as the right thing to highlight AI extinction risk—but it still represents an oligarchic market failure resulting in biased predictions. Unless prediction markets are really very large indeed and have no form of oligarchy, the neoliberal assumption that we can ignore who is putting up the currency is invalid.
This creates interesting possibilities to improve predictions. One, Manifold could have taxes to reduce opinion inequality. Two, Manifold could report the market values for predictions but (for a price) give you the better probabilities calculated by an appropriate algorithm, which is basically guaranteed to be better (it can always degenerate to “just use the market value” if that’s genuinely best). Why hasn’t it done this already? Well, there are probably several reasons, but one that I’m highlighting here is the link to neoliberal ideology. This is not to say that users of the site are all neoliberal – though they probably are more so than average. I mean that the “contrarian” ideologues present at the controversial Manifest24 and that the co-founders of manifold clearly find most interesting are consistently from a right-wing, market-trusting perspective, and this creates material blind spots. In spite of having many tens of talks on how to make predictions at this event, people don’t seem to point out that one of the host organisations is making very basic mistakes in how they go about their main job.
The harm of platforming bigots is manifold: firstly, that the people they hate are directly harmed; secondly that they will most likely stay away, reducing your diversity and thus the variety of perspectives and potential insight present; thirdly, that you will be perceived as bigoted, harming you socially and further reducing the diversity of the event. These problems have been extensively discussed elsewhere, and I see no value in discussing them again here. The issue that hasn’t been discussed is how the people who feel excluded are those who are most used to critiquing power inequality, and if they still engage with your platform at all, will focus on discussing your bigotry, rather than other structural issues.
Manifold markets isn’t very good
Disclaimer
I currently have an around 400-day streak on Manifold Markets (though lately I only spend a minute or two a day on it) and have no particular vendetta against it. I also use Metaculus. I’m reasonably well-ranked on both but have not been paid by either platform, ignoring a few Manifold donations. I have not attended any Manifest. I think Manifold has value as a weird form of social media, but I think it’s important to be clear that this is what it is, and not a manifestation of collective EA or rationalist consciousness, or an effective attempt to improve the world in its current form.
Overview of Manifold
Manifold is a prediction market website where people can put virtual money (called “mana”) into bets on outcomes. There are several key features of this: 1. You’re rewarded with virtual money both for participating and for predicting well, though you can also pay to get more. 2. You can spend this mana to ask questions, which you will generally vet and resolve yourself (allowing many more questions than on comparable sites). Moderators can reverse unjustified decisions but it’s usually self-governed. Until recently, you could also donate mana to real charities, though recently this stopped; now only a few “prize” questions provide a more exclusive currency that can be donated, and most questions produce unredeemable mana.
How might it claim to improve the world?
There are two ways in which Manifold could be improving the world. It could either make good predictions (which would be intrinsically valuable for improving policy or making wealth) or it could donate money to charities. Until recently, the latter looked quite reasonable: the company appeared to be rewarding predictive power with the ability to donate money to charities. The counterfactuality of these donations is questionable, however, since the money for it came from EA-aligned grants, and most of it goes to very mainstream EA charities. It has a revenue stream from people buying mana, but this is less than $10k/month, some of which isn’t really revenue (since it will ultimately be converted to donations), and presumably this doesn’t cover the staff costs. The founders appear to believe that eventually they will get paid enough money to run markets for other organisations, in which case the donations would be counterfactual. But this relies on the markets producing good predictions.
Sadly, Manifold does not produce particularly good predictions. In last year’s ACX contest, it performed worse than simply averaging predictions from the same number of people who took part in each market. Their calibration, while good by human standards, has a clear systematic bias towards predicting things will happen when they don’t (Yes bias). By contrast, rival firm Metaculus has no easily-corrected bias and seems to perform better at making predictions on the same questions (including in the ACX contest). Metaculus’ self-measured Brier score is 0.111, compared to Manifold’s 0.168 (lower is better, and this is quite a lot lower, though they are not answering all the same questions). Metaculus doesn’t publish the number of monthly active users like Manifold does, but the number of site visits they receive are comparable (slightly higher for Metaculus by one measure, lower by another), so it doesn’t seem like the prediction difference can be explained by user numbers alone.
Can the predictive power be improved?
Some of the problems with Manifold, like the systematic Yes bias, can be algorithmically fixed by potential users. Others are more intrinsic to the medium. Many questions resolve based on extensive discussions about exactly how to categorise reality, meaning that subtle clarifications by the author can result in huge swings in probability. The market mechanism of Manifold produces an information premium that incentivises people to act quickly on information, meaning that questions also swing wildly based on rumours.
The idea behind giving people mana for predictions is that wealth should accumulate with predictive prowess, resulting in people who have historically predicted better being able to weight their opinions more strongly. However in practice Manifold has many avenues for people incapable of making correct predictions to get Mana. It:
1. Wants bad predictors to pay money to keep playing
2. Hands out mana for asking engaging questions (which are not at all the same as useful questions)
3. Has many weird meta markets where people with lots of money can typically make more mana without predicting anything other than the spending of the superwealthy (“whalebait”)
4. Allows personal markets, where people can earn mana by doing tasks they set for themselves. This is an officially endorsed form of insider trading, insider trading being generally accepted.
5. Lets people gamble, go into negative equity, then burn their accounts and make new ones if bets go badly. The use of “puppet accounts” to manipulate markets in this or more complex ways is actively fought but still happens, and several of the all-time highest earners have transpired to use them to generate wealth.
6. Tends to leave markets open until after the answer to a question is publicly known and the answer is posted in the comments, so as well as large mana rewards for reading the news fast, simply reacting to posts on the site itself can produce a reliable profit
The first of these is entirely structural to their business model. While in principle the others could be “fixed” (if you consider them bugs) and some have lately become smaller (e.g. you can no longer bet things outside the 1-99% range, reducing profit from publicly known events), several have proven rather robust. This is quite apart from insider trading (which arguably makes the markets more accurate, even if it rewards people unfairly) and pump-and-dump schemes, which are hard to fix in any market system.
Framed as a fun pastime, having tricks to make mana without making predictions is fine, and this could drive up engagement. But these are all signs that the accuracy of the predictions (as opposed to news-reading) is not a highly ranked goal of the site. And it’s important to highlight the tradeoff between attracting idiots who will pay to play, as on gambling sites, and attracting companies who will pay to get good results. The assumption that idiots are distributed across opinions and so their biases will cancel each other out is ill-founded, particularly if a site has poor diversity.
By comparison, Metaculus simply applies an algorithm to weight people’s opinions based on past performance. There are obviously systematic differences between who uses which website, but a study indicated that under ideal conditions, a non-market approach to aggregating opinions was more successful than a prediction market. Prediction markets may be better than just doing a poll (for short-term predictions – Manifold can argue for its calibration over longer times via its loan system, which is plausible but this hasn’t been demonstrated), but worse than measures that can actually look at who is saying what. This would indicate that even without the quirky mana sources and time-dynamics, we wouldn’t expect Manifold to produce the best predictions. You can also see market failures explicitly in a handful of long-term markets (often about AI extinction risk, which cannot possibly pay out to humans) where a small number of very rich people have pegged the value of the market at the levels that they want it, in spite of this being essentially guaranteed to lose mana. Arguably this is the opposite criticism to the free-market criticisms—humans being willing to lose currency to do what they perceive as the right thing to highlight AI extinction risk—but it still represents an oligarchic market failure resulting in biased predictions. Unless prediction markets are really very large indeed and have no form of oligarchy, the neoliberal assumption that we can ignore who is putting up the currency is invalid.
This creates interesting possibilities to improve predictions. One, Manifold could have taxes to reduce opinion inequality. Two, Manifold could report the market values for predictions but (for a price) give you the better probabilities calculated by an appropriate algorithm, which is basically guaranteed to be better (it can always degenerate to “just use the market value” if that’s genuinely best). Why hasn’t it done this already? Well, there are probably several reasons, but one that I’m highlighting here is the link to neoliberal ideology. This is not to say that users of the site are all neoliberal – though they probably are more so than average. I mean that the “contrarian” ideologues present at the controversial Manifest24 and that the co-founders of manifold clearly find most interesting are consistently from a right-wing, market-trusting perspective, and this creates material blind spots. In spite of having many tens of talks on how to make predictions at this event, people don’t seem to point out that one of the host organisations is making very basic mistakes in how they go about their main job.
The harm of platforming bigots is manifold: firstly, that the people they hate are directly harmed; secondly that they will most likely stay away, reducing your diversity and thus the variety of perspectives and potential insight present; thirdly, that you will be perceived as bigoted, harming you socially and further reducing the diversity of the event. These problems have been extensively discussed elsewhere, and I see no value in discussing them again here. The issue that hasn’t been discussed is how the people who feel excluded are those who are most used to critiquing power inequality, and if they still engage with your platform at all, will focus on discussing your bigotry, rather than other structural issues.