I don’t think this will be an issue for Manifold. For example Polymarket allows controversial users and content, and indeed barely moderates comments sections at all, and currently has much higher volume than Manifold. And Polymarket does provide some socially useful predictions—it’s somewhat frequently cited by the mainstream media for presidential election odds.
Manifold user
Karma: 32
I’m a frequent Manifold user.
While I like criticism of Manifold, I think these criticisms miss the mark, and are mostly false or outdated.
The best criticism here is the link the the ACX article. I’d still defend manifold, the ACX predictions occurred very early in the site’s life, and most of the current top traders either didn’t use the site at all or didn’t have enough capital to move markets, so one would expect it to be less accurate. But we did perform badly.
You compare Metaculus’s “Metaculus prediction” calibration graph to Manifold’s calibration. Metaculus has two probabilities for each question—a “Community Prediction”, a simple aggregation of forecasts, and a more complicated Metaculus Prediction that weights by track record. The Metaculus Prediction is not public while markets are open, and I can’t see it for any of the metaculus markets I’m interested in. So, when comparing the utility of the two sites the Community Prediction is the right comparison, and it’s as bad as or worse than Manifold’s.
Manifold’s hosted calibration graph graph covers markets through all of manifold’s history. I’d expect manifold to do better recently. Using https://calibration.city/, using the ‘market midpoint’ and starting from Jul 2023, Manifold’s calibration looks great (later dates seem noisy but unbiased, the other mode is buggy). I also recalculated time-weighted calibration myself over the past months of Manifold, and didn’t see deviation from y=x other than noise. (It does replicate manifold’s old bias on all past data). Manifold’s site shows the old bias, but we’ve improved!
Manifold’s overall Brier score is not comparable to Metaculus’s. “they are not answering all the same questions” is an issue, the distribution of user-generated questions is different from highly curated questions. Manifold also has a much longer tail of less-traded questions that you’d expect to score lower even if the question distributions were the same.
For the analysis of brier scores on comparable questions, it’s early in Manifold’s life, and from the post, “Metaculus, on average had a much higher number of forecasters”, and from a comment “Some of the markets in this dataset probably only ever got single-digit number of trades and were obviously mispriced”. This shouldn’t generalize to comparisons between metaculus and manifold today, with comparable numbers of users.
“Many questions resolve based on extensive discussions about exactly how to categorise reality, meaning that subtle clarifications by the author can result in huge swings in probability”. This does happen, one of the tradeoffs with user-created markets is the criteria are less detailed, but it’s not common enough to significantly affect profits for the top traders. In my rough estimation, around 1 in 20 big manifold questions have criteria issues, and many of those N/A. As a comparison, 3 out of the 50 Bridgewater Metaculus competition questions had to be annulled due to bad criteria.
The study is interesting, but the experiment has significant differences from manifold and I don’t get from the paper that prediction polls are overall superior to markets.
You name “many avenues for people incapable of making correct predictions to get Mana”. After the pivot, in Manifold’s current state, all but ~1.5 of these are no longer relevant.
“2. Hands out mana for asking engaging questions (which are not at all the same as useful questions)”—In the past, Manifold awarded generous creator bonuses and market liquidity for traders, inflating the mana supply. Post-pivot, creators fund liquidity pools themselves and only earn mana through roughly 3-1.5% fees on their markets, so market creation is now a mana sink for creators (and, net, for traders, Manifold gets fees as well). (There remains a partner program to pay selected creators, but that pays out in USD, is smaller, and partners are generally not converting that to mana.)
“3. Has many weird meta markets where people with lots of money can typically make more mana without predicting anything other than the spending of the superwealthy (“whalebait”)”—Also true in the past, with the Whales vs Minnows https://news.manifold.markets/p/isaac-kings-whales-vs-minnows-and incident as the apex, making half the profit of the top trader at the time. (whalebait skill is correlated with forecasting skill, that trader remains #3 now that whalebait profit isn’t counted). But after that Manifold started hiding whalebait, and today large whalebait markets are effectively dead.
“Allows personal markets, where people can earn mana by doing tasks they set for themselves. This is an officially endorsed form of insider trading, insider trading being generally accepted.”—Post pivot, creators fund the liquidity pools for personal markets and trading on markets no longer prints mana via bonuses (it’s negative sum due to fees), so you can’t earn mana like that.
“Lets people gamble, go into negative equity, then burn their accounts and make new ones if bets go badly”—With the pivot, Manifold has stopped giving out loans, which was what allowed people to go into negative equity.
“6. Tends to leave markets open until after the answer to a question is publicly known and the answer is posted in the comments, so as well as large mana rewards for reading the news fast, simply reacting to posts on the site itself can produce a reliable profit”—I traded on this in the past, but it’s 10x less effective now that market creators fund their own liquidity pools, so they now usually close markets before the answer is known to recover some of their money (prize markets do that too). Trading on news still happens though, and is probably inevitable with prediction markets.
“1. Wants bad predictors to pay money to keep playing” is a problem, and is an issue with any prediction market involving real money. Real money has benefits, though—as prediction markets scale, it incentivize professional prediction. Bad traders incentivize good traders to price markets well.
> But these are all signs that the accuracy of the predictions (as opposed to news-reading) is not a highly ranked goal of the site
As I see it, the fact that we did make all of these changes is a sign that prediction accuracy is a top goal!
The counterfactuality of the charity program was an issue, but post-pivot that’s not relevant anymore as you can just withdraw prizepoints as USD and then donate.
I do have concerns about the utility of prediction market (and forecasting generally). Pushing information into probabilities might remove most of the value that a richer written exchange would have. Prediction markets incentivize smart people to keep information other than their bets private. But this post missed the mark, I think.
I probably got one, maybe a few, things wrong here, but the general point stands
(What follows are a lot of disagreements that are very minor in the grand scheme of things, I think the points I made in the first comment are still valid)
I think that, going forward, they have been cured “as thoroughly as [I] indicate”, and the impact from past printed mana is small, although I do wish Manifold had never caused those issues.
Sure. Bad traders who caught whalebait have, mostly, lost their winnings since whalebait was restricted though. Catnee, for instance, was the biggest winner on WvM, but has since lost it all to bad predictions. Marcus was the second biggest winner on WvM, and he hasn’t lost his winnings since he’s the 3rd best trader on ‘real’ questions, and he’s donated away more than his WvM winnings anyway. And the devaluation helps reduce the impact of ill-gotten mana that remains. But you used those points to support this claim in the original post:
Which was very misleading.
I don’t think this is significantly distorting prices. Of the active people on the creator leaderboard, people like Jack and Joshua who are top traders have 10x higher net worths than the highest creators who aren’t top traders like the top 3 highest, BTE, Isaac, and strutheo. And creators need to use that mana to pay for more questions.
I wouldn’t claim that in general (Marcus specifically was good at both). The current top trader leaderboard doesn’t count whalebait profits at all.
I don’t think I understand the claim about exponential growth.
Yeah, this is a big problem for prediction markets. Although I don’t think loans helped in the past so much as masked the problem by allowing people to overleverage.
The fees for a trade at 50% are around 3.5%, which you’d expect to distort the probability by at most 1.8% in either direction.
Manifold bans people if we notice them doing this, and I don’t think this is different in kind from creating sockpuppet accounts on Metaculus.
I strongly agree with that concern about manifold’s old charity program! But in the future if you win on Manifold and withdraw winnings and donate them, that money comes from money other traders used to purchase mana, not EA donations to manifold, so it’s not an issue anymore.
Sure, news trading was the .5 in my 1.5 of 6 - still exists but much lower rewards. The amount of such free mana available now is generally on the order of ones or tens of cents, though, sometimes a dollar or two, so it isn’t really worth it for whales to pick it up.
Yeah, I wouldn’t assume that, I noted benefits and drawbacks. I am not sure which of the models of Manifold or Metaculus are better for prediction, or indeed if either are socially valuable enough to be worth using or watching. I think the same point about limited evidence and different assumptiosn applies to the paper you cite comparing adjusted prediction polls to markets.
Yeah, those incentives won’t work for extinction risk markets on any platform because you’d be dead.
That particular argument is once the money involved is large enough to make a career or run a business off of, it’ll draw smart people in. There are a few people who professionally trade polymarket and make a good income.