Thanks for your considered comments! I agree that Metaculus should make its best prediction more available. I also attach low importance to the self-reported Brier scores, though Manifold already excludes a tail of low-traded questions when reporting, so that’s not really a good explanation for the discrepancy.
To be clear, the paper specifies that *algorithmic adjustments* of polls out-perform markets, not that the means of polls are better than the means of markets (in line with the differences between the two Metaculus predictions). If you don’t adjust, they’re worse, as expected and seen in the Metaculus calibration data. This conclusion is clearly written in the abstract, and they didn’t try very complicated algorithms to combine estimates.
I agree (and mentioned) that recent changes alleviate some of these points. I don’t think it cures them as thoroughly as you indicate though. Firstly, the pivot didn’t retroactively apply these changes, so people who successfully asked engaging questions or caught whalebait still have huge mana supplies. If they’re not limited by engagement time, people with any positive predictive power can exponentially grow the cash injection, and the profit will naturally then be laundered into conventional markets. In practice, I don’t think top whales are exponentially growing their income most of the time—growth usually seems pretty linear, probably due to the difficulty of finding appropriate markets. But if you wanted to prove that good whalebait hunters are good predictors, you will need to demonstrate that they get a good rate of return on their investment, not merely that they have also derived M from other sources.
People can no longer go into negative equity, though you can still create accounts and transfer the M600 or make risky bets, reducing but not fixing the issue.
I just went on the site and found free mana for day-old news within the top 10 links. Ironically the pivot/transaction taxes means that there’s less incentive for people with limited M to pick up these pennies, so they’re left out for longer and mainly benefit whales. There are mechanisms to stop news-based trading (e.g. you could retroactively reverse post-news transactions) but they will create negative equity problems again.
I am generally skeptical that some of the changes made during the pivot will remain in the long term, as it seems like the number of users has trending downwards since it happened, and changes have broken some other things. Most noteworthily, there is now no force mitigating the time value of money effects, so we do not expect the market value of long-term markets to equal the expectation of that market even under ideal circumstances. Also, the transaction taxes are large, which creates market inefficiencies, lowering the precision of the market (because it’s not worth correcting a market error unless it’s wrong by a larger margin now). These problems are ones that neoliberal economists ought be aware of though, so I imagine there are plans to mitigate them.
I think you misunderstood the counterfactuality point. The counterfactuality issue of the charity program is that the EA orgs could just have given them to the charities they normally do, without putting them into Manifold bank accounts in the meantime and waiting for people to choose which ones to give to. Allowing people to take the money out as dollars is irrelevant, and just delays things more.
(What follows are a lot of disagreements that are very minor in the grand scheme of things, I think the points I made in the first comment are still valid)
I think that, going forward, they have been cured “as thoroughly as [I] indicate”, and the impact from past printed mana is small, although I do wish Manifold had never caused those issues.
Firstly, the pivot didn’t retroactively apply these changes, so people who successfully asked engaging questions or caught whalebait still have huge mana supplies
Sure. Bad traders who caught whalebait have, mostly, lost their winnings since whalebait was restricted though. Catnee, for instance, was the biggest winner on WvM, but has since lost it all to bad predictions. Marcus was the second biggest winner on WvM, and he hasn’t lost his winnings since he’s the 3rd best trader on ‘real’ questions, and he’s donated away more than his WvM winnings anyway. And the devaluation helps reduce the impact of ill-gotten mana that remains. But you used those points to support this claim in the original post:
these are all signs that the accuracy of the predictions (as opposed to news-reading) is not a highly ranked goal of the site
Which was very misleading.
people who successfully asked engaging questions or caught whalebait still have huge mana supplies
I don’t think this is significantly distorting prices. Of the active people on the creator leaderboard, people like Jack and Joshua who are top traders have 10x higher net worths than the highest creators who aren’t top traders like the top 3 highest, BTE, Isaac, and strutheo. And creators need to use that mana to pay for more questions.
But if you wanted to prove that good whalebait hunters are good predictors
I wouldn’t claim that in general (Marcus specifically was good at both). The current top trader leaderboard doesn’t count whalebait profits at all.
I don’t think I understand the claim about exponential growth.
Most noteworthily, there is now no force mitigating the time value of money effects, so we do not expect the market value of long-term markets to equal the expectation of that market even under ideal circumstances.
Yeah, this is a big problem for prediction markets. Although I don’t think loans helped in the past so much as masked the problem by allowing people to overleverage.
Also, the transaction taxes are large, which creates market inefficiencies, lowering the precision of the market (because it’s not worth correcting a market error unless it’s wrong by a larger margin now
The fees for a trade at 50% are around 3.5%, which you’d expect to distort the probability by at most 1.8% in either direction.
People can no longer go into negative equity, though you can still create accounts and transfer the M600 or make risky bets, reducing but not fixing the issue.
Manifold bans people if we notice them doing this, and I don’t think this is different in kind from creating sockpuppet accounts on Metaculus.
I think you misunderstood the counterfactuality point. The counterfactuality issue of the charity program is that the EA orgs could just have given them to the charities they normally do, without putting them into Manifold bank accounts in the meantime and waiting for people to choose which ones to give to. Allowing people to take the money out as dollars is irrelevant, and just delays things more.
I strongly agree with that concern about manifold’s old charity program! But in the future if you win on Manifold and withdraw winnings and donate them, that money comes from money other traders used to purchase mana, not EA donations to manifold, so it’s not an issue anymore.
I just went on the site and found free mana for day-old news within the top
Sure, news trading was the .5 in my 1.5 of 6 - still exists but much lower rewards. The amount of such free mana available now is generally on the order of ones or tens of cents, though, sometimes a dollar or two, so it isn’t really worth it for whales to pick it up.
The idea that real money improves performance is another of these neoliberal assumptions with limited evidence. There are a range of papers on this issue that come to different answers as to what, if any, conditions exist for it to be true.
Yeah, I wouldn’t assume that, I noted benefits and drawbacks. I am not sure which of the models of Manifold or Metaculus are better for prediction, or indeed if either are socially valuable enough to be worth using or watching. I think the same point about limited evidence and different assumptiosn applies to the paper you cite comparing adjusted prediction polls to markets.
It is almost certainly not true for extinction risk factors, which is a substantial EA interest for prediction-making.
Yeah, those incentives won’t work for extinction risk markets on any platform because you’d be dead.
It could be true that there is some threshold beyond which money becomes strongly influential, but for instance, Metaculus informally finds running competitions for $1000s to harm engagement in questions.
That particular argument is once the money involved is large enough to make a career or run a business off of, it’ll draw smart people in. There are a few people who professionally trade polymarket and make a good income.
By my accounts, you have implicitly agreed that all of 1-6 used to be issues, but 2-4 are currently not issues and 5 now needs the phrase “negative equity” deleted. I’m still making mana by reading the news, so don’t see that you’ve halved that claim. You’re right that whalebait is less profitable, and I now need to actually search for free mana to find the free mana markets. The fact that I can still do this and then throw all my savings into it means that we should expect exponential growth of mana at some risk-free rate (depending on the saturation of these markets), which is then the comparison point for determining investment skill. In practice there are most likely better things to do with it, and also I can’t be bothered.
I recognise the benefit of inflation as a good thing in countering historic wealth inequality, and will remark that it’s effectively a wealth tax. It unfortunately coincided with the other changes which make it harder and less rewarding to donate and worsening the time-value problem, triggering my general disengagement with the site. I agree that loans never fixed this problem, but they mitigated it partially.
The difference between this and Metaculus sock puppets is that there’s no reward for making them there. The virtual currencies can’t be translated into real-world gain, and only one “reward” depends on other people, so making bad predictions with your sock puppets doesn’t make you look that much better if people look at multiple metrics. Similarly, by requiring currency to express a belief, Manifold structurally limits engagement on questions with no positive resolution possibility—it’s cost-free to predict extinction on Metaculus, but on Manifold, even with perfect foresight (or the guarantee that the market will be NAd later) you still sacrifice the time value of your mana to warn people of a risk. This problem is unique to prediction markets. They make it costly (but potentially remunerated) to express your beliefs.
The other problem unique to adversarial prediction grading is that collaboration is disincentivised. Currently, because mana isn’t that valuable, the comments section is full of people exchanging information for social kudos. But when the market becomes financially lucrative people stop doing this—the comments on polymarket are basically pure spam. This is one of the reasons why I find the idea that Manifold should become more financialised very unwise. It’s not clear that the collaborative factor is smaller than the professionalisation factor for net predictive power (as indicated by the fact that polymarket doesn’t have that good a calibration). To make money on these things, you don’t need to beat a superforecasting team (the thing that actually beats all of these statistical aggregation methods, least we forget), you need to beat the individual whose salary the prize can support.
I don’t believe the original donation has been redistributed and donations are now curtailed by the pivot, so I imagine it will last a while longer. I know the founders believe donations will eventually come from mana purchases (or more venture capital), I’m just skeptical.
Thanks for your considered comments! I agree that Metaculus should make its best prediction more available. I also attach low importance to the self-reported Brier scores, though Manifold already excludes a tail of low-traded questions when reporting, so that’s not really a good explanation for the discrepancy.
To be clear, the paper specifies that *algorithmic adjustments* of polls out-perform markets, not that the means of polls are better than the means of markets (in line with the differences between the two Metaculus predictions). If you don’t adjust, they’re worse, as expected and seen in the Metaculus calibration data. This conclusion is clearly written in the abstract, and they didn’t try very complicated algorithms to combine estimates.
I agree (and mentioned) that recent changes alleviate some of these points. I don’t think it cures them as thoroughly as you indicate though. Firstly, the pivot didn’t retroactively apply these changes, so people who successfully asked engaging questions or caught whalebait still have huge mana supplies. If they’re not limited by engagement time, people with any positive predictive power can exponentially grow the cash injection, and the profit will naturally then be laundered into conventional markets. In practice, I don’t think top whales are exponentially growing their income most of the time—growth usually seems pretty linear, probably due to the difficulty of finding appropriate markets. But if you wanted to prove that good whalebait hunters are good predictors, you will need to demonstrate that they get a good rate of return on their investment, not merely that they have also derived M from other sources.
People can no longer go into negative equity, though you can still create accounts and transfer the M600 or make risky bets, reducing but not fixing the issue.
I just went on the site and found free mana for day-old news within the top 10 links. Ironically the pivot/transaction taxes means that there’s less incentive for people with limited M to pick up these pennies, so they’re left out for longer and mainly benefit whales. There are mechanisms to stop news-based trading (e.g. you could retroactively reverse post-news transactions) but they will create negative equity problems again.
I am generally skeptical that some of the changes made during the pivot will remain in the long term, as it seems like the number of users has trending downwards since it happened, and changes have broken some other things. Most noteworthily, there is now no force mitigating the time value of money effects, so we do not expect the market value of long-term markets to equal the expectation of that market even under ideal circumstances. Also, the transaction taxes are large, which creates market inefficiencies, lowering the precision of the market (because it’s not worth correcting a market error unless it’s wrong by a larger margin now). These problems are ones that neoliberal economists ought be aware of though, so I imagine there are plans to mitigate them.
The idea that real money improves performance is another of these neoliberal assumptions with limited evidence. There are a range of papers on this issue that come to different answers as to what, if any, conditions exist for it to be true.
https://www.tandfonline.com/doi/abs/10.1080/1019678042000245254
https://www.electronicmarkets.org/fileadmin/user_upload/doc/Issues/Volume_16/Issue_01/V16I1_Statistical_Tests_of_Real-Money_versus_Play-Money_Prediction_Markets.pdf
https://ubplj.org/index.php/jpm/article/view/441
https://www.ubplj.org/index.php/jpm/article/view/479
It is almost certainly not true for extinction risk factors, which is a substantial EA interest for prediction-making. It could be true that there is some threshold beyond which money becomes strongly influential, but for instance, Metaculus informally finds running competitions for $1000s to harm engagement in questions.
I think you misunderstood the counterfactuality point. The counterfactuality issue of the charity program is that the EA orgs could just have given them to the charities they normally do, without putting them into Manifold bank accounts in the meantime and waiting for people to choose which ones to give to. Allowing people to take the money out as dollars is irrelevant, and just delays things more.
(What follows are a lot of disagreements that are very minor in the grand scheme of things, I think the points I made in the first comment are still valid)
I think that, going forward, they have been cured “as thoroughly as [I] indicate”, and the impact from past printed mana is small, although I do wish Manifold had never caused those issues.
Sure. Bad traders who caught whalebait have, mostly, lost their winnings since whalebait was restricted though. Catnee, for instance, was the biggest winner on WvM, but has since lost it all to bad predictions. Marcus was the second biggest winner on WvM, and he hasn’t lost his winnings since he’s the 3rd best trader on ‘real’ questions, and he’s donated away more than his WvM winnings anyway. And the devaluation helps reduce the impact of ill-gotten mana that remains. But you used those points to support this claim in the original post:
Which was very misleading.
I don’t think this is significantly distorting prices. Of the active people on the creator leaderboard, people like Jack and Joshua who are top traders have 10x higher net worths than the highest creators who aren’t top traders like the top 3 highest, BTE, Isaac, and strutheo. And creators need to use that mana to pay for more questions.
I wouldn’t claim that in general (Marcus specifically was good at both). The current top trader leaderboard doesn’t count whalebait profits at all.
I don’t think I understand the claim about exponential growth.
Yeah, this is a big problem for prediction markets. Although I don’t think loans helped in the past so much as masked the problem by allowing people to overleverage.
The fees for a trade at 50% are around 3.5%, which you’d expect to distort the probability by at most 1.8% in either direction.
Manifold bans people if we notice them doing this, and I don’t think this is different in kind from creating sockpuppet accounts on Metaculus.
I strongly agree with that concern about manifold’s old charity program! But in the future if you win on Manifold and withdraw winnings and donate them, that money comes from money other traders used to purchase mana, not EA donations to manifold, so it’s not an issue anymore.
Sure, news trading was the .5 in my 1.5 of 6 - still exists but much lower rewards. The amount of such free mana available now is generally on the order of ones or tens of cents, though, sometimes a dollar or two, so it isn’t really worth it for whales to pick it up.
Yeah, I wouldn’t assume that, I noted benefits and drawbacks. I am not sure which of the models of Manifold or Metaculus are better for prediction, or indeed if either are socially valuable enough to be worth using or watching. I think the same point about limited evidence and different assumptiosn applies to the paper you cite comparing adjusted prediction polls to markets.
Yeah, those incentives won’t work for extinction risk markets on any platform because you’d be dead.
That particular argument is once the money involved is large enough to make a career or run a business off of, it’ll draw smart people in. There are a few people who professionally trade polymarket and make a good income.
By my accounts, you have implicitly agreed that all of 1-6 used to be issues, but 2-4 are currently not issues and 5 now needs the phrase “negative equity” deleted. I’m still making mana by reading the news, so don’t see that you’ve halved that claim. You’re right that whalebait is less profitable, and I now need to actually search for free mana to find the free mana markets. The fact that I can still do this and then throw all my savings into it means that we should expect exponential growth of mana at some risk-free rate (depending on the saturation of these markets), which is then the comparison point for determining investment skill. In practice there are most likely better things to do with it, and also I can’t be bothered.
I recognise the benefit of inflation as a good thing in countering historic wealth inequality, and will remark that it’s effectively a wealth tax. It unfortunately coincided with the other changes which make it harder and less rewarding to donate and worsening the time-value problem, triggering my general disengagement with the site. I agree that loans never fixed this problem, but they mitigated it partially.
The difference between this and Metaculus sock puppets is that there’s no reward for making them there. The virtual currencies can’t be translated into real-world gain, and only one “reward” depends on other people, so making bad predictions with your sock puppets doesn’t make you look that much better if people look at multiple metrics. Similarly, by requiring currency to express a belief, Manifold structurally limits engagement on questions with no positive resolution possibility—it’s cost-free to predict extinction on Metaculus, but on Manifold, even with perfect foresight (or the guarantee that the market will be NAd later) you still sacrifice the time value of your mana to warn people of a risk. This problem is unique to prediction markets. They make it costly (but potentially remunerated) to express your beliefs.
The other problem unique to adversarial prediction grading is that collaboration is disincentivised. Currently, because mana isn’t that valuable, the comments section is full of people exchanging information for social kudos. But when the market becomes financially lucrative people stop doing this—the comments on polymarket are basically pure spam. This is one of the reasons why I find the idea that Manifold should become more financialised very unwise. It’s not clear that the collaborative factor is smaller than the professionalisation factor for net predictive power (as indicated by the fact that polymarket doesn’t have that good a calibration). To make money on these things, you don’t need to beat a superforecasting team (the thing that actually beats all of these statistical aggregation methods, least we forget), you need to beat the individual whose salary the prize can support.
I don’t believe the original donation has been redistributed and donations are now curtailed by the pivot, so I imagine it will last a while longer. I know the founders believe donations will eventually come from mana purchases (or more venture capital), I’m just skeptical.