Manifold Markets ran a prediction tournament to see whether forecaster would be able to predict the winners! For each Cause Exploration Prize entry, we had a market on “Will this entry win first or second place?”. Check out the tournament rules and view all predictions here.
I think overall, the markets did okay—they managed to get the first place entry (“Organophosphate pesticides and other neurotoxicants”) as the highest % to win, and one of the other winners was ranked 4th (“Violence against women and girls”). However, they did miss out on the two dark horse winners (“Sickle cell disease” and “shareholder activism”), which could have been one hypothetical way markets would outperform karma. Specifically, none of the Manifold forecasters placed a positive YES bet on either of the dark horse candidates.
I’m not sure that the markets were much better predictors than just EA Forum Karma—and it’s possible that most of the signal from the markets were just forecasters incorporating EA Forum Karma into their predictions. The top 10 predictions by Karma also had 2 of the 1st/2nd place winners:
And if you include honorable mentions in the analysis, EA Forum Karma actually did somewhat better. Manifold Markets had 7⁄10 “winners” (first/second/honorable), while EA Forum Karma had 9⁄10.
Thanks again for the team at OpenPhil (especially Chris and Aaron) for hosting these prizes and thereby sponsoring so many great essays! Would love to see that writeup about learnings, especially curious what the decision process was that lead to these winners and honorable mentions.
This makes me interested to know whether EA Forum engagement and the (real or perceived) popularity of these topics among the community contributed to Open Philanthropy’s decision-making process.
I am hugely grateful for the opportunity and participation award, but I’m sure like many other contest entrants, I submitted my entry in the belief that my cause area (TB) met and even exceeded the neglectedness, tractability, and cost-effectiveness criteria. The above post suggests to me that maybe other criteria were used. Can entrants expect anything in the way of feedback or some general comments on how entries were selected?
Congratulations to the winners, and thank you to Open Philanthropy for taking this initiative. I would welcome another like it in the future.
Hi jserv! I’ll aim to say a bit more about the nuts and bolts of the process in an update before the end of the year, but prize selection was dominated by blind, independent review. I’m following up with you privately on tuberculosis.
Mainly that forum karma was highly predictive of winners when you include honourable mentions, even moreso than Manifold Markets. It could be for several reasons; forum votes were taken into consideration, there was a high degree of alignment between forum voters and the Open Philanthropy panel on the known call criteria (slightly more than MM, another related community), karma acted as a proxy for something else that factored into selection like timing of the submission, general community interest, familiarity with the topic, or true quality of the submissions, or it could be chance.
As an entrant, I am very interested to know whether any of the above played a role. Also, Open Philanthropy have mentioned this was an experiment, so it would be great to know whether anything unexpected came up during their decision-making.
Manifold Markets ran a prediction tournament to see whether forecaster would be able to predict the winners! For each Cause Exploration Prize entry, we had a market on “Will this entry win first or second place?”. Check out the tournament rules and view all predictions here.
I think overall, the markets did okay—they managed to get the first place entry (“Organophosphate pesticides and other neurotoxicants”) as the highest % to win, and one of the other winners was ranked 4th (“Violence against women and girls”). However, they did miss out on the two dark horse winners (“Sickle cell disease” and “shareholder activism”), which could have been one hypothetical way markets would outperform karma. Specifically, none of the Manifold forecasters placed a positive YES bet on either of the dark horse candidates.
I’m not sure that the markets were much better predictors than just EA Forum Karma—and it’s possible that most of the signal from the markets were just forecasters incorporating EA Forum Karma into their predictions. The top 10 predictions by Karma also had 2 of the 1st/2nd place winners:
And if you include honorable mentions in the analysis, EA Forum Karma actually did somewhat better. Manifold Markets had 7⁄10 “winners” (first/second/honorable), while EA Forum Karma had 9⁄10.
Thanks again for the team at OpenPhil (especially Chris and Aaron) for hosting these prizes and thereby sponsoring so many great essays! Would love to see that writeup about learnings, especially curious what the decision process was that lead to these winners and honorable mentions.
This makes me interested to know whether EA Forum engagement and the (real or perceived) popularity of these topics among the community contributed to Open Philanthropy’s decision-making process.
I am hugely grateful for the opportunity and participation award, but I’m sure like many other contest entrants, I submitted my entry in the belief that my cause area (TB) met and even exceeded the neglectedness, tractability, and cost-effectiveness criteria. The above post suggests to me that maybe other criteria were used. Can entrants expect anything in the way of feedback or some general comments on how entries were selected?
Congratulations to the winners, and thank you to Open Philanthropy for taking this initiative. I would welcome another like it in the future.
Hi jserv! I’ll aim to say a bit more about the nuts and bolts of the process in an update before the end of the year, but prize selection was dominated by blind, independent review. I’m following up with you privately on tuberculosis.
Thank you!
Out of curiosity, which parts of Austin’s comment suggests to you something about the criteria used?
Mainly that forum karma was highly predictive of winners when you include honourable mentions, even moreso than Manifold Markets. It could be for several reasons; forum votes were taken into consideration, there was a high degree of alignment between forum voters and the Open Philanthropy panel on the known call criteria (slightly more than MM, another related community), karma acted as a proxy for something else that factored into selection like timing of the submission, general community interest, familiarity with the topic, or true quality of the submissions, or it could be chance.
As an entrant, I am very interested to know whether any of the above played a role. Also, Open Philanthropy have mentioned this was an experiment, so it would be great to know whether anything unexpected came up during their decision-making.