I like to feel like additional effort makes a difference. The easiest way to get to the top of the metaculus leaderboard is to forecast on everything. Not well, but just to do it. This is pretty dispiriting.
I did Metaculus for a while but I wasn’t quite sure how to assess how well I was doing and I lost interest. I know Brier score isn’t the greatest metric. Just try to accumulate points?
Not being able to mark predictions as quick and slow.
Sometimes I spend a long time on a forecast (10-20 mins) sometimes I spend about 30s. It frustrates me that all the current sites don’t let me track these two quite different types.
Forecasts are rather time-intensive—I used to make a bunch of metaculus forecasts, but putting serious effort into researching something, just to come up with a number to toss into the community mix, rarely seems worth the effort these days. Instead, I have a backlog of blog posts and other projects that seem higher expected value. (But going forwards, I should try to contribute forecasts when I find myself already researching something EA-adjacent anyways.)
Related to this, unclear impact—as Nuno judges here, it’s hard for forecasts to really hit the sweet spot of being at the right difficulty level and also being decision-relevant. If I had more of an understanding of who finds forecasts useful, I might be more interested in contributing.
As you say, there’s a tension between effortful vs informal forecasts: Out of humility / modesty / imposter syndrome, sometimes I don’t want to mess up a pool of expert forecasts by tossing out an ill-informed hunch.
I think Metaculus could benefit from adopting the more stock-market / prediction-market style mechanism of letting players stake different amounts of their total forecasting “budget” on individual questions… prediction-market style, experienced players or players who wanted to make a very confident prediction could pour lots of points into the system. Then I would be better able to distinguish my confident from my less-confident predictions, and small numbers of experts could bet their beliefs to overturn an ill-judged majority opinion.
Although I recognize that Metaculus’s current, more egalitarian style probably helps to grow the userbase and encourage participation.
Scoring systems
I like to feel like additional effort makes a difference. The easiest way to get to the top of the metaculus leaderboard is to forecast on everything. Not well, but just to do it. This is pretty dispiriting.
I did Metaculus for a while but I wasn’t quite sure how to assess how well I was doing and I lost interest. I know Brier score isn’t the greatest metric. Just try to accumulate points?
Not being able to mark predictions as quick and slow.
Sometimes I spend a long time on a forecast (10-20 mins) sometimes I spend about 30s. It frustrates me that all the current sites don’t let me track these two quite different types.
Forecasts are rather time-intensive—I used to make a bunch of metaculus forecasts, but putting serious effort into researching something, just to come up with a number to toss into the community mix, rarely seems worth the effort these days. Instead, I have a backlog of blog posts and other projects that seem higher expected value. (But going forwards, I should try to contribute forecasts when I find myself already researching something EA-adjacent anyways.)
Related to this, unclear impact—as Nuno judges here, it’s hard for forecasts to really hit the sweet spot of being at the right difficulty level and also being decision-relevant. If I had more of an understanding of who finds forecasts useful, I might be more interested in contributing.
Overthinking forecasts, causing writing them down & tracking them diligently to be too much of a mental-overhead for me to bother with.
As you say, there’s a tension between effortful vs informal forecasts: Out of humility / modesty / imposter syndrome, sometimes I don’t want to mess up a pool of expert forecasts by tossing out an ill-informed hunch.
I think Metaculus could benefit from adopting the more stock-market / prediction-market style mechanism of letting players stake different amounts of their total forecasting “budget” on individual questions… prediction-market style, experienced players or players who wanted to make a very confident prediction could pour lots of points into the system. Then I would be better able to distinguish my confident from my less-confident predictions, and small numbers of experts could bet their beliefs to overturn an ill-judged majority opinion.
Although I recognize that Metaculus’s current, more egalitarian style probably helps to grow the userbase and encourage participation.