Overall I like this idea, appreciate the expansiveness of the considerations discussed in the post, and would excited to hear takes from people working at social media companies.
Thoughts on the post directly
Broadly, we envision i) automatically suggesting questions of likely interest to the user—e.g., questions related to the user’s current post or trending topics—and ii) rewarding users with higher than average forecasting accuracy with increased visibility
I think some version of some type of boosting visibility based on forecasting accuracy seems promising, but I feel uneasy about how this would be implemented. I’m concerned about (a) how this will be traded off with other qualities and (b) ensuring that current forecasting accuracy is actually a good proxy.
On (a), I think forecasting accuracy and the qualities it’s a proxy for represent a small subset of the space that determines which content I’d like to see promoted; e.g. it seems likely to be loosely correlated with writing quality. It may be tricky to strike the right balance in terms of how the promotion system works.
On (b):
Promoting and demoting content based on a small sample size of forecasts. In practice it often takes many resolved questions to discern which forecasters are more accurate, and I’m worried that it will be easy to increase/decrease visibility too early.
Even without a small sample size, there may be issues with many of the questions being correlated. I’m imagining a world in which lots of people predict on correlated questions about the 2016 presidential election, then Trump supporters get a huge boost in visibility after he wins because they do well on all of them.
That said, these issues can be mitigated with iteration on the forecasting feature if the people implementing it are careful and aware of these considerations.
Generally, it might be best if the recommendation algorithms don’t reward accurate forecasts in socially irrelevant domains such as sports—or reward them less so.
Insofar as the intent is to incentivize people to predict on more socially relevant domains, I agree. But I think forecasting accuracy on sports, etc. is likely strongly correlated with performance in other domains. Additionally, people may feel more comfortable forecasting on things like sports than other domains which may be more politically charged.
My experience with Facebook Forecast compared to Metaculus
I’ve been forecasting regularly on Metaculus for about 9 months and Forecast for about 1 month.
I don’t feel as pressured to regularly go back and update my old predictions on Forecast as on Metaculus since Forecast is a play-money prediction market rather than a prediction platform. On Metaculus if I predict 60% and the community is at 50%, then don’t update for 6 months and the community has over time moved to 95%, I’m at a huge disadvantage in terms of score relative to predictors who did update. But with a prediction market, if I buy shares at 50 cents and the price of the shares go up to 95 cents, it just helps me. The prediction market structure makes me feel less pressured to continually update on old questions, which has both its positives and negatives but seems good for a social media forecasting structure.
The aggregate on Forecast is often decent, but occasionally horrible more egregiously and more often than on Metaculus (e.g. this morning I bought some shares for Kelly Loeffler to win the Georgia senate runoff at as low as ~5 points implying 5% odds, while election betting odds currently have Loeffler at 62%). The most common reasons I’ve noticed are:
People misunderstand how the market works and bet on whichever outcome they think is most probable, regardless of the prices.
People don’t make the error described in (1) (that I can tell), but are over-confident.
People don’t read the resolution criteria carefully.
Political biases.
There aren’t many predictors so the aggregate can be swung easily.
As hinted at in the post, there’s an issue with being able to copy the best predictors. I’ve followed 2 of the top predictors on Forecast and usually agree with their analyses and buy into the same markets with the same positions.
Forecast currently gives points when other people forecast based on your “reasons” (aka comments), and these points are then aggregated on the leaderboard with points gained from actual predictions. I wish there were separate leaderboards for these.
would excited to hear takes from people working at social media companies.
Yeah, me too. For what it’s worth, Forecast mentions our post here.
On (a), I think forecasting accuracy and the qualities it’s a proxy for represent a small subset of the space that determines which content I’d like to see promoted
Yeah, as we discuss in this section, forecasting accuracy is surely not the most important thing. If it were up to me, I’d focus on spreading (sophisticated) content on, say, effective altruism, AI safety, and so on. Of course, most people would never agree with this. In contrast, forecasting is perhaps something almost everyone can get behind and is also objectively measurable.
I agree that the concerns you list under (b) need to be addressed.
Overall I like this idea, appreciate the expansiveness of the considerations discussed in the post, and would excited to hear takes from people working at social media companies.
Thoughts on the post directly
I think some version of some type of boosting visibility based on forecasting accuracy seems promising, but I feel uneasy about how this would be implemented. I’m concerned about (a) how this will be traded off with other qualities and (b) ensuring that current forecasting accuracy is actually a good proxy.
On (a), I think forecasting accuracy and the qualities it’s a proxy for represent a small subset of the space that determines which content I’d like to see promoted; e.g. it seems likely to be loosely correlated with writing quality. It may be tricky to strike the right balance in terms of how the promotion system works.
On (b):
Promoting and demoting content based on a small sample size of forecasts. In practice it often takes many resolved questions to discern which forecasters are more accurate, and I’m worried that it will be easy to increase/decrease visibility too early.
Even without a small sample size, there may be issues with many of the questions being correlated. I’m imagining a world in which lots of people predict on correlated questions about the 2016 presidential election, then Trump supporters get a huge boost in visibility after he wins because they do well on all of them.
That said, these issues can be mitigated with iteration on the forecasting feature if the people implementing it are careful and aware of these considerations.
Insofar as the intent is to incentivize people to predict on more socially relevant domains, I agree. But I think forecasting accuracy on sports, etc. is likely strongly correlated with performance in other domains. Additionally, people may feel more comfortable forecasting on things like sports than other domains which may be more politically charged.
My experience with Facebook Forecast compared to Metaculus
I’ve been forecasting regularly on Metaculus for about 9 months and Forecast for about 1 month.
I don’t feel as pressured to regularly go back and update my old predictions on Forecast as on Metaculus since Forecast is a play-money prediction market rather than a prediction platform. On Metaculus if I predict 60% and the community is at 50%, then don’t update for 6 months and the community has over time moved to 95%, I’m at a huge disadvantage in terms of score relative to predictors who did update. But with a prediction market, if I buy shares at 50 cents and the price of the shares go up to 95 cents, it just helps me. The prediction market structure makes me feel less pressured to continually update on old questions, which has both its positives and negatives but seems good for a social media forecasting structure.
The aggregate on Forecast is often decent, but occasionally horrible more egregiously and more often than on Metaculus (e.g. this morning I bought some shares for Kelly Loeffler to win the Georgia senate runoff at as low as ~5 points implying 5% odds, while election betting odds currently have Loeffler at 62%). The most common reasons I’ve noticed are:
People misunderstand how the market works and bet on whichever outcome they think is most probable, regardless of the prices.
People don’t make the error described in (1) (that I can tell), but are over-confident.
People don’t read the resolution criteria carefully.
Political biases.
There aren’t many predictors so the aggregate can be swung easily.
As hinted at in the post, there’s an issue with being able to copy the best predictors. I’ve followed 2 of the top predictors on Forecast and usually agree with their analyses and buy into the same markets with the same positions.
Forecast currently gives points when other people forecast based on your “reasons” (aka comments), and these points are then aggregated on the leaderboard with points gained from actual predictions. I wish there were separate leaderboards for these.
Thanks, great points!
Yeah, me too. For what it’s worth, Forecast mentions our post here.
Yeah, as we discuss in this section, forecasting accuracy is surely not the most important thing. If it were up to me, I’d focus on spreading (sophisticated) content on, say, effective altruism, AI safety, and so on. Of course, most people would never agree with this. In contrast, forecasting is perhaps something almost everyone can get behind and is also objectively measurable.
I agree that the concerns you list under (b) need to be addressed.