I only skimmed your post (let me know if I’m misunderstanding) but I have an issue with this idea. Many forecasts require complicated mathematical models to describe. You can’t simply link to sources. You also need to link to a model. Blog posts/txt files, which are essentially what the forum is, are extremely hard to scrape and parse unless everyone starts adopting conventions. So you max you functionality out at linking, this isn’t very automated.
If you are recommending connecting a full mathematical model from the forum, let me suggest that rather than connecting Metaculus to the forum, you connect it to https://www.getguesstimate.com/models, as this is much more scalable and clear.
thank you for thinking about these things, it inspired me to make my own post.
I would say it a little differently. I would say that “judgmental” forecasting, the kind typically done on Metaculus or Good Judgement Open or similar platforms, CAN involve mathemtical models, but oftentimes people are just doing some simple math, if any at all. In cases where people do use models, sure it would make sense to link to them as sources, and I agree that would also be valuable to track for similar reasons. Guesstimate seems like the obvious place to do that.
I think that is separate from the proposition I intended to communicate for primarily text based research.
I also wasn’t anticipating any need to do scraping if this was implemented by the two platforms themselves. It should be easy enough for them to tell if a citation is linking to an EA forum post? Metaculus doesn’t have a footnote/citation formatting tool today like the EA Forum’s. (Although if you were to scrape, finding EA forum links within citations on this forum seems pretty well defined and achievable? idk, I don’t write much code, thus me floating this out here for feedback)
I would say we are basically on the exact same page in terms of the overall vision. I’m also trying to get at these logical chains of information that we can travel backwards through to easily sanity check and also do data analysis.
Where I think we break is if there is no underlying structure to these logical chains outside of a bunch of arrows pointing between links, it reduces our ability to automate and take away insights.
A few examples
you link to a ea forum post with multiple claims. In order to build logical chains, we now need a database to store each claim in each post. In order to do this, we now need to convince everyone to use certain formatting on claims or try to use an LLM to parse.
you link multiple sources, which themselves link multiple sources. Since linking is just drawing arrows in an abstract sense, I have no ability to discern how much each source went into the guess. I assume we would just use a uniform distribution to model how much each source went into the final guess? but this is clearly terribly off in many cases so we lose a lot of information.
If we link to models we hold a lot more information down the chain.
Overall I wouldn’t say my proposition isn’t a full substitute for your idea, but I think there is overlapping functionality.
Few Things
I only skimmed your post (let me know if I’m misunderstanding) but I have an issue with this idea. Many forecasts require complicated mathematical models to describe. You can’t simply link to sources. You also need to link to a model. Blog posts/txt files, which are essentially what the forum is, are extremely hard to scrape and parse unless everyone starts adopting conventions. So you max you functionality out at linking, this isn’t very automated.
If you are recommending connecting a full mathematical model from the forum, let me suggest that rather than connecting Metaculus to the forum, you connect it to https://www.getguesstimate.com/models, as this is much more scalable and clear.
thank you for thinking about these things, it inspired me to make my own post.
I would say it a little differently. I would say that “judgmental” forecasting, the kind typically done on Metaculus or Good Judgement Open or similar platforms, CAN involve mathemtical models, but oftentimes people are just doing some simple math, if any at all. In cases where people do use models, sure it would make sense to link to them as sources, and I agree that would also be valuable to track for similar reasons. Guesstimate seems like the obvious place to do that.
I think that is separate from the proposition I intended to communicate for primarily text based research.
I also wasn’t anticipating any need to do scraping if this was implemented by the two platforms themselves. It should be easy enough for them to tell if a citation is linking to an EA forum post? Metaculus doesn’t have a footnote/citation formatting tool today like the EA Forum’s. (Although if you were to scrape, finding EA forum links within citations on this forum seems pretty well defined and achievable? idk, I don’t write much code, thus me floating this out here for feedback)
Thanks for the thoughts!
I would say we are basically on the exact same page in terms of the overall vision. I’m also trying to get at these logical chains of information that we can travel backwards through to easily sanity check and also do data analysis.
Where I think we break is if there is no underlying structure to these logical chains outside of a bunch of arrows pointing between links, it reduces our ability to automate and take away insights.
A few examples
you link to a ea forum post with multiple claims. In order to build logical chains, we now need a database to store each claim in each post. In order to do this, we now need to convince everyone to use certain formatting on claims or try to use an LLM to parse.
you link multiple sources, which themselves link multiple sources. Since linking is just drawing arrows in an abstract sense, I have no ability to discern how much each source went into the guess. I assume we would just use a uniform distribution to model how much each source went into the final guess? but this is clearly terribly off in many cases so we lose a lot of information.
If we link to models we hold a lot more information down the chain.
Overall I wouldn’t say my proposition isn’t a full substitute for your idea, but I think there is overlapping functionality.