I think this idea is really cool (albeit hard to pull off successfully)! You’re definitely not the first person to think of it, but I don’t know of any comparable efforts that have turned into actual products yet. I think the biggest challenge will be maintaining the right ratio of models to estimators, as you could very easily have the former outrun the latter without some kind of subsidy for people’s time. There’s already a challenge around recruiting and retaining talented forecasters, and this sort of estimation may be in some ways more cognitively demanding / less rewarding for the participants. So you might want to have a setup where the impact models are tightly curated and there are incentives to attract estimators, at least at the beginning.
If you end up moving forward with a prototype, I’d be interested in providing input on the product design as an alpha user.
Seems like I forgot to change “last updated 04.january 2021” to “last updated 04. january 2022″ when I made changes in january haha.
I am still working on this. I agree with Ozzie’s comment below that doing a small part of this well is the best way to make progress. We are currently looking at the UX part of things. As I describe under this heading in the doc, I don’t think it is feasible to expect many non-expert forecasters to enter a platform to give their credences on claims. And the expert forecasters are, as Ian mentions below, in short supply. Therefore, we are trying to make it easier to give credences on issues while reading about them the same place you read about them. I tested this idea out in a small experiment this fall (with google docs), and it does seem like motivated people who would not enter prediction platforms to forecast issues might give their takes if elicited this way. Right now we are investigating this idea further through an mvp of a browser extension that lets users give credences on claims found in texts on the web. We will experiment some more with this during the fall. A more tractable version of the long doc is likely to appear at the forum at some point.
I’m not wedded to the concrete ideas presented in the doc, I just happen to think they are good ways to move closer to the grand vision. I’d be happy to help any project moving in that direction:)
I think this idea is really cool (albeit hard to pull off successfully)! You’re definitely not the first person to think of it, but I don’t know of any comparable efforts that have turned into actual products yet. I think the biggest challenge will be maintaining the right ratio of models to estimators, as you could very easily have the former outrun the latter without some kind of subsidy for people’s time. There’s already a challenge around recruiting and retaining talented forecasters, and this sort of estimation may be in some ways more cognitively demanding / less rewarding for the participants. So you might want to have a setup where the impact models are tightly curated and there are incentives to attract estimators, at least at the beginning.
If you end up moving forward with a prototype, I’d be interested in providing input on the product design as an alpha user.
Actually, come to think of it, the S-Process used for the Survival and Flourishing Fund is an implementation of one version of this idea.
Thanks, that’s good feedback. I will check out the linked video. If you know anyone to get in touch with, I’d be keen to talk to them.
My guess is that this idea has been independently thought about many times, and if it’s not bad for some reason, already funded.
I’ve found a similar project by Paal Kvarberg, described in this document here: https://docs.google.com/document/d/1An-NGrQUPSJ4v8HdsZO-BcAq5NpyrEqv32rdk-N4guo/edit#
I think he didn’t pursue it—the document hasn’t been updated for a year.
Seems like I forgot to change “last updated 04.january 2021” to “last updated 04. january 2022″ when I made changes in january haha.
I am still working on this. I agree with Ozzie’s comment below that doing a small part of this well is the best way to make progress. We are currently looking at the UX part of things. As I describe under this heading in the doc, I don’t think it is feasible to expect many non-expert forecasters to enter a platform to give their credences on claims. And the expert forecasters are, as Ian mentions below, in short supply. Therefore, we are trying to make it easier to give credences on issues while reading about them the same place you read about them. I tested this idea out in a small experiment this fall (with google docs), and it does seem like motivated people who would not enter prediction platforms to forecast issues might give their takes if elicited this way. Right now we are investigating this idea further through an mvp of a browser extension that lets users give credences on claims found in texts on the web. We will experiment some more with this during the fall. A more tractable version of the long doc is likely to appear at the forum at some point.
I’m not wedded to the concrete ideas presented in the doc, I just happen to think they are good ways to move closer to the grand vision. I’d be happy to help any project moving in that direction:)