Can someone clarify these statements from Summary (3a)? They seem to be at odds....
A: “A common misconception is that superforecasters outperformed intelligence analysts by 30%.”
B: “Instead: Goldstein et al showed that superforecasters outperformed the intelligence community...”[then a table listing the ICPM MMDB as 0.23 versus the GJP Best MMDB as 0.15].
Indeed, but the misconception/lack of nuance is specifically about 30% here is Wikipedia on Good Judgement Project. I guess it’s either about looking at preliminary data or rounding.
The top forecasters in GJP are “reportedly 30% better than intelligence officers with access to actual classified information.”
It’s indeed the case that GJP was 34.7% better than the ICPM. But it’s not the case that GJP participants were 34.7% better than intelligence analysts. The intelligent analyst used prediction markets that are generally worse than prediction pools (see Appendix A), so we are not comparing apples to apples.
It would be fair to judge IC for using prediction markets rather than prediction pools after seeing research coming out of GJP. But we don’t know how an intelligence analyst prediction pool would perform compared to the GJP prediction pool. We have reasons to believe that difference might not be that impressive based on ICPM vs GJP PM and based on Sell et al (2021).
The true performance difference between forecasters and CIA analysts with classified info (0%??)
What Goldstein found about a related but quite different quantity (34.7%)
What NPR etc reported (30%)
The important misconception is using (2) as if it was (1). Sentence A is about misunderstanding the relationship between the above three things, so it seems fine to use the number from (3). We haven’t seen anyone with misconceptions about the precise 34.7% figure and we’re not attributing the error to Goldstein et al.
Can someone clarify these statements from Summary (3a)? They seem to be at odds....
A: “A common misconception is that superforecasters outperformed intelligence analysts by 30%.”
B: “Instead: Goldstein et al showed that superforecasters outperformed the intelligence community...”[then a table listing the ICPM MMDB as 0.23 versus the GJP Best MMDB as 0.15].
--> Wouldn’t that be 34% better?
Indeed, but the misconception/lack of nuance is specifically about 30% here is Wikipedia on Good Judgement Project. I guess it’s either about looking at preliminary data or rounding.
It is, but we’re talking about the misconception, which became “30 percent” in (e.g.) this article.
Sorry, I’m confused. Do you mean the misconception is that rather than “30%” we should be saying that GJP was “34.7%” better than the ICPM?
It’s indeed the case that GJP was 34.7% better than the ICPM. But it’s not the case that GJP participants were 34.7% better than intelligence analysts. The intelligent analyst used prediction markets that are generally worse than prediction pools (see Appendix A), so we are not comparing apples to apples.
It would be fair to judge IC for using prediction markets rather than prediction pools after seeing research coming out of GJP. But we don’t know how an intelligence analyst prediction pool would perform compared to the GJP prediction pool. We have reasons to believe that difference might not be that impressive based on ICPM vs GJP PM and based on Sell et al (2021).
There’s three things
The true performance difference between forecasters and CIA analysts with classified info (0%??)
What Goldstein found about a related but quite different quantity (34.7%)
What NPR etc reported (30%)
The important misconception is using (2) as if it was (1). Sentence A is about misunderstanding the relationship between the above three things, so it seems fine to use the number from (3). We haven’t seen anyone with misconceptions about the precise 34.7% figure and we’re not attributing the error to Goldstein et al.