It’s indeed the case that GJP was 34.7% better than the ICPM. But it’s not the case that GJP participants were 34.7% better than intelligence analysts. The intelligent analyst used prediction markets that are generally worse than prediction pools (see Appendix A), so we are not comparing apples to apples.
It would be fair to judge IC for using prediction markets rather than prediction pools after seeing research coming out of GJP. But we don’t know how an intelligence analyst prediction pool would perform compared to the GJP prediction pool. We have reasons to believe that difference might not be that impressive based on ICPM vs GJP PM and based on Sell et al (2021).
The true performance difference between forecasters and CIA analysts with classified info (0%??)
What Goldstein found about a related but quite different quantity (34.7%)
What NPR etc reported (30%)
The important misconception is using (2) as if it was (1). Sentence A is about misunderstanding the relationship between the above three things, so it seems fine to use the number from (3). We haven’t seen anyone with misconceptions about the precise 34.7% figure and we’re not attributing the error to Goldstein et al.
Sorry, I’m confused. Do you mean the misconception is that rather than “30%” we should be saying that GJP was “34.7%” better than the ICPM?
It’s indeed the case that GJP was 34.7% better than the ICPM. But it’s not the case that GJP participants were 34.7% better than intelligence analysts. The intelligent analyst used prediction markets that are generally worse than prediction pools (see Appendix A), so we are not comparing apples to apples.
It would be fair to judge IC for using prediction markets rather than prediction pools after seeing research coming out of GJP. But we don’t know how an intelligence analyst prediction pool would perform compared to the GJP prediction pool. We have reasons to believe that difference might not be that impressive based on ICPM vs GJP PM and based on Sell et al (2021).
There’s three things
The true performance difference between forecasters and CIA analysts with classified info (0%??)
What Goldstein found about a related but quite different quantity (34.7%)
What NPR etc reported (30%)
The important misconception is using (2) as if it was (1). Sentence A is about misunderstanding the relationship between the above three things, so it seems fine to use the number from (3). We haven’t seen anyone with misconceptions about the precise 34.7% figure and we’re not attributing the error to Goldstein et al.