Not sure what the finding here is: ”...the 30% edge is likely partly due to the different aggregation techniques used....” [emphasis mine]
How can we know more than likely partly? On what basis can we make a determination? Goldstein et. al. posit several hypotheses for the 30% advantage Good Judgment had over the ICPM: 1) GJ folks were paid; 2) a “secrecy heuristic” posited by Travers et. al.; 3) aggregation algorithms; 4) etc.
Have you disaggregated these effects such that we can know the extent to which the aggregation techniques boosted accuracy? Maybe the effect was entirely related to the $150 Amazon gift cards that GJ forecasters received for 12 months work? Maybe the “secrecy heuristic” explains the delta?
Thank you, Tim! Likely partly due to is my impressions of what’s going on based on existing research; I think we know that it is “likely partly” but probably not much more based on current literature.
The line of reasoning which I find plausible is “GJP PM and GJP All Surveys Logit” is more or less the same pool of people but the one aggregation algorithm is much better than another; it’s plausible that “IC All Surveys Logit would improve on ICPM quite dramatically.” And because the difference between GJP PM and ICPM is small it feels plausible that if the best aggregation method would be applied to IC, IC would cut the aforementioned 30% gap.
(I am happy to change my mind upon seeing more research comparing strong forecasters and domain experts.)
Hi again Misha,
Not sure what the finding here is: ”...the 30% edge is likely partly due to the different aggregation techniques used....” [emphasis mine]
How can we know more than likely partly? On what basis can we make a determination? Goldstein et. al. posit several hypotheses for the 30% advantage Good Judgment had over the ICPM: 1) GJ folks were paid; 2) a “secrecy heuristic” posited by Travers et. al.; 3) aggregation algorithms; 4) etc.
Have you disaggregated these effects such that we can know the extent to which the aggregation techniques boosted accuracy? Maybe the effect was entirely related to the $150 Amazon gift cards that GJ forecasters received for 12 months work? Maybe the “secrecy heuristic” explains the delta?
Thank you, Tim! Likely partly due to is my impressions of what’s going on based on existing research; I think we know that it is “likely partly” but probably not much more based on current literature.
The line of reasoning which I find plausible is “GJP PM and GJP All Surveys Logit” is more or less the same pool of people but the one aggregation algorithm is much better than another; it’s plausible that “IC All Surveys Logit would improve on ICPM quite dramatically.” And because the difference between GJP PM and ICPM is small it feels plausible that if the best aggregation method would be applied to IC, IC would cut the aforementioned 30% gap.
(I am happy to change my mind upon seeing more research comparing strong forecasters and domain experts.)