Great resource. The google doc guide is a really nice touch—very helpful. Two comments:
1) Re: ‘don’t know’ answers. A confidence/credence slider may help. This would allow people to give more fine-grained responses. Scores could be modified to some version of credence*correctness. Then you don’t punish those who don’t know they don’t know whether it was a 2% or 1% renewables increase (ignorant, not biased), and increase the punishment for those who are 100% sure that the wrong answer is correct (biased). This would retain the punishment for always leaning the same way. An example here: http://www.2pih.com/caltest/
2) The questions must be tough to phrase, but I found a couple ambiguous.
Q5 on foreign aid: ‘minor reason’ and ‘barely a reason’ sounded synonymous to me. I picked ‘minor’ because any aid has an opportunity cost to domestic spending. I know the source can’t really help with this.
Q6 on emissions. It may help to clarify that this is production, not consumption emissions. (Having briefly looked into it, however, this doesn’t seem to make much of a difference).
Q17 on emigration destination popularity. I read this as ‘US compared to [ALL COMBINED] other countries’, as opposed to ‘US compared to [INDIVIDUAL] other countries’, which changes the answer. Silly on my part, but might be worth clarifying.
Great resource. The google doc guide is a really nice touch—very helpful. Two comments:
1) Re: ‘don’t know’ answers. A confidence/credence slider may help. This would allow people to give more fine-grained responses. Scores could be modified to some version of credence*correctness. Then you don’t punish those who don’t know they don’t know whether it was a 2% or 1% renewables increase (ignorant, not biased), and increase the punishment for those who are 100% sure that the wrong answer is correct (biased). This would retain the punishment for always leaning the same way. An example here: http://www.2pih.com/caltest/
2) The questions must be tough to phrase, but I found a couple ambiguous.
Q5 on foreign aid: ‘minor reason’ and ‘barely a reason’ sounded synonymous to me. I picked ‘minor’ because any aid has an opportunity cost to domestic spending. I know the source can’t really help with this.
Q6 on emissions. It may help to clarify that this is production, not consumption emissions. (Having briefly looked into it, however, this doesn’t seem to make much of a difference).
Q17 on emigration destination popularity. I read this as ‘US compared to [ALL COMBINED] other countries’, as opposed to ‘US compared to [INDIVIDUAL] other countries’, which changes the answer. Silly on my part, but might be worth clarifying.