Yes, you’re right about the ceiling effects. We started out with questions where science hasn’t established an answer, but where we guesstimated that the probability that the conservative/liberal answer is right was 50 % on average. That system wouldn’t have had these ceiling effects. We ran a pre-test of that test on Mechanical Turk, but the results were very askew (conservatives came out as much less biased than liberals, which was probably due to poorly constructed questions). I then decided to abandon that strategy for this one.
Here are two other posts I’ve written on this general strategy of inferring bias from belief structures. If you have any ideas of how to devise smarter ways to develop bias tests, I’d be very interested in that.
Thanks, Carl!
Yes, you’re right about the ceiling effects. We started out with questions where science hasn’t established an answer, but where we guesstimated that the probability that the conservative/liberal answer is right was 50 % on average. That system wouldn’t have had these ceiling effects. We ran a pre-test of that test on Mechanical Turk, but the results were very askew (conservatives came out as much less biased than liberals, which was probably due to poorly constructed questions). I then decided to abandon that strategy for this one.
Here are two other posts I’ve written on this general strategy of inferring bias from belief structures. If you have any ideas of how to devise smarter ways to develop bias tests, I’d be very interested in that.