This is a very interesting approach, which I praise. However, the test has some ceiling effects (Sally Murray noted that the percentiles for a given score have been plummeting). It might help to have more and harder questions, or to use continuous variables to pick up finer distinctions.
Yes, you’re right about the ceiling effects. We started out with questions where science hasn’t established an answer, but where we guesstimated that the probability that the conservative/liberal answer is right was 50 % on average. That system wouldn’t have had these ceiling effects. We ran a pre-test of that test on Mechanical Turk, but the results were very askew (conservatives came out as much less biased than liberals, which was probably due to poorly constructed questions). I then decided to abandon that strategy for this one.
Here are two other posts I’ve written on this general strategy of inferring bias from belief structures. If you have any ideas of how to devise smarter ways to develop bias tests, I’d be very interested in that.
This is a very interesting approach, which I praise. However, the test has some ceiling effects (Sally Murray noted that the percentiles for a given score have been plummeting). It might help to have more and harder questions, or to use continuous variables to pick up finer distinctions.
Thanks, Carl!
Yes, you’re right about the ceiling effects. We started out with questions where science hasn’t established an answer, but where we guesstimated that the probability that the conservative/liberal answer is right was 50 % on average. That system wouldn’t have had these ceiling effects. We ran a pre-test of that test on Mechanical Turk, but the results were very askew (conservatives came out as much less biased than liberals, which was probably due to poorly constructed questions). I then decided to abandon that strategy for this one.
Here are two other posts I’ve written on this general strategy of inferring bias from belief structures. If you have any ideas of how to devise smarter ways to develop bias tests, I’d be very interested in that.