This test, though it has good intentions, has several possible flaws:
I suppose it’s looking for a concordance between people’s political beliefs and their views on factual issues? But this type of concordance isn’t necessarily indicative of political bias: if someone believes certain facts, they’re going to base their political beliefs on those facts. For example, it would be bizarre if someone believed that the welfare state increases poverty but also identified as a Democrat. Like, “You do realize what the Democrats support, right?”
The “facts” in this test are not necessarily the product of widespread consensus. For example, did the stimulus package reduce unemployment compared to the counterfactual? I don’t think that’s a settled issue.
This test is easy to game, just as it’s easy to get a diagnosis of ADHD by answering “strongly agree” to statements like “I have trouble paying attention”. Many questions are clearly trick questions, which everyone knows how to answer.
There’s little opportunity to say “I don’t know.” Did renewable energy increase by 1 or 2 percent? Hell if I know. Is that considered a big difference? I just had to guess.
1) This question is discussed at length in the sections after the test.
2) It is according to our source. But some of the questions could have been better phrased. We will update them.
3) I wouldn’t say it’s easy to game. In fact, saying that it is a bias test had little effect in our Mechanical Turk pre-tests, which suggests that most people don’t try to game it. That said, it is possible to game. It is very hard to construct a test like this which is impossible to game. Other similar tests are far easier to game (see, e.g. Hans Rosling’s test of global progress which in effect is a bias test).
The test obviously isn’t going to be a reliable measure of bias if people try to game it—if they try to be more unbiased than they normally are. Still, taking the test could make these people think why they don’t normally adjust for their biases in this way. Hence the test could to some extent fulfill its ultimate purpose—getting people to think more about their biases and of the problem of political bias—even when it isn’t accurate as a measure of political bias.
4) True, but if you consistently guess in a direction that favours your political opinions, that suggests bias. That said, ideally it should be possible to indicate how strong your confidence in your answers is.
This test, though it has good intentions, has several possible flaws:
I suppose it’s looking for a concordance between people’s political beliefs and their views on factual issues? But this type of concordance isn’t necessarily indicative of political bias: if someone believes certain facts, they’re going to base their political beliefs on those facts. For example, it would be bizarre if someone believed that the welfare state increases poverty but also identified as a Democrat. Like, “You do realize what the Democrats support, right?”
The “facts” in this test are not necessarily the product of widespread consensus. For example, did the stimulus package reduce unemployment compared to the counterfactual? I don’t think that’s a settled issue.
This test is easy to game, just as it’s easy to get a diagnosis of ADHD by answering “strongly agree” to statements like “I have trouble paying attention”. Many questions are clearly trick questions, which everyone knows how to answer.
There’s little opportunity to say “I don’t know.” Did renewable energy increase by 1 or 2 percent? Hell if I know. Is that considered a big difference? I just had to guess.
1) This question is discussed at length in the sections after the test.
2) It is according to our source. But some of the questions could have been better phrased. We will update them.
3) I wouldn’t say it’s easy to game. In fact, saying that it is a bias test had little effect in our Mechanical Turk pre-tests, which suggests that most people don’t try to game it. That said, it is possible to game. It is very hard to construct a test like this which is impossible to game. Other similar tests are far easier to game (see, e.g. Hans Rosling’s test of global progress which in effect is a bias test).
The test obviously isn’t going to be a reliable measure of bias if people try to game it—if they try to be more unbiased than they normally are. Still, taking the test could make these people think why they don’t normally adjust for their biases in this way. Hence the test could to some extent fulfill its ultimate purpose—getting people to think more about their biases and of the problem of political bias—even when it isn’t accurate as a measure of political bias.
4) True, but if you consistently guess in a direction that favours your political opinions, that suggests bias. That said, ideally it should be possible to indicate how strong your confidence in your answers is.