1) This question is discussed at length in the sections after the test.
2) It is according to our source. But some of the questions could have been better phrased. We will update them.
3) I wouldn’t say it’s easy to game. In fact, saying that it is a bias test had little effect in our Mechanical Turk pre-tests, which suggests that most people don’t try to game it. That said, it is possible to game. It is very hard to construct a test like this which is impossible to game. Other similar tests are far easier to game (see, e.g. Hans Rosling’s test of global progress which in effect is a bias test).
The test obviously isn’t going to be a reliable measure of bias if people try to game it—if they try to be more unbiased than they normally are. Still, taking the test could make these people think why they don’t normally adjust for their biases in this way. Hence the test could to some extent fulfill its ultimate purpose—getting people to think more about their biases and of the problem of political bias—even when it isn’t accurate as a measure of political bias.
4) True, but if you consistently guess in a direction that favours your political opinions, that suggests bias. That said, ideally it should be possible to indicate how strong your confidence in your answers is.
1) This question is discussed at length in the sections after the test.
2) It is according to our source. But some of the questions could have been better phrased. We will update them.
3) I wouldn’t say it’s easy to game. In fact, saying that it is a bias test had little effect in our Mechanical Turk pre-tests, which suggests that most people don’t try to game it. That said, it is possible to game. It is very hard to construct a test like this which is impossible to game. Other similar tests are far easier to game (see, e.g. Hans Rosling’s test of global progress which in effect is a bias test).
The test obviously isn’t going to be a reliable measure of bias if people try to game it—if they try to be more unbiased than they normally are. Still, taking the test could make these people think why they don’t normally adjust for their biases in this way. Hence the test could to some extent fulfill its ultimate purpose—getting people to think more about their biases and of the problem of political bias—even when it isn’t accurate as a measure of political bias.
4) True, but if you consistently guess in a direction that favours your political opinions, that suggests bias. That said, ideally it should be possible to indicate how strong your confidence in your answers is.