Unless a study is done with participants who are selected heavily for numeracy and fluency in probabilities, I would not interpret stated probabilities literally as a numerical representation of their beliefs, especially near the extremes of the scale. People are giving an answer that vaguely feels like it matches the degree of unlikeliness that they feel, but they don’t have that clear a sense of what (e.g.) a probability of 1⁄100 means. That’s why studies can get such drastically different answers depending on the response format, and why (I predict) effects like scope insensitivity are likely to show up.
I wouldn’t expect the confidence question to pick up on this. e.g., Suppose that experts think that something has a 1 in a million chance and a person basically agrees with the experts’ viewpoint but hasn’t heard/remembered that number. So they indicate “that’s very unlikely” by entering “1%” which feels like it’s basically the bottom of the scale. Then on the confidence question they say that they’re very confident of that answer because they feel sure that it’s very unlikely.
Unless a study is done with participants who are selected heavily for numeracy and fluency in probabilities, I would not interpret stated probabilities literally as a numerical representation of their beliefs, especially near the extremes of the scale. People are giving an answer that vaguely feels like it matches the degree of unlikeliness that they feel, but they don’t have that clear a sense of what (e.g.) a probability of 1⁄100 means. That’s why studies can get such drastically different answers depending on the response format, and why (I predict) effects like scope insensitivity are likely to show up.
I wouldn’t expect the confidence question to pick up on this. e.g., Suppose that experts think that something has a 1 in a million chance and a person basically agrees with the experts’ viewpoint but hasn’t heard/remembered that number. So they indicate “that’s very unlikely” by entering “1%” which feels like it’s basically the bottom of the scale. Then on the confidence question they say that they’re very confident of that answer because they feel sure that it’s very unlikely.