People likely read it as a standalone statement without referring back to the megapost, and gave “select” its most common meaning in ordinary jargon. I agree that the wording of these items is tricky and can skew outcomes, I just feel the summary here did not accurately capture what the broader statement said. So I am not convinced that voters were actually inconsistent or that this finding represents a deep problem with this kind of sorting exercise.
To be clear, you’re saying that Nathan took the megapost out of context in a way that suggested a different interpretation of their words, which lead to a highly downvoted answer. (I’m not suggesting he did this on purpose). In other words, the framing of an answer has a large effect on the final result.
I think this does represent a problem with the sorting exercise. If it hadn’t been for my followup, the takeaway could have easily been “EA doesn’t like diversity”, when the actual takeaway is “EA likes diversity, but doesn’t like this one specific hiring tactic, which was never actually mentioned anywhere”.
Yes. We may not be that far apart on this one now. The validity of the results is only as good as the extent to which the answer stems accurately convey what you are trying to measure.
Although I understand why Nathan wrote it as he did, this answer stem isn’t (in my opinion) a good reflection of the underlying text because that text used “select” in a less common way that is only clear in context. Thus, the response to the stem only has validity, at most, for what the stem itself actually says.
I think the need for a summary to accurately reflect the idea in question is endemic to all attempts to gauge opinion, not just this method. Writing good summaries can be hard.
People likely read it as a standalone statement without referring back to the megapost, and gave “select” its most common meaning in ordinary jargon. I agree that the wording of these items is tricky and can skew outcomes, I just feel the summary here did not accurately capture what the broader statement said. So I am not convinced that voters were actually inconsistent or that this finding represents a deep problem with this kind of sorting exercise.
To be clear, you’re saying that Nathan took the megapost out of context in a way that suggested a different interpretation of their words, which lead to a highly downvoted answer. (I’m not suggesting he did this on purpose). In other words, the framing of an answer has a large effect on the final result.
I think this does represent a problem with the sorting exercise. If it hadn’t been for my followup, the takeaway could have easily been “EA doesn’t like diversity”, when the actual takeaway is “EA likes diversity, but doesn’t like this one specific hiring tactic, which was never actually mentioned anywhere”.
Yes. We may not be that far apart on this one now. The validity of the results is only as good as the extent to which the answer stems accurately convey what you are trying to measure.
Although I understand why Nathan wrote it as he did, this answer stem isn’t (in my opinion) a good reflection of the underlying text because that text used “select” in a less common way that is only clear in context. Thus, the response to the stem only has validity, at most, for what the stem itself actually says.
I think the need for a summary to accurately reflect the idea in question is endemic to all attempts to gauge opinion, not just this method. Writing good summaries can be hard.