I also feel sad that your comments feel slightly condescending or uncharitable, which makes it difficult for me to have a productive conversation.
I’m really sorry to come off that way, James. Please know it’s not my intention, but duly noted, and I’ll try to do better in the future.
Got it; that’s helpful to know, and thank you for taking the time to explain!
SDB is generally hard to test for post hoc, which is why it’s so important to design studies to avoid it. As the surveys suggest, not supporting protests doesn’t imply people don’t report support for climate action; so, for example, the responses about support for climate action could be biased upwards by the social desirability of climate action, even though those same respondents don’t support protests. Regardless, I don’t allege to know for certain these estimates are biased upwards (or downwards for that matter, in which case maybe the study is a false negative!). Instead, I’d argue the design itself is susceptible to social desirability and other biases. It’s difficult, if not impossible, to sort out how those biases affected the result, which is why I don’t find this study very informative. I’m curious why, if you think the results weren’t likely biased, you chose to down-weight it?
Understood; thank you for taking the time to clarify here. I agree this would be quite dubious. I don’t mean to be uncharitable in my interpretation: unfortunately, dubious research is the norm, and I’ve seen errors like this in the literature regularly. I’m glad they didn’t occur here!
Great, this makes sense and seems like standard practice. My misunderstanding arose from an error in the labeling of the tables: Uncertainty level 1 is labeled “highly uncertain,” but this is not the case for all values in that range. For example, suppose you were 1% confident that protests led to a large change. Contrary to the label, we would be quite certain protests did not lead to a large change. 20% confidence would make sense to label as highly uncertain as it reflects a uniform distribution of confidence across the five effect size bins. But confidences below that, in fact, reflect increasing certainty about the negation of the claim. I’d suggest using traditional confidence intervals here instead as they’re more familiar and standard, eg: We believe the average effects of protests on voting behavior is in the interval of [1, 8] percentage points with 90% confidence, or [3, 6] pp with 80% confidence.
Further adding to my confusion, the usage of “confidence interval” in “which can also be interpreted as 0-100% confidence intervals,” doesn’t reflect the standard usage of the term.
The reasons why I don’t find these critiques as highlighting significant methodological flaws is that:
Sorry, I think this was a miscommunication in our comments. I was referring to “Issues you raise are largely not severe nor methodological,” which gave me the impression you didn’t think the issues were related to the research methods. I understand your position here better.
Anyway, I’ll edit my top-level comment to reflect some of this new information; this generally updates me toward thinking this research may be more informative. I appreciate your taking the time to engage so thoroughly, and apologies again for giving an impression of anything less than the kindness and grace we should all aspire to.
I’m really sorry to come off that way, James. Please know it’s not my intention, but duly noted, and I’ll try to do better in the future.
Got it; that’s helpful to know, and thank you for taking the time to explain!
SDB is generally hard to test for post hoc, which is why it’s so important to design studies to avoid it. As the surveys suggest, not supporting protests doesn’t imply people don’t report support for climate action; so, for example, the responses about support for climate action could be biased upwards by the social desirability of climate action, even though those same respondents don’t support protests. Regardless, I don’t allege to know for certain these estimates are biased upwards (or downwards for that matter, in which case maybe the study is a false negative!). Instead, I’d argue the design itself is susceptible to social desirability and other biases. It’s difficult, if not impossible, to sort out how those biases affected the result, which is why I don’t find this study very informative. I’m curious why, if you think the results weren’t likely biased, you chose to down-weight it?
Understood; thank you for taking the time to clarify here. I agree this would be quite dubious. I don’t mean to be uncharitable in my interpretation: unfortunately, dubious research is the norm, and I’ve seen errors like this in the literature regularly. I’m glad they didn’t occur here!
Great, this makes sense and seems like standard practice. My misunderstanding arose from an error in the labeling of the tables: Uncertainty level 1 is labeled “highly uncertain,” but this is not the case for all values in that range. For example, suppose you were 1% confident that protests led to a large change. Contrary to the label, we would be quite certain protests did not lead to a large change. 20% confidence would make sense to label as highly uncertain as it reflects a uniform distribution of confidence across the five effect size bins. But confidences below that, in fact, reflect increasing certainty about the negation of the claim. I’d suggest using traditional confidence intervals here instead as they’re more familiar and standard, eg: We believe the average effects of protests on voting behavior is in the interval of [1, 8] percentage points with 90% confidence, or [3, 6] pp with 80% confidence.
Further adding to my confusion, the usage of “confidence interval” in “which can also be interpreted as 0-100% confidence intervals,” doesn’t reflect the standard usage of the term.
Sorry, I think this was a miscommunication in our comments. I was referring to “Issues you raise are largely not severe nor methodological,” which gave me the impression you didn’t think the issues were related to the research methods. I understand your position here better.
Anyway, I’ll edit my top-level comment to reflect some of this new information; this generally updates me toward thinking this research may be more informative. I appreciate your taking the time to engage so thoroughly, and apologies again for giving an impression of anything less than the kindness and grace we should all aspire to.