Hi, happy to speak to the methodological points here.
Thanks for sharing the link and suggestion. We agree that understanding how much we can trust the data is crucial for interpreting our results, so thank you for engaging with this critically.
We didn’t measure individual reaction times for questions so using RT modelling isn’t an option. Modelling carelessness in other ways (e.g., modelling it as a latent tendency) would be fascinating, but I don’t endorse the assumptions we’d need to model carelessness as a latent variable. (I like how Rohrer & Paulewicz 2025 point out that latent variable modelling requires making some strong assumptions.) So even though we could try to model carelessness, I currently don’t think we should do so.
The headline results were designed such that it would be very unlikely for participants selecting at random to meet our definition of “consistent and concerning” endorsers. In the main piece, the focus is on who agreed with the hell question, AND selected “Forever” for the duration question (the last of 11 options), AND selected 1% or more for the proportion question. The supplementary materials additionally report those who endorse BOTH the “endorses system” question and “would create” question, AND selected “Forever” for the duration question (the last of 11 options), AND selected 1% or more for the proportion question. Partly due to the design of these “headline” results variables, the proportion of the sample meeting our definition of “consistent and concerning” turned out to be robust to the removal of all attention checks (i.e., the inclusion of everyone who didn’t drop out before the questions of interest, with no filtering).
Regarding the other things in the list you linked to from that LLM chat, though, we did do most of those things. For example, we included unobtrusive checks and multiple different quality measures, not just attention checks—I’d be interested in your thoughts on the checks outlined in our supplementary folder. And importantly, for our headline results, we did sensitivity analyses and shared the results (including confidence intervals) in our supplementary materials folder.
(Also, just to address the point about the N varying between questions—that’s because different numbers of participants completed some questions; slightly fewer completed the duration and hell questions because we had tested different wordings for both questions early in the study, before they were replaced with new versions that were used for the rest of the study.)
Would be happy to answer follow-up questions too. Thanks!
Hi, happy to speak to the methodological points here.
Thanks for sharing the link and suggestion. We agree that understanding how much we can trust the data is crucial for interpreting our results, so thank you for engaging with this critically.
We didn’t measure individual reaction times for questions so using RT modelling isn’t an option. Modelling carelessness in other ways (e.g., modelling it as a latent tendency) would be fascinating, but I don’t endorse the assumptions we’d need to model carelessness as a latent variable. (I like how Rohrer & Paulewicz 2025 point out that latent variable modelling requires making some strong assumptions.) So even though we could try to model carelessness, I currently don’t think we should do so.
The headline results were designed such that it would be very unlikely for participants selecting at random to meet our definition of “consistent and concerning” endorsers. In the main piece, the focus is on who agreed with the hell question, AND selected “Forever” for the duration question (the last of 11 options), AND selected 1% or more for the proportion question. The supplementary materials additionally report those who endorse BOTH the “endorses system” question and “would create” question, AND selected “Forever” for the duration question (the last of 11 options), AND selected 1% or more for the proportion question. Partly due to the design of these “headline” results variables, the proportion of the sample meeting our definition of “consistent and concerning” turned out to be robust to the removal of all attention checks (i.e., the inclusion of everyone who didn’t drop out before the questions of interest, with no filtering).
Regarding the other things in the list you linked to from that LLM chat, though, we did do most of those things. For example, we included unobtrusive checks and multiple different quality measures, not just attention checks—I’d be interested in your thoughts on the checks outlined in our supplementary folder. And importantly, for our headline results, we did sensitivity analyses and shared the results (including confidence intervals) in our supplementary materials folder.
(Also, just to address the point about the N varying between questions—that’s because different numbers of participants completed some questions; slightly fewer completed the duration and hell questions because we had tested different wordings for both questions early in the study, before they were replaced with new versions that were used for the rest of the study.)
Would be happy to answer follow-up questions too. Thanks!