Hi there, Angel here, John’s co-founder who ran the study.
1) Question on “20 multiply imputed datasets”
a) What does this mean? What are you imputing and how are you imputing it? Excluding 3 participants who later withdrew, we had 114 participants in the trial. However, only 94 participants had complete data (pre & post for control; pre, post & follow-up for intervention). This is around 82% of complete cases. One way of handling missing data is multiple imputation, which makes plausible guesses about missing values using the information you do have. I used multiple imputation (via programming in R) to create 20 complete datasets of all 114 participants, with guesses based on each participant’s demographics (age, gender, group) and outcome scores (pre, post, and follow-up scores; follow-up for the intervention group only).
b) What are the results if you don’t do any imputation? See the attached graphs for the results of the complete-case analysis of 94 participants. The results are fairly similar:
Procrastination: The four-week intervention led to a statistically significant reduction in procrastination compared to the waitlist (p < .001, Cohen’s d = 1.56). Effect size is comparable to 10-week internet CBT guided by professional therapists (Rozental et al., 2017).
Life satisfaction: The four-week intervention led to a statistically significant increase in life satisfaction compared to the waitlist (p < .001, Cohen’s d = 0.95).
2) Question on 1-month follow-up effects
You raise a fair point. The 1-month follow-up was only collected for the intervention group. The waitlist control group began the intervention immediately after the waitlist period ended, and time constraints meant we couldn’t collect follow-up data from them.
Because of this, we didn’t compare intervention and control groups at follow-up. Instead, the claim that effects were maintained is based on within-group comparisons for the intervention group only (post-intervention vs. 1-month follow-up), using Hedges’ g rather than Cohen’s d to reflect this. We should have been clearer about that in the post.
You’re right that without a control group follow-up, we can’t rule out that scores would have continued to improve naturally over time. We’re running a larger RCT with a longer follow-up period in the future and plan to collect follow-up data from the control group too, which will let us make stronger claims about longer-term effects.
Hi there, Angel here, John’s co-founder who ran the study.
1) Question on “20 multiply imputed datasets”
a) What does this mean? What are you imputing and how are you imputing it?
Excluding 3 participants who later withdrew, we had 114 participants in the trial. However, only 94 participants had complete data (pre & post for control; pre, post & follow-up for intervention). This is around 82% of complete cases. One way of handling missing data is multiple imputation, which makes plausible guesses about missing values using the information you do have. I used multiple imputation (via programming in R) to create 20 complete datasets of all 114 participants, with guesses based on each participant’s demographics (age, gender, group) and outcome scores (pre, post, and follow-up scores; follow-up for the intervention group only).
b) What are the results if you don’t do any imputation?
See the attached graphs for the results of the complete-case analysis of 94 participants. The results are fairly similar:
Procrastination: The four-week intervention led to a statistically significant reduction in procrastination compared to the waitlist (p < .001, Cohen’s d = 1.56). Effect size is comparable to 10-week internet CBT guided by professional therapists (Rozental et al., 2017).
Life satisfaction: The four-week intervention led to a statistically significant increase in life satisfaction compared to the waitlist (p < .001, Cohen’s d = 0.95).
2) Question on 1-month follow-up effects
You raise a fair point. The 1-month follow-up was only collected for the intervention group. The waitlist control group began the intervention immediately after the waitlist period ended, and time constraints meant we couldn’t collect follow-up data from them.
Because of this, we didn’t compare intervention and control groups at follow-up. Instead, the claim that effects were maintained is based on within-group comparisons for the intervention group only (post-intervention vs. 1-month follow-up), using Hedges’ g rather than Cohen’s d to reflect this. We should have been clearer about that in the post.
You’re right that without a control group follow-up, we can’t rule out that scores would have continued to improve naturally over time. We’re running a larger RCT with a longer follow-up period in the future and plan to collect follow-up data from the control group too, which will let us make stronger claims about longer-term effects.