Exciting results! Is the RCT written up in more detail anywhere? I’m confused by the current reporting because it seems to compare pre-post intervention results, rather than treatments and controls. Also, how was the control group set up? (Eg, no intervention, active control, waitlist control etc)
Quick answers Control group was a four-week waitlist. 1-month follow-up was only collected for intervention group as the waitlist group started the intervention as soon as waitlist ended, plus time constraints for the project.
When analysing the results with 20 multiply imputed datasets:
1. Procrastination: Statistically significant pre-post reduction in intervention compared to waitlist (p < .001, Cohen’s d = 1.52). Within intervention group, further small reduction at one-month follow-up compared to post-intervention (n = 47, p < .001, Hedges’ g = 0.36).
2. Life satisfaction: Statistically significant pre-post increase in intervention compared to waitlist (p < .001, Cohen’s d = 0.93). Within intervention group, further small increase at one-month follow-up compared to post-intervention (n = 47, p < .001, Hedges’ g = 0.42).
A couple of further questions that would help me interpret the results:
”with 20 multiply imputed datasets”—What does this mean? What are you imputing and how are you imputing it? What are the results if you don’t do any imputation?
How can you say the effect strengthens or is maintained after 1 month if you don’t observe the control group outcomes after 1 month? Generally, you see control group outcomes continue to improve over time even if they don’t get treated (as you can see by doing control group pre-post comparisons for every outcome), so doesn’t seem like you can claim much about whether the effect grows or shrinks over time.
Hi there, Angel here, John’s co-founder who ran the study.
1) Question on “20 multiply imputed datasets”
a) What does this mean? What are you imputing and how are you imputing it? Excluding 3 participants who later withdrew, we had 114 participants in the trial. However, only 94 participants had complete data (pre & post for control; pre, post & follow-up for intervention). This is around 82% of complete cases. One way of handling missing data is multiple imputation, which makes plausible guesses about missing values using the information you do have. I used multiple imputation (via programming in R) to create 20 complete datasets of all 114 participants, with guesses based on each participant’s demographics (age, gender, group) and outcome scores (pre, post, and follow-up scores; follow-up for the intervention group only).
b) What are the results if you don’t do any imputation? See the attached graphs for the results of the complete-case analysis of 94 participants. The results are fairly similar:
Procrastination: The four-week intervention led to a statistically significant reduction in procrastination compared to the waitlist (p < .001, Cohen’s d = 1.56). Effect size is comparable to 10-week internet CBT guided by professional therapists (Rozental et al., 2017).
Life satisfaction: The four-week intervention led to a statistically significant increase in life satisfaction compared to the waitlist (p < .001, Cohen’s d = 0.95).
2) Question on 1-month follow-up effects
You raise a fair point. The 1-month follow-up was only collected for the intervention group. The waitlist control group began the intervention immediately after the waitlist period ended, and time constraints meant we couldn’t collect follow-up data from them.
Because of this, we didn’t compare intervention and control groups at follow-up. Instead, the claim that effects were maintained is based on within-group comparisons for the intervention group only (post-intervention vs. 1-month follow-up), using Hedges’ g rather than Cohen’s d to reflect this. We should have been clearer about that in the post.
You’re right that without a control group follow-up, we can’t rule out that scores would have continued to improve naturally over time. We’re running a larger RCT with a longer follow-up period in the future and plan to collect follow-up data from the control group too, which will let us make stronger claims about longer-term effects.
Exciting results! Is the RCT written up in more detail anywhere? I’m confused by the current reporting because it seems to compare pre-post intervention results, rather than treatments and controls. Also, how was the control group set up? (Eg, no intervention, active control, waitlist control etc)
Quick answers
Control group was a four-week waitlist. 1-month follow-up was only collected for intervention group as the waitlist group started the intervention as soon as waitlist ended, plus time constraints for the project.
When analysing the results with 20 multiply imputed datasets:
1. Procrastination: Statistically significant pre-post reduction in intervention compared to waitlist (p < .001, Cohen’s d = 1.52). Within intervention group, further small reduction at one-month follow-up compared to post-intervention (n = 47, p < .001, Hedges’ g = 0.36).
2. Life satisfaction: Statistically significant pre-post increase in intervention compared to waitlist (p < .001, Cohen’s d = 0.93). Within intervention group, further small increase at one-month follow-up compared to post-intervention (n = 47, p < .001, Hedges’ g = 0.42).
Detailed version: https://www.canva.com/design/DAGqbQzPKJY/8NkRiubgsgDUpDPIbIkwqg/edit?utm_content=DAGqbQzPKJY&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton
A couple of further questions that would help me interpret the results:
”with 20 multiply imputed datasets”—What does this mean? What are you imputing and how are you imputing it? What are the results if you don’t do any imputation?
How can you say the effect strengthens or is maintained after 1 month if you don’t observe the control group outcomes after 1 month? Generally, you see control group outcomes continue to improve over time even if they don’t get treated (as you can see by doing control group pre-post comparisons for every outcome), so doesn’t seem like you can claim much about whether the effect grows or shrinks over time.
Hi there, Angel here, John’s co-founder who ran the study.
1) Question on “20 multiply imputed datasets”
a) What does this mean? What are you imputing and how are you imputing it?
Excluding 3 participants who later withdrew, we had 114 participants in the trial. However, only 94 participants had complete data (pre & post for control; pre, post & follow-up for intervention). This is around 82% of complete cases. One way of handling missing data is multiple imputation, which makes plausible guesses about missing values using the information you do have. I used multiple imputation (via programming in R) to create 20 complete datasets of all 114 participants, with guesses based on each participant’s demographics (age, gender, group) and outcome scores (pre, post, and follow-up scores; follow-up for the intervention group only).
b) What are the results if you don’t do any imputation?
See the attached graphs for the results of the complete-case analysis of 94 participants. The results are fairly similar:
Procrastination: The four-week intervention led to a statistically significant reduction in procrastination compared to the waitlist (p < .001, Cohen’s d = 1.56). Effect size is comparable to 10-week internet CBT guided by professional therapists (Rozental et al., 2017).
Life satisfaction: The four-week intervention led to a statistically significant increase in life satisfaction compared to the waitlist (p < .001, Cohen’s d = 0.95).
2) Question on 1-month follow-up effects
You raise a fair point. The 1-month follow-up was only collected for the intervention group. The waitlist control group began the intervention immediately after the waitlist period ended, and time constraints meant we couldn’t collect follow-up data from them.
Because of this, we didn’t compare intervention and control groups at follow-up. Instead, the claim that effects were maintained is based on within-group comparisons for the intervention group only (post-intervention vs. 1-month follow-up), using Hedges’ g rather than Cohen’s d to reflect this. We should have been clearer about that in the post.
You’re right that without a control group follow-up, we can’t rule out that scores would have continued to improve naturally over time. We’re running a larger RCT with a longer follow-up period in the future and plan to collect follow-up data from the control group too, which will let us make stronger claims about longer-term effects.