Thanks for sharing this, Aaron! Really interesting pilot work.
One quick thought—I wouldn’t rely too heavily on statistical significance tests, particularly with small sample sizes. P-values are largely a function of sample size, and it’s nearly impossible to get statistical significance with 44 participants (unless your effect size is huge!).
Speaking of effect sizes, it seems like you powered to detect an effect of d=0.7. For a messaging study with rather subtle manipulations, an effect of d=0.7 seems huge! I would be pretty impressed if giving people CE info resulted in an effect size of d=0.2 or d=0.3, for instance. I’m guessing you were constrained by the # of participants you could recruit (which is quite reasonable—lots of pilot studies are underpowered). But given the low power, I’d be reluctant to draw strong conclusions.
I also appreciate that you reported the mean scores in the results section of your paper, which allowed me to skim to see if there’s anything interesting. I think there might be!
There was no significant difference in Effective Donation between the Info (M = 80.21, SD = 18.79) and No Info (M = 71.79, SD = 17.05) conditions, F(1, 34) = 1.85, p = .183, ηp2 = .052.
If this effect is real, I think this is pretty impressive/interesting. On average, the Effective Donation scores are about 10% higher for the Info Group participants than the No Info group participants (and I didn’t do a formal calculation for Cohen’s d but it looks like it’d be about d=0.5).
Of course, given the small sample size, it’s hard to draw any definitive conclusions. But it seems quite plausible to me that the Info condition worked—and at the very least, I don’t think these findings provide evidence against the idea that the info condition worked.
Would be curious to see if you have any thoughts on this. If you end up having an opportunity to test this with a larger sample size, that would be super interesting. Great work & excited to see what you do next!
Thanks for your thorough comment! Yeah I was shooting for about 60 participants, but due to time constraints and this being a pilot study I only ended up with 44, so even more underpowered.
Intuitively I would expect a larger effect size, given that I don’t consider the manipulation to be particularly subtle; but yes, it was much subtler than it could have been. This is something I will definitely explore more if I continue this project; for example, adding visuals and a manipulation check might do a better job of making the manipulation salient. I would like to have a manipulation check like “What is the difference between average and highly cost-effective charities?” And then set it up so that participants who get it wrong have to try again.
The fact that Donation Change differed significantly between Info groups does support that second main hypothesis, suggesting that CE info affects effective donations. This result, however, is not novel. So yes, the effect you picked up on is probably real – but this study was underpowered to detect it at a level of p<.05 (or even marginal significance).
In terms of CE info being ineffective, I’m thinking mainly about interest in EA – to which there really seems to be nothing going on, “There was no significant difference between the Info (M = 32.52, SD = 5.92) and No Info (M = 33.12, SD = 4.01) conditions, F(1, 40) = .118, p = .733, ηp2 = .003.” There isn’t even a trend in the expected direction. This was most important to me because, as far as I know, there is no previous empirical evidence to suggest that CE info affects interest in EA. It’s also more relevant to me as somebody running an EA group and trying to generate interest from people outside the group.
Thanks for sharing this, Aaron! Really interesting pilot work.
One quick thought—I wouldn’t rely too heavily on statistical significance tests, particularly with small sample sizes. P-values are largely a function of sample size, and it’s nearly impossible to get statistical significance with 44 participants (unless your effect size is huge!).
Speaking of effect sizes, it seems like you powered to detect an effect of d=0.7. For a messaging study with rather subtle manipulations, an effect of d=0.7 seems huge! I would be pretty impressed if giving people CE info resulted in an effect size of d=0.2 or d=0.3, for instance. I’m guessing you were constrained by the # of participants you could recruit (which is quite reasonable—lots of pilot studies are underpowered). But given the low power, I’d be reluctant to draw strong conclusions.
I also appreciate that you reported the mean scores in the results section of your paper, which allowed me to skim to see if there’s anything interesting. I think there might be!
If this effect is real, I think this is pretty impressive/interesting. On average, the Effective Donation scores are about 10% higher for the Info Group participants than the No Info group participants (and I didn’t do a formal calculation for Cohen’s d but it looks like it’d be about d=0.5).
Of course, given the small sample size, it’s hard to draw any definitive conclusions. But it seems quite plausible to me that the Info condition worked—and at the very least, I don’t think these findings provide evidence against the idea that the info condition worked.
Would be curious to see if you have any thoughts on this. If you end up having an opportunity to test this with a larger sample size, that would be super interesting. Great work & excited to see what you do next!
Thanks for your thorough comment! Yeah I was shooting for about 60 participants, but due to time constraints and this being a pilot study I only ended up with 44, so even more underpowered.
Intuitively I would expect a larger effect size, given that I don’t consider the manipulation to be particularly subtle; but yes, it was much subtler than it could have been. This is something I will definitely explore more if I continue this project; for example, adding visuals and a manipulation check might do a better job of making the manipulation salient. I would like to have a manipulation check like “What is the difference between average and highly cost-effective charities?” And then set it up so that participants who get it wrong have to try again.
The fact that Donation Change differed significantly between Info groups does support that second main hypothesis, suggesting that CE info affects effective donations. This result, however, is not novel. So yes, the effect you picked up on is probably real – but this study was underpowered to detect it at a level of p<.05 (or even marginal significance).
In terms of CE info being ineffective, I’m thinking mainly about interest in EA – to which there really seems to be nothing going on, “There was no significant difference between the Info (M = 32.52, SD = 5.92) and No Info (M = 33.12, SD = 4.01) conditions, F(1, 40) = .118, p = .733, ηp2 = .003.” There isn’t even a trend in the expected direction. This was most important to me because, as far as I know, there is no previous empirical evidence to suggest that CE info affects interest in EA. It’s also more relevant to me as somebody running an EA group and trying to generate interest from people outside the group.
Thanks again for your comment! Edit: Here’s the previous study suggesting CE info influences effective donations: http://journal.sjdm.org/20/200504/jdm200504.pdf