Thanks for sharing and for putting this on OSF. Some thoughts and suggestions, echoing those below.
Maybe consider rewriting/re-titling this? To say “did not increase” seems too strong and definitive.
You “failed to find a statistically significant effect” in standard tests that were basically underpowered. This is not strong evidence of a near-zero true effect. If anything, you found evidence suggesting a positive effect, at least on the donation ‘action’ (if I read Aaron’s comment carefully).
You might consider a Bayesian approach, and then put some confidence bounds on the true effects, given a reasonably flat/informative prior. (You can do something similar with ‘CIs’ in a standard frequentist approach.)
Then you will be able to say something about ‘with this prior, our posterior 80% credible interval over the true effect is between -X% and +X%’ (perhaps stated in terms of Cook’s d or something relatable) … if that interval rules out a ‘substantial effect’ then you could make a more meaningful statement. (With appropriate caveats about the nature of the sample, the context, etc., as you do.)
(Also, if you rewrite, can you break this into shorter paragraphs—the long paragraph chunks become overwhelming to read.)
Thanks for sharing and for putting this on OSF. Some thoughts and suggestions, echoing those below.
Maybe consider rewriting/re-titling this? To say “did not increase” seems too strong and definitive.
You “failed to find a statistically significant effect” in standard tests that were basically underpowered. This is not strong evidence of a near-zero true effect. If anything, you found evidence suggesting a positive effect, at least on the donation ‘action’ (if I read Aaron’s comment carefully).
You might consider a Bayesian approach, and then put some confidence bounds on the true effects, given a reasonably flat/informative prior. (You can do something similar with ‘CIs’ in a standard frequentist approach.)
Then you will be able to say something about ‘with this prior, our posterior 80% credible interval over the true effect is between -X% and +X%’ (perhaps stated in terms of Cook’s d or something relatable) … if that interval rules out a ‘substantial effect’ then you could make a more meaningful statement. (With appropriate caveats about the nature of the sample, the context, etc., as you do.)
(Also, if you rewrite, can you break this into shorter paragraphs—the long paragraph chunks become overwhelming to read.)