I find the very long list of badly-designed studies you note in the introduction a cause for consternation, and I’m glad this has been done much better.
However, I couldn’t see a power calculation in the study, nor in the pre-registration, so I worry the planned recruitment of 3000 was either plucked from the air or decided on due to budget constraint. Yet performing this calculation given an effect size you’d be interested in is generally preferable to spending money on an underpowered study (which I’m pretty sure this is).
Given the large temporal fluctuations (e.g. the large reduction in control group), the pretty modest effects, I remain sceptical—leave alone the obvious biases like social desirability etc. Another reanalysis which might reassure would be monte carlo permutation of food groups: if very few random groups show reduction in consumption to a similar magnitude as meat, great (and, of course, vice versa).
Here is the main plot from our power calculations that informed our sample size selection (alongside budget constraints): http://i.imgur.com/aeEYagA.png
I worry the planned recruitment of 3000 was either plucked from the air or decided on due to budget constraint
I want to be careful not to speak for the authors here, but I’m personally pretty sure it was picked by budget constraint, though with an eye to power calculations (that I saw, not sure why they weren’t published) suggesting it would be sufficient.
-
Given the large temporal fluctuations (e.g. the large reduction in control group), the pretty modest effects, I remain sceptical—leave alone the obvious biases like social desirability etc.
Agreed.
-
Another reanalysis which might reassure would be monte carlo permutation of food groups: if very few random groups show reduction in consumption to a similar magnitude as meat, great (and, of course, vice versa).
Good write-up.
I find the very long list of badly-designed studies you note in the introduction a cause for consternation, and I’m glad this has been done much better.
However, I couldn’t see a power calculation in the study, nor in the pre-registration, so I worry the planned recruitment of 3000 was either plucked from the air or decided on due to budget constraint. Yet performing this calculation given an effect size you’d be interested in is generally preferable to spending money on an underpowered study (which I’m pretty sure this is).
Given the large temporal fluctuations (e.g. the large reduction in control group), the pretty modest effects, I remain sceptical—leave alone the obvious biases like social desirability etc. Another reanalysis which might reassure would be monte carlo permutation of food groups: if very few random groups show reduction in consumption to a similar magnitude as meat, great (and, of course, vice versa).
Here is the main plot from our power calculations that informed our sample size selection (alongside budget constraints): http://i.imgur.com/aeEYagA.png
I want to be careful not to speak for the authors here, but I’m personally pretty sure it was picked by budget constraint, though with an eye to power calculations (that I saw, not sure why they weren’t published) suggesting it would be sufficient.
-
Agreed.
-
In my re-analysis, I did make a “bogustarian” label looking at reduction in beans, fruits, nuts, vegetables, and grains and found no statistically significant results (see https://github.com/bnjmacdonald/reducetarian-messaging-study/blob/master/peter-reanalysis/analysis.R#L124-L132). So maybe that’s reassuring, but one could extend this to be a true monte carlo method.