I would be especially wary of conducting more studies if we plan on trying to “prove” or “disprove” the effectiveness of ads with so dubious a tool as null hypothesis significance tests.
Even if in a new study we were to reject the null hypothesis of no effect, this would arguably still be pretty weak evidence in favor of the effectiveness of ads.
What are you worried about here? The same studies will give confidence intervals on effect sizes, which are actionable, and reliable significance at a given sample size indicates an effect of a given magnitude..
Sure, one should attend to priors in interpretation, but that doesn’t make the experiment useless.
If a pre-registered experiment reliably gives you a severalfold likelihood ratio, you can repeat it or scale it up and overcome significant prior skepticism (although limited by credence in hidden flaws).
I’m not saying any experiment is necessarily useless, but if MFA is going to spend a bunch of resources on another study they should use methods that won’t exaggerate effectiveness.
And it’s not only that “one should attend to priors in interpretation”—one should specify priors beforehand and explicitly update conditional on the data.
I would be especially wary of conducting more studies if we plan on trying to “prove” or “disprove” the effectiveness of ads with so dubious a tool as null hypothesis significance tests.
Even if in a new study we were to reject the null hypothesis of no effect, this would arguably still be pretty weak evidence in favor of the effectiveness of ads.
What are you worried about here? The same studies will give confidence intervals on effect sizes, which are actionable, and reliable significance at a given sample size indicates an effect of a given magnitude..
Confidence intervals still don’t incorporate prior information and so give undue weight to large effects.
Sure, one should attend to priors in interpretation, but that doesn’t make the experiment useless.
If a pre-registered experiment reliably gives you a severalfold likelihood ratio, you can repeat it or scale it up and overcome significant prior skepticism (although limited by credence in hidden flaws).
I’m not saying any experiment is necessarily useless, but if MFA is going to spend a bunch of resources on another study they should use methods that won’t exaggerate effectiveness.
And it’s not only that “one should attend to priors in interpretation”—one should specify priors beforehand and explicitly update conditional on the data.