What are you worried about here? The same studies will give confidence intervals on effect sizes, which are actionable, and reliable significance at a given sample size indicates an effect of a given magnitude..
Sure, one should attend to priors in interpretation, but that doesn’t make the experiment useless.
If a pre-registered experiment reliably gives you a severalfold likelihood ratio, you can repeat it or scale it up and overcome significant prior skepticism (although limited by credence in hidden flaws).
I’m not saying any experiment is necessarily useless, but if MFA is going to spend a bunch of resources on another study they should use methods that won’t exaggerate effectiveness.
And it’s not only that “one should attend to priors in interpretation”—one should specify priors beforehand and explicitly update conditional on the data.
What are you worried about here? The same studies will give confidence intervals on effect sizes, which are actionable, and reliable significance at a given sample size indicates an effect of a given magnitude..
Confidence intervals still don’t incorporate prior information and so give undue weight to large effects.
Sure, one should attend to priors in interpretation, but that doesn’t make the experiment useless.
If a pre-registered experiment reliably gives you a severalfold likelihood ratio, you can repeat it or scale it up and overcome significant prior skepticism (although limited by credence in hidden flaws).
I’m not saying any experiment is necessarily useless, but if MFA is going to spend a bunch of resources on another study they should use methods that won’t exaggerate effectiveness.
And it’s not only that “one should attend to priors in interpretation”—one should specify priors beforehand and explicitly update conditional on the data.