Thanks, in general pre-registration is a great tool to help have appropriate confidence in results.
I may disagree with this slightly:
More subtle cheating is also discouraged, such as adding extra statistical tests because the data seem to be pointing that way.
There are some forms of that which are pure p-value hacking, and should definitely be discouraged. But experiments have a couple of purposes, only one of which is hypothesis testing. The other is exploration, which can help with hypothesis generation.
It’s absolutely legitimate to notice an odd correlation between variables you didn’t plan in advance to test for correlation and form a hypothesis about this. It may be helpful to use a statistical test to see how suggestive the data is. What you shouldn’t do is confuse the strength of evidence for such hypotheses with ones which you set out to explore.
Thanks, in general pre-registration is a great tool to help have appropriate confidence in results.
I may disagree with this slightly:
There are some forms of that which are pure p-value hacking, and should definitely be discouraged. But experiments have a couple of purposes, only one of which is hypothesis testing. The other is exploration, which can help with hypothesis generation.
It’s absolutely legitimate to notice an odd correlation between variables you didn’t plan in advance to test for correlation and form a hypothesis about this. It may be helpful to use a statistical test to see how suggestive the data is. What you shouldn’t do is confuse the strength of evidence for such hypotheses with ones which you set out to explore.
I agree. Thanks for clarifying.