Error

This is a great idea. The only tweak I’d suggest is to let people do data analysis that’s not in the pre-registration as long as it’s clearly marked. Exploratory data analysis doesn’t have to be a bad; if nothing else it can be a good way to get ideas for future studies.

Even though this seems like a pretty clear improvement over the status quo, it also feels like a band-aid. In the worst case I could imagine data falsification becoming more common if data analysis pre-registration became required. Funding-starved academics desperately need to create interesting publishable results (“publish or perish”), and you removed one of the main things that made this possible.

I would like to see more people write about what academia might look like if it was rethought from the ground up. OpenPhil has $10 billion, and funding is to academics what candy is to kids. If OpenPhil tells researchers they have to work a particular way, my guess is they will do it. This could be a really valuable opportunity to experiment with alternative models.

I agree with this entirely (most of my ‘proper’ academic work has an analysis plan and released code, but not code prior to analysis—I wish I had thought of using dummy data). It is not even clear it is that more arduous than thrashing it out on the console—this will happen either way, so the only additional cost is generating the dummy data.

Specification curves might help: https://​​www.youtube.com/​​watch?v=g75jstZidX0

I’ll also point out that pre-registration is one of those ideas that is useful for casual reasoning as well: figuring out your evidence buckets and how they would sway your opinion on being filled with various pieces of evidence is often very helpful for sharpening a nebulous problem into an understandable one.