Cross-posted from my blog—inspired by the recent call for more monitoring and evaluation
Hi, it’s Hauke, the founder of Let’s Fund. We research pressing problems, like climate change or the replication crisis in science, and then crowdfund for particularly effective policy solutions.
Ages ago, you signed up to my newsletter. Now I’ve evaluated the $1M+ in grants you donated, and they had a big impact. Below I present the Better Science / Registered Report campaign evaluation, but stay tuned for the climate policy campaign impact evaluation (spoiler: clean energy R&D increased by billions of dollars).
Let’s Fund: Better Science
Chris Chambers giving a talk on Registered Reports
We crowdfunded ~$80k for Prof. Chambers to promote Registered Reports, a new publication format, where research is peer-reviewed before the results are known. This fundamentally changes the way research is done across all scientific fields. For instance, one recent Registered Report studied COVID patients undergoing ventilation1 (but there are examples in other areas including climate science,2 development economics,3 biosecurity, 4 farm animal welfare,5 etc.).
Registered Reports have higher quality than normal publications,6 because they
make science more theory-driven, open and transparent
find methodological weaknesses and also potential biosafety failures of dangerous dual-use research prior to publication (e.g. gain of function research)7
get more papers published that fail to confirm the original hypothesis
increase the credibility of non-randomized natural experiments using observational data
If Registered Reports become widely adopted, it might lead to a paradigm shift and better science. 300+ journals have already adapted Registered Reports. And just last week Nature, the most prestigious academic journal, adopted it:
This is big and Registered Reports might soon become the gold standard.
Why? Imagine you’re a scientist with a good idea for an experiment with high value of information (think: a simple cure for depression). If that has a low chance of working out (say 1%), then previously you had little incentive to run it.
Now, if your idea is really good, and based on strong theory, Registered Reports derisks running the experiment. You can first submit the idea and methodology to Nature and the reviewers might say: ‘This idea is nuts, but we agree there’s a small chance it might work and really interested in this works. If you run the experiment, we’ll publish this independent of results!’ Now you can go ahead spend a lot of effort on running the experiment, because even if it doesn’t work, you still get a Nature paper (which you wouldn’t with null results).
This will lead to more high risk, high reward research (share this post or the tweet with academics! They might thank you for the Nature publication).
Many people were integral to this progress, but I think Chambers, the co-inventor and prime proponent of Registered Reports deserves special credit. In turn he credited:
Chris Chambers @chrisdc77: ‘You. That’s right. Some of the most useful and flexible funding I’ve received has been donated by hundreds of generous members of public (& small orgs) via our @LetsFundOrg-supported crowd sourcing fund’
You may feel smug.
If you want to make a bigger donation (>$1k), click here. There are proposals to improve Registered Reports even further, like trinity review which integrates Registered Reports with ethics and funding reviews, and peer community review which reduces review burdens (also see The past, present and future of Registered Reports for more).
Acknowledgments
We would like to thank the following organizations and people for helping Let’s Fund in various ways: Two anonymous EA donors, The Effective Altruism Infrastructure-Fund, the Center for Effective Altruism, The Survival and Flourishing Fund, Jacob Hilton, Founders Pledge, Effektiv-Spenden.org, Rethink Charity: Forward, the Effective Altruism Foundation, Slate Star Codex, EA Giving Tuesday, Vox.com, Legacies Now, the founding team including Henry Stanley and Sahil Shah (see Lets-Fund.org/About), and everyone who has reviewed our research and donated to our crowdfunding campaigns.
This is amazing! Reading this made me very happy and more hopeful for the future!
This is fantastic news!
As an experimental economist, I hope this has spillovers to our field (as well as others).
At the feedback level (referee reports, presentations etc), I believe there is significantly more value to be gained when discussing the experimental design itself before any data is collected.
Congrats to Hauke, Chris, and all others involved.
Great work :)
Awesome, Hauke. Really excited to see this.