Within the academic community, many people do important research in things like medicine, technology, etc that require a lot of money up-front but can pay back very well/ do a lot of good in the long term.
However, a general issue with research is that there is small monetary incentive to reproduce research. This is a huge issue because it undermines one of the axioms of why science is so important: repeatability.
For example, in 2022, a fundamental study for the theory of where Alzheimer’s comes from (from 16 years prior) was found to have been forged. Prior to awareness of the forgery, the FDA approved a drug that, based on this research, should significantly decrease intensity or even completely cure Alzheimer’s. In that same fiscal year, the NIH even spent $1.6 billion on research that mentions the results of the study, representing about half of overall Alzheimer’s funding (see the article attached).
Therefore, I propose that money ought to be raised to ensure that this sort of thing does not happen again by incentivizing academics to repeat studies.
Comment below additional events similar to the Alzheimer’s one below or any criticisms to this point
There are some non profits working on it. See here:
https://en.m.wikipedia.org/wiki/Arnold_Ventures_LLC
The Arnolds look to be more focused on social science, but they are behind several “open science” initiatives.
I’m an early career academic (accounting) and this was a big discussion in my phd program.
As a phd student, we completed multiple replications as an exercise in learning the research process. It is exhausting work, in part because authors often don’t explain their methodology in sufficient detail to complete an exact replication. Best we could hope for was similar sample/descriptives/coefficients on main tests after following their process as best we could.
Another issue is that in many cases, the data used is proprietary and cannot be shared due to a data license agreement.
As you allude to, the main problem is that there is no real incentive for active researchers to work on replications, because generally journals do not usually publish replications (and of course, publish or perish!). You do occasionally see papers that are published which point out a major flaw in a published article, but these are rare and controversial (why make an enemy?).
I know there have been some studies that basically show that a very large (50%?) percentage of papers (I think in econ/finance/accounting) cannot be replicated, which is obviously concerning, and points to the scope of the problem.
I think the most successful work that could be done in this area is lobbying journals to:
Require authors to include both their data and code and open-source it.
If that isn’t possible, require authors to include data and code specifically to the anonymous reviewers + editor.
If that isn’t possible, journals should employ an expert on methods who’s full time job is replicating new studies.
I cannot stress enough how expensive this person would be and journals probably wouldn’t be willing to pay.
You would have to 3x my compensation AND you would have to guarantee me I would only replicate studies in my niche research area to do this job. And I still would probably decline an offer to do this work, it would suck that much.
Agreed. I think culture change can be created cost-efficiently by funding replication attempts for the most famous / reputable papers, (eg—by most citations, by Nobel Prize winners, by Nature front covers). Failed replications for the most famous papers could have big impacts on culture inside academia.
You might like to look through the metascience tag on the forum to look at other similar issues.
Double-checking research seems really important and neglected. This can be valuable even if you don’t rerun the experiments and just try to replicate the analyses.
A couple of years ago, I was hired to review seven econometric papers, and even as an outsider to the field it was easy to contribute to find flaws and assess the strength of the papers.
Writing these reviews seems like a great activity, especially for junior researchers who want to learn good research practices while making a substantial contribution.
A related problem is that researchers who cite other published research sometimes misinterpret that research or take findings out of context, and this can be hard for readers of the new paper to detect. I’ve learned to be suspicious of meta-analyses for this reason. On numerous occasions in my work (mostly in infectious disease research), I’ve gone to check underlying references and found that they were either misquoted or missing important context that affects the interpretation.
A five-sentence letter to the editor of the New England Journal of Medicine, which appeared in 1980, was cited hundreds of times during the early years of the opioid crisis in the 1990s, usually to support claims that opioid addiction is very rare when opioids are medically prescribed. This letter may have played a significant role in fueling the crisis. https://www.nejm.org/doi/10.1056/NEJMc1700150The letter did in fact report on hospitalized patients prescribed opioids, and the authors did find that it was very rare for opioid addiction to develop during the closely monitored hospital stay. However, the study was not peer reviewed, included a single hospital, did not follow the patients to see if they were addicted after they went home, and did not include any data on patients prescribed opioids for use at home.