I’m an early career academic (accounting) and this was a big discussion in my phd program.
As a phd student, we completed multiple replications as an exercise in learning the research process. It is exhausting work, in part because authors often don’t explain their methodology in sufficient detail to complete an exact replication. Best we could hope for was similar sample/descriptives/coefficients on main tests after following their process as best we could.
Another issue is that in many cases, the data used is proprietary and cannot be shared due to a data license agreement.
As you allude to, the main problem is that there is no real incentive for active researchers to work on replications, because generally journals do not usually publish replications (and of course, publish or perish!). You do occasionally see papers that are published which point out a major flaw in a published article, but these are rare and controversial (why make an enemy?).
I know there have been some studies that basically show that a very large (50%?) percentage of papers (I think in econ/finance/accounting) cannot be replicated, which is obviously concerning, and points to the scope of the problem.
I think the most successful work that could be done in this area is lobbying journals to:
Require authors to include both their data and code and open-source it.
If that isn’t possible, require authors to include data and code specifically to the anonymous reviewers + editor.
If that isn’t possible, journals should employ an expert on methods who’s full time job is replicating new studies.
I cannot stress enough how expensive this person would be and journals probably wouldn’t be willing to pay.
You would have to 3x my compensation AND you would have to guarantee me I would only replicate studies in my niche research area to do this job. And I still would probably decline an offer to do this work, it would suck that much.
I’m an early career academic (accounting) and this was a big discussion in my phd program.
As a phd student, we completed multiple replications as an exercise in learning the research process. It is exhausting work, in part because authors often don’t explain their methodology in sufficient detail to complete an exact replication. Best we could hope for was similar sample/descriptives/coefficients on main tests after following their process as best we could.
Another issue is that in many cases, the data used is proprietary and cannot be shared due to a data license agreement.
As you allude to, the main problem is that there is no real incentive for active researchers to work on replications, because generally journals do not usually publish replications (and of course, publish or perish!). You do occasionally see papers that are published which point out a major flaw in a published article, but these are rare and controversial (why make an enemy?).
I know there have been some studies that basically show that a very large (50%?) percentage of papers (I think in econ/finance/accounting) cannot be replicated, which is obviously concerning, and points to the scope of the problem.
I think the most successful work that could be done in this area is lobbying journals to:
Require authors to include both their data and code and open-source it.
If that isn’t possible, require authors to include data and code specifically to the anonymous reviewers + editor.
If that isn’t possible, journals should employ an expert on methods who’s full time job is replicating new studies.
I cannot stress enough how expensive this person would be and journals probably wouldn’t be willing to pay.
You would have to 3x my compensation AND you would have to guarantee me I would only replicate studies in my niche research area to do this job. And I still would probably decline an offer to do this work, it would suck that much.