This definitely seems interesting! I’m curious whether you would also be interested in seeing how other, later studies have used any findings that you cannot replicate, and thus get a sense of any “epistemic contagion” in the literature? Or would the studies you try to replicate be too new for that to make sense? (Or do you simply think that’s better left to other researchers?)
It at least seems to me that if you had a good sense of “which findings/studies would involve high amounts of epistemic contagion if they do fail to replicate” then that might help with choosing which studies to focus on.
I wrote an EA Forum post describing the concept of epistemic mapping (with pictures) here, but I’ll avoid going into detail on that. I just bring it up because one of the reasons that I’ve thought that epistemic mapping may be valuable is that it could potentially help with understanding research/epistemic contagion: i.e., how flawed datasets, regression analyses, experimental findings, or other inputs might produce inaccurate findings in the broader research literature.
I guess if you found reproducibility problems in a bunch of related papers, that would point to a common cause. In fact, I found a case like this in my dissertation: the entire literature on meritocratic promotion in China is unreliable, and is based on a highly-cited 2005 article.
This definitely seems interesting! I’m curious whether you would also be interested in seeing how other, later studies have used any findings that you cannot replicate, and thus get a sense of any “epistemic contagion” in the literature? Or would the studies you try to replicate be too new for that to make sense? (Or do you simply think that’s better left to other researchers?)
It at least seems to me that if you had a good sense of “which findings/studies would involve high amounts of epistemic contagion if they do fail to replicate” then that might help with choosing which studies to focus on.
I wrote an EA Forum post describing the concept of epistemic mapping (with pictures) here, but I’ll avoid going into detail on that. I just bring it up because one of the reasons that I’ve thought that epistemic mapping may be valuable is that it could potentially help with understanding research/epistemic contagion: i.e., how flawed datasets, regression analyses, experimental findings, or other inputs might produce inaccurate findings in the broader research literature.
I guess if you found reproducibility problems in a bunch of related papers, that would point to a common cause. In fact, I found a case like this in my dissertation: the entire literature on meritocratic promotion in China is unreliable, and is based on a highly-cited 2005 article.