A system somewhat similar to what you are talking about exists. Pubpeer, for example, is a place where post-publication peer reviews of papers are posted publicly (https://pubpeer.com/static/about). I’m not sure at this stage how much it is used, but in principle it allows you to see criticism on any article.
Scite.ai is also relevant—it uses AI to try and say whether citations of an article are positive or negative. I don’t know about its accuracy.
Neither of these address the problem of what happens if a study fails to replicate—often what happens is that the original study continues to be cited more than the replication effort.
Thanks for sharing those sources! I think a system like Pubpeer could partially address some of the issues/functions I mentioned, although I don’t think it quite went as far as I was hoping (in part because it doesn’t seem to have the “relies upon” aspect, but I also couldn’t find that many criticisms/analyses in the fields I’m more familiar with so it is hard to tell what kinds of analysis takes place there). The Scite.ai system seems more interesting—in part because I have specifically thought that it would be interesting to see whether machine learning could assist with this kind of semantic-richer bibliometrics.
Also, I wouldn’t judge based solely off of this, but the Nature article you linked has this quote regarding Scite’s accuracy: “According to Nicholson, eight out of every ten papers flagged by the tool as supporting or contradicting a study are correctly categorized.”
A couple of other new publication models that might be worth looking at are discussed here (Octopus and hypergraph, both of which are modular). Also this recent article about ‘publomics’ might have interesting ideas. Happy to talk about any of this if you are thinking about doing something in the space.
Those both seem interesting! I’ll definitely try to remember to reach out if I start doing more work in this field/on this project. Right now it’s just a couple of ideas that keep nagging at me but I’m not exactly sure what to do with them and they aren’t currently the focus of my research, but if I could see options for progress (or even just some kind of literature/discussion on the epistemap/repository concept, which I have not really found yet) I’d probably be interested.
A system somewhat similar to what you are talking about exists. Pubpeer, for example, is a place where post-publication peer reviews of papers are posted publicly (https://pubpeer.com/static/about). I’m not sure at this stage how much it is used, but in principle it allows you to see criticism on any article.
Scite.ai is also relevant—it uses AI to try and say whether citations of an article are positive or negative. I don’t know about its accuracy.
Neither of these address the problem of what happens if a study fails to replicate—often what happens is that the original study continues to be cited more than the replication effort.
Thanks for sharing those sources! I think a system like Pubpeer could partially address some of the issues/functions I mentioned, although I don’t think it quite went as far as I was hoping (in part because it doesn’t seem to have the “relies upon” aspect, but I also couldn’t find that many criticisms/analyses in the fields I’m more familiar with so it is hard to tell what kinds of analysis takes place there). The Scite.ai system seems more interesting—in part because I have specifically thought that it would be interesting to see whether machine learning could assist with this kind of semantic-richer bibliometrics.
Also, I wouldn’t judge based solely off of this, but the Nature article you linked has this quote regarding Scite’s accuracy: “According to Nicholson, eight out of every ten papers flagged by the tool as supporting or contradicting a study are correctly categorized.”
A couple of other new publication models that might be worth looking at are discussed here (Octopus and hypergraph, both of which are modular). Also this recent article about ‘publomics’ might have interesting ideas. Happy to talk about any of this if you are thinking about doing something in the space.
Those both seem interesting! I’ll definitely try to remember to reach out if I start doing more work in this field/on this project. Right now it’s just a couple of ideas that keep nagging at me but I’m not exactly sure what to do with them and they aren’t currently the focus of my research, but if I could see options for progress (or even just some kind of literature/discussion on the epistemap/repository concept, which I have not really found yet) I’d probably be interested.