Thanks a lot for writing this post. I’m interested in these topics and was just thinking the other day that a write up of this sort would be valuable.
A relevant and fairly detailed write-up (not mine) of this problem area and how meta-research might help is available here: https://lets-fund.org/better-science/ (I didn’t see it cited but may have missed it).
In terms of the content of the post, a couple of things that I might push back on a little:
Peer review: I’m not sure that poor peer review (of papers) is a major cause of ineffective value production (though I agree that it is a minor contributor). By the time a project is written up as a paper, it will invariably be published somewhere in the literature in roughly the format that it was first submitted. If top journals had better peer review (but other journals did not), the research would likely be published elsewhere anyway. Basically it strikes me as too late in the process to be that important. Poor methodology (which I would attribute largely to lack of training and the incentives to rush research) seems more important . Lack of peer review at an appropriate time in the research process (i.e. before the research is done to get feedback on methods) also seems more important than the quality of peer review of the final paper (which is what i understood the section on peer review to be describing).
Intellectual property: this seems mostly relevant to a smallish subset of research that is directly involved in making products. Even in those cases, it isn’t clear that IP is a big barrier. In fact, it can be argued that not patenting is better for development of products in some cases, because it allows multiple commercialisation attempts in parallel with slightly different aims. For an example of this in the context of drug development, see here: https://www.thesgc.org/. The basic idea is that if e.g. a molecule is not patented when it is initially described, you can still patent the use of that molecule for a particular indication, so the molecule can still be commercialised for that indication, while another organisation may pursue the same molecule for another indication. This potentially increases rather than decreases the potential for commercialisation of the molecule.
I’d be interested in learning what projects you have planned and discussing some solutions to the problems that you have mapped.
So glad to hear that, and thanks for the added reference to letsfund!
On peer review I agree with Edo’s comment, I think it’s more about setting a standard than about improving specific papers.
On IP, I think this is very complex and I think “IP issues” can be a barrier both when something is protected and when it’s not. I have personally worked in the periphery of projects where failing to protect/maintain IP has been the end of road for potentially great discoveries, but also seen the other phenomena where researchers avoid a specific area because someone else holds the IP. It would be interesting to get a better understanding both of the scale of these problems and if any of the initiatives that currently exists seem promising for improving it.
Great points! Re peer-review, I think that your argument makes sense but I feel like most of the impact on quality from better peer review would actually be in raising standards for the field as a whole, rather than the direct impact on the papers who didn’t pass peer review. I’d love to have a much clearer analysis of the whole situation :)
That view seems reasonable to me and I agree that a clearer analysis would be useful.
An additional and very minor point I missed out from my comment is that I’m sceptical that the relationship between impact factor and retraction (original paper here) is causal. It seems very likely to me that something like “number of views of articles” would be a confounder, and it is not adjusted for as far as I can tell. I’m not totally sure that is the part of the article that you were referring to when citing this, so apologies if not!
Thanks a lot for writing this post. I’m interested in these topics and was just thinking the other day that a write up of this sort would be valuable.
A relevant and fairly detailed write-up (not mine) of this problem area and how meta-research might help is available here: https://lets-fund.org/better-science/ (I didn’t see it cited but may have missed it).
In terms of the content of the post, a couple of things that I might push back on a little:
Peer review: I’m not sure that poor peer review (of papers) is a major cause of ineffective value production (though I agree that it is a minor contributor). By the time a project is written up as a paper, it will invariably be published somewhere in the literature in roughly the format that it was first submitted. If top journals had better peer review (but other journals did not), the research would likely be published elsewhere anyway. Basically it strikes me as too late in the process to be that important. Poor methodology (which I would attribute largely to lack of training and the incentives to rush research) seems more important . Lack of peer review at an appropriate time in the research process (i.e. before the research is done to get feedback on methods) also seems more important than the quality of peer review of the final paper (which is what i understood the section on peer review to be describing).
Intellectual property: this seems mostly relevant to a smallish subset of research that is directly involved in making products. Even in those cases, it isn’t clear that IP is a big barrier. In fact, it can be argued that not patenting is better for development of products in some cases, because it allows multiple commercialisation attempts in parallel with slightly different aims. For an example of this in the context of drug development, see here: https://www.thesgc.org/. The basic idea is that if e.g. a molecule is not patented when it is initially described, you can still patent the use of that molecule for a particular indication, so the molecule can still be commercialised for that indication, while another organisation may pursue the same molecule for another indication. This potentially increases rather than decreases the potential for commercialisation of the molecule.
I’d be interested in learning what projects you have planned and discussing some solutions to the problems that you have mapped.
So glad to hear that, and thanks for the added reference to letsfund!
On peer review I agree with Edo’s comment, I think it’s more about setting a standard than about improving specific papers.
On IP, I think this is very complex and I think “IP issues” can be a barrier both when something is protected and when it’s not. I have personally worked in the periphery of projects where failing to protect/maintain IP has been the end of road for potentially great discoveries, but also seen the other phenomena where researchers avoid a specific area because someone else holds the IP. It would be interesting to get a better understanding both of the scale of these problems and if any of the initiatives that currently exists seem promising for improving it.
Great points! Re peer-review, I think that your argument makes sense but I feel like most of the impact on quality from better peer review would actually be in raising standards for the field as a whole, rather than the direct impact on the papers who didn’t pass peer review. I’d love to have a much clearer analysis of the whole situation :)
That view seems reasonable to me and I agree that a clearer analysis would be useful.
An additional and very minor point I missed out from my comment is that I’m sceptical that the relationship between impact factor and retraction (original paper here) is causal. It seems very likely to me that something like “number of views of articles” would be a confounder, and it is not adjusted for as far as I can tell. I’m not totally sure that is the part of the article that you were referring to when citing this, so apologies if not!