Can you clarify the point you’re trying to make with the reference to spurious correlations, Will? I don’t think the author is trying to make any deep claim about causation here, but just pointing out that a growing amount of taxpayer money is wasted due to retractions. (I appreciate the point from other commenters that this is still presumably a small fraction of the total funding though and so might not be as big a concern as the author suggests.)
Taken at face value, the claim is that taxpayer funding and number of retractions have increased over time, at rates not hugely different from one another. I think both can almost entirely be accounted for by an increase in the total number of researchers. If you have more researchers producing papers, this will result in both a big increase in funding required and in number of papers retracted without any change in the quality distribution.
I would want to see evidence for a big increase in retractions per number of researchers, researcher hours or some other aggregative measure before taking this seriously as a claim that science has got worse over time. It’s well-known that if you don’t control for the total number of people in a place or doing a thing, all sorts of things will correlate (homicides and priests, ice-cream sales and suicides, etc.).
More substantively, I also disagree with the claim that a big increase in retractions is evidence of scientific decline. Insofar as there has been any increase in the per-capita rate of retractions, I regard this as a sign of increasing epistemic standards, and think both editors and scientists are still way too reluctant to retract papers. It’s like the replication crisis: the problems have always been there, but we only started paying attention to them recently. That’s a good sign, not a bad one.
Agreed that the increase in funding for science will generally just increase the size of science, and the base assumption should be that the retraction rate will stay the same, which would lead to a roughly proportionate increase in the number of retractions with science funding. The 700% vs. 900% roughly agrees with that assumption (although it could still be that the reasons for retraction change over time).
The idea of increasing retractions being a beneficial sign of better epistemic standards is interesting. My observation is that papers are usually basically only retracted if scientific fraud or misconduct was committed (e.g. falsifying or manipulating research data) - questionable research practices (e.g. P-hacking, optional stopping or HARKing), failure to replicate, or even technical errors don’t usually lead to a retraction (Wikipedia also notes that plagiarism is a common cause of retractions). It is a pity there is no ground truth for scientific misconduct to reference the retraction rate against.
I’d be interested in seeing data on the distribution of causes of retraction and how it’s changed over time. I know RetractionWatch likes to say that scientists tend to underestimate the proportion of retractions that are down to fraud. I do think some (many?) retractions are due to serious technical errors with no implication of deliberate fraud or misconduct. I suspect RetractionWatch has data on this.
I’m not claiming that it’s inevitably true that more retractions indicates better community epistemics, but I do think it’s a big part of the story in this case. A paper retraction requires someone to notice that the paper is worthy of retraction, bring that to the editors and, very often, put a lot of pressure on the editors to retract the paper (who are usually extremely reluctant to do so). That requires people to be on the lookout for things that might need to be retracted and willing to put in the time and effort to get it retracted.
In the past this was very rare, and only extremely flagrant fraud or misconduct (or unusually honest scientists retracting their own work) led to retractions. Now, partly as a side consequence of the replication crisis but also more general (and incomplete) changes in norms, we have a lot more people who spend a lot of time actively searching for data manipulation and other retraction-worthy things in papers.
This is just the science version of the common claim that a recorded increase (or decrease) in the rate of a particular crime, or a particular mental disorder, or some such, is mainly due to changes in how closely we’re looking for it.
Can you clarify the point you’re trying to make with the reference to spurious correlations, Will? I don’t think the author is trying to make any deep claim about causation here, but just pointing out that a growing amount of taxpayer money is wasted due to retractions. (I appreciate the point from other commenters that this is still presumably a small fraction of the total funding though and so might not be as big a concern as the author suggests.)
Sure.
Taken at face value, the claim is that taxpayer funding and number of retractions have increased over time, at rates not hugely different from one another. I think both can almost entirely be accounted for by an increase in the total number of researchers. If you have more researchers producing papers, this will result in both a big increase in funding required and in number of papers retracted without any change in the quality distribution.
I would want to see evidence for a big increase in retractions per number of researchers, researcher hours or some other aggregative measure before taking this seriously as a claim that science has got worse over time. It’s well-known that if you don’t control for the total number of people in a place or doing a thing, all sorts of things will correlate (homicides and priests, ice-cream sales and suicides, etc.).
More substantively, I also disagree with the claim that a big increase in retractions is evidence of scientific decline. Insofar as there has been any increase in the per-capita rate of retractions, I regard this as a sign of increasing epistemic standards, and think both editors and scientists are still way too reluctant to retract papers. It’s like the replication crisis: the problems have always been there, but we only started paying attention to them recently. That’s a good sign, not a bad one.
Thanks for elaborating Will.
Agreed that the increase in funding for science will generally just increase the size of science, and the base assumption should be that the retraction rate will stay the same, which would lead to a roughly proportionate increase in the number of retractions with science funding. The 700% vs. 900% roughly agrees with that assumption (although it could still be that the reasons for retraction change over time).
The idea of increasing retractions being a beneficial sign of better epistemic standards is interesting. My observation is that papers are usually basically only retracted if scientific fraud or misconduct was committed (e.g. falsifying or manipulating research data) - questionable research practices (e.g. P-hacking, optional stopping or HARKing), failure to replicate, or even technical errors don’t usually lead to a retraction (Wikipedia also notes that plagiarism is a common cause of retractions). It is a pity there is no ground truth for scientific misconduct to reference the retraction rate against.
Aside, this summary of the influence of retractions and failure to replicate on later citations may be of interest. Thankfully, retraction usually has a strong reduction on the amount of citations the retracted paper receives.
Thanks Gavin.
I’d be interested in seeing data on the distribution of causes of retraction and how it’s changed over time. I know RetractionWatch likes to say that scientists tend to underestimate the proportion of retractions that are down to fraud. I do think some (many?) retractions are due to serious technical errors with no implication of deliberate fraud or misconduct. I suspect RetractionWatch has data on this.
I’m not claiming that it’s inevitably true that more retractions indicates better community epistemics, but I do think it’s a big part of the story in this case. A paper retraction requires someone to notice that the paper is worthy of retraction, bring that to the editors and, very often, put a lot of pressure on the editors to retract the paper (who are usually extremely reluctant to do so). That requires people to be on the lookout for things that might need to be retracted and willing to put in the time and effort to get it retracted.
In the past this was very rare, and only extremely flagrant fraud or misconduct (or unusually honest scientists retracting their own work) led to retractions. Now, partly as a side consequence of the replication crisis but also more general (and incomplete) changes in norms, we have a lot more people who spend a lot of time actively searching for data manipulation and other retraction-worthy things in papers.
This is just the science version of the common claim that a recorded increase (or decrease) in the rate of a particular crime, or a particular mental disorder, or some such, is mainly due to changes in how closely we’re looking for it.