academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc
I think we could quibble on the scale and importance of all of these points, but I’m not prepared to confidently deny any of them. The important I want to make is: compared to what alternatives? The problem is hard, and even the best solution can be expected to have many visible imperfections. How persuaded you be by a revoluationary enumerating the many weaknesses of democratic government without comparing them to a proposed (often imagined) alternative?
people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,
Again, compared to what alternative? I’m guessing they would say that at a company, the “idea people” get their ideas tested by the free market, and this grounds them, and makes their work practical. I am willing to believe that free-market-testing is a more reliable institution than peer review for evaluating certain propositions, and I am willing to buy that this is conventionally well-understood. But for many propositions, I would think there is no way to “productize” it such that profitability of the product is logically equivalent to the truth of the proposition. (This may not be the alternative you had in mind at all, if you had an alternative in mind).
the peer review system didn’t obviously arise from a really intentional and thoughtful design effort
This is definitely not a precondition for a successful social institution.
at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didn’t exist, so it really arose in an environment quite different from the one it’s now in.
But conventional wisdom persists in endorsing peer-review anyway!
This is definitely not a precondition for a successful social institution.
I want to differentiate two kinds of success for a social institution:
“reproductive” success, by analogy with evolution: how well the institution establishes and maintains itself as dominant,
success at stated goals: for peer review, success at finding the truth, producing high quality research, etc.
Your argument seems to be (at least in part) that because peer review has achieved success 1, that is strong evidence that it’s better than its alternatives at success 2. My argument (in part) is that this is only true if the two kinds of success have some mechanism tying them together. Some example mechanisms could be:
the institution achieved reproductive success by means of being pushed really hard by people motivated by the desire to build and maintain a really high quality system,
the institution is easy to replace with better systems, and better systems are easy to try, so the fact that it hasn’t been replaced must mean better systems are hard to find.
I don’t think either of these things are true of peer review. (The second is true of AWS, for example.) So what’s the mechanism that established peer review as the consensus system that relates to it being a high quality system?
(I’m not saying I have alternatives, just that “consensus means a thing is good” is only sometimes a good argument.)
I think we could quibble on the scale and importance of all of these points, but I’m not prepared to confidently deny any of them. The important I want to make is: compared to what alternatives? The problem is hard, and even the best solution can be expected to have many visible imperfections. How persuaded you be by a revoluationary enumerating the many weaknesses of democratic government without comparing them to a proposed (often imagined) alternative?
Again, compared to what alternative? I’m guessing they would say that at a company, the “idea people” get their ideas tested by the free market, and this grounds them, and makes their work practical. I am willing to believe that free-market-testing is a more reliable institution than peer review for evaluating certain propositions, and I am willing to buy that this is conventionally well-understood. But for many propositions, I would think there is no way to “productize” it such that profitability of the product is logically equivalent to the truth of the proposition. (This may not be the alternative you had in mind at all, if you had an alternative in mind).
This is definitely not a precondition for a successful social institution.
But conventional wisdom persists in endorsing peer-review anyway!
I want to differentiate two kinds of success for a social institution:
“reproductive” success, by analogy with evolution: how well the institution establishes and maintains itself as dominant,
success at stated goals: for peer review, success at finding the truth, producing high quality research, etc.
Your argument seems to be (at least in part) that because peer review has achieved success 1, that is strong evidence that it’s better than its alternatives at success 2. My argument (in part) is that this is only true if the two kinds of success have some mechanism tying them together. Some example mechanisms could be:
the institution achieved reproductive success by means of being pushed really hard by people motivated by the desire to build and maintain a really high quality system,
the institution is easy to replace with better systems, and better systems are easy to try, so the fact that it hasn’t been replaced must mean better systems are hard to find.
I don’t think either of these things are true of peer review. (The second is true of AWS, for example.) So what’s the mechanism that established peer review as the consensus system that relates to it being a high quality system?
(I’m not saying I have alternatives, just that “consensus means a thing is good” is only sometimes a good argument.)