I upvoted this post and am strongly in favour of more institutional expertise in the community.
However, to take the example of peer review specifically, my sense is that:
academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc
people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,
the peer review system didn’t obviously arise from a really intentional and thoughtful design effort (though maybe this is just my historical ignorance?), and there are institutional-inertia reasons why it would be hard to replace with something better even once we had a good proposal,
at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didn’t exist, so it really arose in an environment quite different from the one it’s now in.
It feels to me both like the consensus view isn’t as strongly in favour of peer review as you suggest, and that there are some structural reasons to think that the dominance of peer review in academic contexts isn’t so strong an indicator of its fitness.
I recognise my above criticisms have holes in them, but they seemed worth airing anyway, to at least gesture at why people might end up where they are on this topic.
(Also, obviously I’ve done nothing to demonstrate that forum posts aren’t worse on every axis. I just think that if we’re really in the situation of “we should use this moderately terrible system instead of this extremely terrible system”, we need to acknowledge that if we’re going to get people who can see the terribleness on board.)
academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc
I think we could quibble on the scale and importance of all of these points, but I’m not prepared to confidently deny any of them. The important I want to make is: compared to what alternatives? The problem is hard, and even the best solution can be expected to have many visible imperfections. How persuaded you be by a revoluationary enumerating the many weaknesses of democratic government without comparing them to a proposed (often imagined) alternative?
people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,
Again, compared to what alternative? I’m guessing they would say that at a company, the “idea people” get their ideas tested by the free market, and this grounds them, and makes their work practical. I am willing to believe that free-market-testing is a more reliable institution than peer review for evaluating certain propositions, and I am willing to buy that this is conventionally well-understood. But for many propositions, I would think there is no way to “productize” it such that profitability of the product is logically equivalent to the truth of the proposition. (This may not be the alternative you had in mind at all, if you had an alternative in mind).
the peer review system didn’t obviously arise from a really intentional and thoughtful design effort
This is definitely not a precondition for a successful social institution.
at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didn’t exist, so it really arose in an environment quite different from the one it’s now in.
But conventional wisdom persists in endorsing peer-review anyway!
This is definitely not a precondition for a successful social institution.
I want to differentiate two kinds of success for a social institution:
“reproductive” success, by analogy with evolution: how well the institution establishes and maintains itself as dominant,
success at stated goals: for peer review, success at finding the truth, producing high quality research, etc.
Your argument seems to be (at least in part) that because peer review has achieved success 1, that is strong evidence that it’s better than its alternatives at success 2. My argument (in part) is that this is only true if the two kinds of success have some mechanism tying them together. Some example mechanisms could be:
the institution achieved reproductive success by means of being pushed really hard by people motivated by the desire to build and maintain a really high quality system,
the institution is easy to replace with better systems, and better systems are easy to try, so the fact that it hasn’t been replaced must mean better systems are hard to find.
I don’t think either of these things are true of peer review. (The second is true of AWS, for example.) So what’s the mechanism that established peer review as the consensus system that relates to it being a high quality system?
(I’m not saying I have alternatives, just that “consensus means a thing is good” is only sometimes a good argument.)
I upvoted this post and am strongly in favour of more institutional expertise in the community.
However, to take the example of peer review specifically, my sense is that:
academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc
people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,
the peer review system didn’t obviously arise from a really intentional and thoughtful design effort (though maybe this is just my historical ignorance?), and there are institutional-inertia reasons why it would be hard to replace with something better even once we had a good proposal,
at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didn’t exist, so it really arose in an environment quite different from the one it’s now in.
It feels to me both like the consensus view isn’t as strongly in favour of peer review as you suggest, and that there are some structural reasons to think that the dominance of peer review in academic contexts isn’t so strong an indicator of its fitness.
I recognise my above criticisms have holes in them, but they seemed worth airing anyway, to at least gesture at why people might end up where they are on this topic.
(Also, obviously I’ve done nothing to demonstrate that forum posts aren’t worse on every axis. I just think that if we’re really in the situation of “we should use this moderately terrible system instead of this extremely terrible system”, we need to acknowledge that if we’re going to get people who can see the terribleness on board.)
I think we could quibble on the scale and importance of all of these points, but I’m not prepared to confidently deny any of them. The important I want to make is: compared to what alternatives? The problem is hard, and even the best solution can be expected to have many visible imperfections. How persuaded you be by a revoluationary enumerating the many weaknesses of democratic government without comparing them to a proposed (often imagined) alternative?
Again, compared to what alternative? I’m guessing they would say that at a company, the “idea people” get their ideas tested by the free market, and this grounds them, and makes their work practical. I am willing to believe that free-market-testing is a more reliable institution than peer review for evaluating certain propositions, and I am willing to buy that this is conventionally well-understood. But for many propositions, I would think there is no way to “productize” it such that profitability of the product is logically equivalent to the truth of the proposition. (This may not be the alternative you had in mind at all, if you had an alternative in mind).
This is definitely not a precondition for a successful social institution.
But conventional wisdom persists in endorsing peer-review anyway!
I want to differentiate two kinds of success for a social institution:
“reproductive” success, by analogy with evolution: how well the institution establishes and maintains itself as dominant,
success at stated goals: for peer review, success at finding the truth, producing high quality research, etc.
Your argument seems to be (at least in part) that because peer review has achieved success 1, that is strong evidence that it’s better than its alternatives at success 2. My argument (in part) is that this is only true if the two kinds of success have some mechanism tying them together. Some example mechanisms could be:
the institution achieved reproductive success by means of being pushed really hard by people motivated by the desire to build and maintain a really high quality system,
the institution is easy to replace with better systems, and better systems are easy to try, so the fact that it hasn’t been replaced must mean better systems are hard to find.
I don’t think either of these things are true of peer review. (The second is true of AWS, for example.) So what’s the mechanism that established peer review as the consensus system that relates to it being a high quality system?
(I’m not saying I have alternatives, just that “consensus means a thing is good” is only sometimes a good argument.)