I upvoted this post and am strongly in favour of more institutional expertise in the community.
However, to take the example of peer review specifically, my sense is that:
academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc
people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,
the peer review system didnât obviously arise from a really intentional and thoughtful design effort (though maybe this is just my historical ignorance?), and there are institutional-inertia reasons why it would be hard to replace with something better even once we had a good proposal,
at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didnât exist, so it really arose in an environment quite different from the one itâs now in.
It feels to me both like the consensus view isnât as strongly in favour of peer review as you suggest, and that there are some structural reasons to think that the dominance of peer review in academic contexts isnât so strong an indicator of its fitness.
I recognise my above criticisms have holes in them, but they seemed worth airing anyway, to at least gesture at why people might end up where they are on this topic.
(Also, obviously Iâve done nothing to demonstrate that forum posts arenât worse on every axis. I just think that if weâre really in the situation of âwe should use this moderately terrible system instead of this extremely terrible systemâ, we need to acknowledge that if weâre going to get people who can see the terribleness on board.)
academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc
I think we could quibble on the scale and importance of all of these points, but Iâm not prepared to confidently deny any of them. The important I want to make is: compared to what alternatives? The problem is hard, and even the best solution can be expected to have many visible imperfections. How persuaded you be by a revoluationary enumerating the many weaknesses of democratic government without comparing them to a proposed (often imagined) alternative?
people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,
Again, compared to what alternative? Iâm guessing they would say that at a company, the âidea peopleâ get their ideas tested by the free market, and this grounds them, and makes their work practical. I am willing to believe that free-market-testing is a more reliable institution than peer review for evaluating certain propositions, and I am willing to buy that this is conventionally well-understood. But for many propositions, I would think there is no way to âproductizeâ it such that profitability of the product is logically equivalent to the truth of the proposition. (This may not be the alternative you had in mind at all, if you had an alternative in mind).
the peer review system didnât obviously arise from a really intentional and thoughtful design effort
This is definitely not a precondition for a successful social institution.
at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didnât exist, so it really arose in an environment quite different from the one itâs now in.
But conventional wisdom persists in endorsing peer-review anyway!
This is definitely not a precondition for a successful social institution.
I want to differentiate two kinds of success for a social institution:
âreproductiveâ success, by analogy with evolution: how well the institution establishes and maintains itself as dominant,
success at stated goals: for peer review, success at finding the truth, producing high quality research, etc.
Your argument seems to be (at least in part) that because peer review has achieved success 1, that is strong evidence that itâs better than its alternatives at success 2. My argument (in part) is that this is only true if the two kinds of success have some mechanism tying them together. Some example mechanisms could be:
the institution achieved reproductive success by means of being pushed really hard by people motivated by the desire to build and maintain a really high quality system,
the institution is easy to replace with better systems, and better systems are easy to try, so the fact that it hasnât been replaced must mean better systems are hard to find.
I donât think either of these things are true of peer review. (The second is true of AWS, for example.) So whatâs the mechanism that established peer review as the consensus system that relates to it being a high quality system?
(Iâm not saying I have alternatives, just that âconsensus means a thing is goodâ is only sometimes a good argument.)
I upvoted this post and am strongly in favour of more institutional expertise in the community.
However, to take the example of peer review specifically, my sense is that:
academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc
people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,
the peer review system didnât obviously arise from a really intentional and thoughtful design effort (though maybe this is just my historical ignorance?), and there are institutional-inertia reasons why it would be hard to replace with something better even once we had a good proposal,
at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didnât exist, so it really arose in an environment quite different from the one itâs now in.
It feels to me both like the consensus view isnât as strongly in favour of peer review as you suggest, and that there are some structural reasons to think that the dominance of peer review in academic contexts isnât so strong an indicator of its fitness.
I recognise my above criticisms have holes in them, but they seemed worth airing anyway, to at least gesture at why people might end up where they are on this topic.
(Also, obviously Iâve done nothing to demonstrate that forum posts arenât worse on every axis. I just think that if weâre really in the situation of âwe should use this moderately terrible system instead of this extremely terrible systemâ, we need to acknowledge that if weâre going to get people who can see the terribleness on board.)
I think we could quibble on the scale and importance of all of these points, but Iâm not prepared to confidently deny any of them. The important I want to make is: compared to what alternatives? The problem is hard, and even the best solution can be expected to have many visible imperfections. How persuaded you be by a revoluationary enumerating the many weaknesses of democratic government without comparing them to a proposed (often imagined) alternative?
Again, compared to what alternative? Iâm guessing they would say that at a company, the âidea peopleâ get their ideas tested by the free market, and this grounds them, and makes their work practical. I am willing to believe that free-market-testing is a more reliable institution than peer review for evaluating certain propositions, and I am willing to buy that this is conventionally well-understood. But for many propositions, I would think there is no way to âproductizeâ it such that profitability of the product is logically equivalent to the truth of the proposition. (This may not be the alternative you had in mind at all, if you had an alternative in mind).
This is definitely not a precondition for a successful social institution.
But conventional wisdom persists in endorsing peer-review anyway!
I want to differentiate two kinds of success for a social institution:
âreproductiveâ success, by analogy with evolution: how well the institution establishes and maintains itself as dominant,
success at stated goals: for peer review, success at finding the truth, producing high quality research, etc.
Your argument seems to be (at least in part) that because peer review has achieved success 1, that is strong evidence that itâs better than its alternatives at success 2. My argument (in part) is that this is only true if the two kinds of success have some mechanism tying them together. Some example mechanisms could be:
the institution achieved reproductive success by means of being pushed really hard by people motivated by the desire to build and maintain a really high quality system,
the institution is easy to replace with better systems, and better systems are easy to try, so the fact that it hasnât been replaced must mean better systems are hard to find.
I donât think either of these things are true of peer review. (The second is true of AWS, for example.) So whatâs the mechanism that established peer review as the consensus system that relates to it being a high quality system?
(Iâm not saying I have alternatives, just that âconsensus means a thing is goodâ is only sometimes a good argument.)