We should use blogposts, Google Docs, and similar works as accessible ways of opening discussions and providing preliminary thoughts, but rely on peer-reviewed research when making important decisions, creating educational materials, and communicating to the public
Bold type mine. No I think peer review is cumbersome and that to use it would slow work down a lot. Are there never mistakes in peer reviewed science? No, there are. I think we should aim to build better systems of peer review.
My take is different. As a working scientist/engineer in the hard sciences, I use peer-reviewed research when possible, but I temper that with phone calls to companies, emails to other labs, posts on ResearchGate, informal conversations with colleagues, and of course my own experiments, mechanistic models, and critical thinking skills. Peer-reviewed research is nearly always my starting point because it’s typically more information and data-rich and specific than blog posts, and because the information I need is more often contained within peer-reviewed research than in blog posts.
That said, there are a lot of issues and concerns raised when blog posts are too heavily a source (although here that’s very much the pot calling the kettle black, with most of the footnotes being the author’s own unsourced personal opinions). When people lean too heavily on blog posts, it may illustrate that they’re unfamiliar with the scientific literature relevant to the issue, and that they themselves have mostly learned about the information by consuming other blog posts. Also, a compelling post that’s full of blog post links (or worse, unsourced claims) gives the interested reader little opportunity to check the underpinnings of the argument or get connected with working scientists in the field.
I’m fine with using the medium of blog posts to convey an idea, or of citing blog posts in specific circumstances. Where a peer-reviewed source is available, I think it’s better to either use that, or to cite it and give the blog post as an accessible alternative.
Are there never mistakes in peer reviewed science? No, there are
The question isn’t “are there zero mistakes”, the question is, “is peer reviewed research generally of higher quality than blogposts?”. To which the answer is obviously yes (at least in my opinion), although the peer review process is cumbersome and slow, and so will have less output and cover less area.
When there are both peer reviewed research and blogposts on a subject matter, I think the peer-reviewed research will be of higher quality and more correct a vast majority of the time.
Upvotes on an internet forum are not a good replacement for peer review. I’m surprised I even have to argue for this, but here goes:
the vast majority of people upvoting/downvoting are not experts in the topic of the blog post.
The vast majority of upvoting/downvoting occurs before a blogpost has been thoroughly checked for accuracy. If theres a serious mistake in a blogpost, and it’s not caught right away, almost no-one will see it.
Upvoting/downvoting is mostly a response to the percieved effort of a post and on whether they personally agree with it.
Yes, peer review is flawed, but the response isn’t to revert to blogposts, it’s to build a better system.
I think that peer review is so poor that probably just the forum alone produces work that is less in need of replication. I guess that’s not really about the system.
And yes, we should build a better system, but still. Peer review vs upvotes on journal sites, I would pick the latter.
Bold type mine. No I think peer review is cumbersome and that to use it would slow work down a lot. Are there never mistakes in peer reviewed science? No, there are. I think we should aim to build better systems of peer review.
My take is different. As a working scientist/engineer in the hard sciences, I use peer-reviewed research when possible, but I temper that with phone calls to companies, emails to other labs, posts on ResearchGate, informal conversations with colleagues, and of course my own experiments, mechanistic models, and critical thinking skills. Peer-reviewed research is nearly always my starting point because it’s typically more information and data-rich and specific than blog posts, and because the information I need is more often contained within peer-reviewed research than in blog posts.
That said, there are a lot of issues and concerns raised when blog posts are too heavily a source (although here that’s very much the pot calling the kettle black, with most of the footnotes being the author’s own unsourced personal opinions). When people lean too heavily on blog posts, it may illustrate that they’re unfamiliar with the scientific literature relevant to the issue, and that they themselves have mostly learned about the information by consuming other blog posts. Also, a compelling post that’s full of blog post links (or worse, unsourced claims) gives the interested reader little opportunity to check the underpinnings of the argument or get connected with working scientists in the field.
I’m fine with using the medium of blog posts to convey an idea, or of citing blog posts in specific circumstances. Where a peer-reviewed source is available, I think it’s better to either use that, or to cite it and give the blog post as an accessible alternative.
The question isn’t “are there zero mistakes”, the question is, “is peer reviewed research generally of higher quality than blogposts?”. To which the answer is obviously yes (at least in my opinion), although the peer review process is cumbersome and slow, and so will have less output and cover less area.
When there are both peer reviewed research and blogposts on a subject matter, I think the peer-reviewed research will be of higher quality and more correct a vast majority of the time.
Compared to EA blog posts weighted by karma? The Answer is not obviously yes in my opinion. I think we’ll fare better in the replication crisis.
Upvotes on an internet forum are not a good replacement for peer review. I’m surprised I even have to argue for this, but here goes:
the vast majority of people upvoting/downvoting are not experts in the topic of the blog post.
The vast majority of upvoting/downvoting occurs before a blogpost has been thoroughly checked for accuracy. If theres a serious mistake in a blogpost, and it’s not caught right away, almost no-one will see it.
Upvoting/downvoting is mostly a response to the percieved effort of a post and on whether they personally agree with it.
Yes, peer review is flawed, but the response isn’t to revert to blogposts, it’s to build a better system.
And yet argue it you shall.
I think that peer review is so poor that probably just the forum alone produces work that is less in need of replication. I guess that’s not really about the system.
And yes, we should build a better system, but still. Peer review vs upvotes on journal sites, I would pick the latter.
Maybe we could discuss it in the comments of https://forum.effectivealtruism.org/topics/peer-review