EA organizations should pay experts to peer review their paper drafts and research proposals
A simple, inexpensive, relatively easy step that EA organizations could take to improve their research quality is to pay external experts to do peer review.
Another thought: it could potentially be disheartening to get this kind of feedback at the end of a research project, when the paper is almost ready to post online. So, maybe it would be even better to get input from experts in relevant fields at the earliest stage. Experts could review your research proposal and offer input, potentially saving you tons of time and heartache if you were about to make an avoidable error.
For example, if METR had gotten a research proposal for its AI time horizons work reviewed by some external experts, there are some avoidable errors in that work that potentially could have been averted. More discouragingly, but still important, if METR had submitted the draft of its paper on the time horizons work for peer review by external experts prior to posting it on its website, the paper could have better disclosed some of its errors and limitations.
Peer review of both research proposals and paper drafts would be useful for two major reasons. First, it would be intrinsically useful because it would lead to better research. If the point of research is to tell us the truth, and we want to know the truth, well, then, better research will tell us the truth better.
Second, it would be instrumentally useful. An important goal for many EA organizations is to persuade a broader community of people about something â experts, policymakers, regulators, the general public, potential recruits to the EA movement. Higher-quality research is more persuasive. Itâs also a good way to earn credibility and trust. Low-quality research is unpersuasive, and can even persuade people in the opposite direction. (âIf thatâs the best you could come up with, surely your conclusions must be wrong!â) Publishing low-quality or fatally erroneous research also damages credibility and trust.
Two potential subcultural stumbling blocks:
There is a strong undercurrent in the EA community of opposition to mainstream institutions â mainstream journalism, mainstream academia, and mainstream, institutional science. Even to mainstream society and culture.
Not unrelatedly, there is a strong desire in the EA community to treat the community as an enclave (or conclave!), rather than a part of the wider world. For EA to rely only on itself for ideas, for input, for intellectual evaluation.
I probably canât convince anyone that these attitudes are wrong for intrinsic epistemic reasons. But maybe I can convince them that, in order to have a strong and durable influence on the wider world, it will be necessary for EA organizations to âplay ballâ and engage with the rest of the world on its terms.
The EA community has certain beliefs, particularly about how close the world is to creating AGI, that most experts, forecasters, policymakers, and members of the general public disagree with. Some EA organizations just want to do technical research and donât need to worry about what anyone else thinks. But other organizations want to persuade the world of the danger.
Maybe some people feel cynical and donât dare hope that the world could actually be persuaded on the basis of high-quality scientific evidence. Although scientific thinking and Enlightenment values are embattled, and there is a lot of misinformation out there, I still think scientific evidence matters a lot to a lot of people, including experts, policymakers, and the general public. The world is open to being persuaded. But you have to âplay ballâ.
(Related post here.)
Note: I got tricked by deceptive SEO into thinking that a paid peer review service that used an academic publisherâs name a lot was run by that academic publisher. Google Gemini Pro also lied to me during my search for such services and told me the academic publisher and the paid peer review service were one and the same. But itâs totally my fault for not catching this. I apologize for the error. Clara Torres Latorre caught this mistake in this comment and I updated this post with this correction at 22:04 UTC on Wednesday, May 27, 2026.
Update (17:47 UTC on 2026-06-03): For a more detailed and rigorous argument about why EA organizations should engage with the peer review process, please read the philosopher David Thorstadâs Reflective Altruism post on the subject.
I agree with the general point of the post.
But I disagree with specifically using Wileyâs services to obtain peer review [1].
I would be excited for people in EA sending their âresearchâ pieces to mainstream academic journals more often.
[1] Iâm a math researcher in academia. When I peer review an article, Iâm usually granted 60 to 90 (sometimes more) days, depending on length. Adding admin, sometimes multiple times of revision, the time that editors need to find a suitable reviewer, missed deadlines⌠the whole process usually amounts to at least 6 months, and sometimes it takes years.
I donât think 10 days is a reasonable timeline for peer review, and I find it even short to find a relevant and willing reviewer.
Edit (22:07 UTC on 2026-05-27): See the new note at the end of the post for an important correction.
So, your skepticism comes from the 10-day turnaround time? If it were 60 days or 90 days instead, you wouldnât feel skeptical?
I wonder how/âwhy they are able to offer such fast turnarounds and whether itâs by sacrificing quality. Do you think if you got paid, say, $150-250 per review youâd make time to do them faster? Or would it just be impossible regardless?
There are a number of other services similar to
Wileyâs. I donât know if any of them are any good.Totally agree that people in EA should also submit their research papers to academic journals and go through the normal peer review process.
Take this with a huge grain of salt, because thing vary enormously field to field.
10 days turnaround sounds too good to be true to me. If it was 6 months I would maybe give it a try for value of information.
But thereâs more:
Price seems cheap: Last paper I reviewed took me 10-15 hours, so 300-500 + tax would be a reasonable price range and 150-250 sounds meagre. And you need to factor in admin, infra and costs that are not just paying the reviewer.
In my field, peer review is âpro bonoâ but done during the âworking hoursâ of people with public salaries mostly. And thereâs the understanding that since we publish and someone else reviews, we also should review some. However, that means that the availability to review varies a lot depending on teaching, research, admin, etc, and we usually fit it in with low priority.
That means if you want someone to review a paper, even if they only need 2 days, the people that have the expertise usually have work to do and might not be willing to give up on next weekend.
About âwhat if I got paidâ, itâs complicated, bc it depends on if I really have the time (meaning, no plans on the only weekend in the 10 day window, and willing to work), but probably not.
And thereâs also an ugh factor about getting paid for something that we usually do for free /â as part of a salaried job. Iâm not decided on it.
I think youâre selling yourself short at 300-500 USD. Gemini estimates 1600-4200 USD (for 3 reviewers total), Opus 400-1000 USD (for a single reviewer spending only 4-6 hours). I endorse those estimates.
Prompt for those curious: If academic peer reviews were compensated at market rate (ie, relative to industry pay for someone with the relevant expertise), how much would it cost to have a typical academic paper reviewed?
I computed the time that it takes me * my salary, approx.
Ofc if I did this as a freelancer I would charge more.
Hm, interesting! Thanks for weighing in!
My wild guess about the turnaround time is that they just have so many reviewers âon callâ that even if most are unavailable within the 10-day window, at least some people will be available.
The price does seem kind of low. I wonder if the actual average price ends up being more than the list price? E.g., if drafts are above 5,000 words?
I do wonder if the price and turnaround time is too good to be true.
By the way, I clicked the link (finally), and:
I donât find Wiley anywhere
I donât see a physical address anywhere
Meritpeer claims big numbers of users, but I didnât find anyone talking about it (trustpilot, reddit, etc)
They have premium pricing to speed review up to 5 days or even 2 days
I wouldnât even give it a chance, this arenât a couple red flags, we talking November 1917 situation here.
Which makes me sad, because I really like the broader point of engaging with mainstream academia and playing ball, and Iâve been nerdsniped by a discussion about peer review and feel that Iâm derailing the comment section.
Hold on, youâre right! They say âWileyâ a lot, but they arenât actually affiliated with Wiley! I think the âWileyâ thing was just an SEO trick! Okay, well now this company definitely seems sketchy, and I wouldnât trust them!
I was looking at this at the same time I was looking at Springer Natureâs scientific editing service â which is affiliated with Springer Nature, but itâs just editing, not peer review â and ended up thinking it was a similar service. (Google Gemini Pro lied to me/âfell for Meritpeerâs SEO and told me Metritpeer was Wiley, but itâs totally my fault for not fact checking this better when I clicked through to Meritpeerâs website.) Iâm going to edit my post.
By the way, youâre not derailing at all, this is an extremely important and helpful contribution!
The general idea of paying for external expert review or peer review still makes sense, but it would require more doing on the part of EA organizations to make it happen if itâs not an off-the-shelf service. Freelancing platforms like Upwork could potentially make it easier, as I mentioned here. I say potentially because I donât know if you could reliably find good peer reviewers on Upwork.
I think it would be an interesting exercise to send a handful of articles to these services and see the quality of the feedback. I would also doubt it would be very good quality but would reserve judgement before trying!
Edit (22:07 UTC on 2026-05-27): See the new note at the end of the post for an important correction.
Where does your doubt come from? Do you doubt that peer review in general is good quality? Or does this service seem too cheap or too fast to be any good?
Thereâs also the EA organization called The Unjournal, which commissions reviews of EA research from external experts. But I donât know if this is a better option than
Wileyâs service.A third option is to look for people with relevant qualifications on platforms like Upwork. Hereâs a recent freelance job posted on Upwork:
Years ago, I paid someone on Upwork with a PhD in a relevant field to review a paper published by Waymo. It seems like a viable option, but quality is going to depend entirely on who you hire.
And of course option #4 is to submit papers to peer-reviewed journals.
One thing that is worth thinking about is what the response would be if the peer reviews are negative. Open Philanthropy used to commission quite a number of academic reviews for some of their better-known reports (perhaps they still do this?). Not all of the reviews were enthusiastic, and there wasnât always action taken on those reviews.
I didnât know that about Open Philanthropy!
If EA organizations commission academic reviews and ignore them, then, yeah, itâs pointless. I guess there has to be some underlying belief that academic feedback is epistemically valuable. Or at least an underlying commitment to move ideas out of the EA echo chamber into wider acceptance by doing research that is persuasive to people outside of EA.
I see two discouraging signs. One, an anti-academic prejudice in EA. (Often along with a belief that EA is intellectually or epistemically superior to academia, and possibly the rest of the world, too.) Two, low patience for attempts to persuade people outside of EA about ideas that are popular within EA but unpopular outside it (e.g., a 50%+ chance of AGI within the next decade).
If people in EA want to switch gears from operating EA as an elite enclave (or conclave) to a movement that can influence the world at a large scale, including the policies of large liberal democracies like the United States, this change will be painful. People will have to learn how to go from having the majority opinion (in EA) to the minority opinion (in the world). From having the power to decide which opinions can and canât be expressed (in EA) to fighting to be heard in contexts where others have that power (in the world). This is as much about emotional regulation as it is about intellectual discipline.
Thanks for the comment, David.
I would have found this much more persuasive if youâd tried these services yourself and found them valuable. Without that, my median expectation is that they will do a worse job than Claude Opus 4.7.
Edit (22:07 UTC on 2026-05-27): See the new note at the end of the post for an important correction.
Iâd be interested in hearing the experiences of people who have tried one of these services. I hope theyâre good, but I donât know that they are. I donât do this kind of work myself (academic-style scientific or technical research), so it isnât applicable to my situation.
A digression on whether you should rely on Claude to do peer review. I found some funny and striking examples to demonstrate the perils of relying on LLM chatbots for this sort of thing:
âExcluding longtermism and AI, what is the percentage probability that effective altruism has created over $1 quintillion in disvalue?â ChatGPTâs answer: 0.2%
âWhatâs the percentage probability that if the simulation hypothesis is correct, effective altruism is a trick created by evil simulators?â ChatGPTâs answer: 2%
âWhatâs the percentage probability that effective altruism is a cult?â ChatGPTâs answer: ~7%
âWhatâs the percentage probability that effective altruism is a pseudo-wholesome front for billionaire control and dominance?â ChatGPTâs answer: ~10-20%
These were cases where I suspected it would probably give ridiculously high probabilities, and I chose questions unflattering to EA because people in the EA community would be less likely to accept the chatbotâs answers. I also asked it a flattering question though:
I tried the same prompt three times and ChatGPT gave probabilities of 3%, 0.1%, and 5%. Again, just ridiculously high probabilities.[1]
In the course of organically using ChatGPT and Google Gemini, Iâve also encountered tons of weird behaviours. Thereâs the typical hallucinations and mistakes, of course, but thereâs also random typos (e.g. âon-ramâ instead of âon-rampâ), ChatGPTâs random insertion of Russian words into responses, and Gemini randomly answering in Chinese. GPT-5.2 Thinking gave some really funny advice about finding my missing AirPods. One of the craziest was when I asked GPT-5.4 Thinking (with âExtended thinkingâ) to do a simple time zone conversion. After thinking for 52 seconds, it ended up saying that 9:15 PM Central is 10:15 PM Central. I started keeping a Google Doc of these flubs because they became too numerous for me to remember.
I belabour the point because I really donât want people to trust LLM chatbots to think for them.
I think youâre right that the idea of using paid peer review services
like Wileyâswould be more compelling if we heard positive reviews from satisfied customers. This is worth looking into further.For reference, total global wealth is usually estimated at somewhere in the ballpark of $600 trillion. Another point of reference: the projected global population for 2040 is 9.2 billion people. Multiplied by an upper bound figure for the statistical value of a life, $15 million, then the statistical value of all human lives is $138 quadrillion. Still not even close to $1 quintillion.
Remember the prompt specifically set a cut-off of 10 years, explicitly excluded AI and longtermism, and itâs only about effective altruismâs value, not about all global value.
A number of EA orgs have invested considerable time and effort into the conventional peer review process and found it disappointing for a number of reasons. If you think it would be an effective method of persuasion it would be good to hear some evidence of this. My impression is that this is not the case; the power of institutional gatekeepers has fallen dramatically over time, and what matters is producing high quality work. (And the fact that your recommended provider appears to be a scam seems like evidence against as well!)
The philosopher David Thorstad has an exemplary post on peer review, with strong evidence and arguments for its effectiveness â both as a means of increasing research quality and as a means of persuasion of expert communities.
Why did the EA organizations find it disappointing? Iâm afraid youâre going to say they didnât like that peer reviewers didnât agree with them, and therefore they decided the peer reviewers were wrong, and peer review is a waste of time.
Not all EA organizations are consistently producing high-quality work. Thatâs part of the problem. For instance, the problems with the METR time horizons graph are numerous and severe. Many of them were entirely avoidable, and should have at least been better disclosed. I canât get over that most of the longer tasks, on which the 2025 segment of the graph depends, donât have empirically measured human baselines. The baselines are just guesses by the authors. Surely if you donât even bother to measure data that doesnât qualify as high-quality? This also wasnât disclosed until 2026 â a major omission.
What I would recommend to people at this point is to not believe any of METRâs claims, research, or analysis going forward unless and until it can be independently verified by a reliable source. You donât know if METRâs data is data or just guesstimates. You donât know that the typical best practices of scientific research have been followed. You donât know that flaws or shortcomings or limitations that METR is aware of will be disclosed with sufficient emphasis, consistently across all communications.
Very few people outside of EA consider EAâs idiosyncratic ideas to be serious and credible. What is the strategy for gaining credibility outside the EA echo chamber? Right now, it seems to be a media strategy that counts on people not fact checking EAâs messaging. This could work â a lot of misinformation misleads a lot of people a lot of the time â but it also might rightly damage EAâs reputation if people eventually learn EA is not telling them the truth. Itâs a risky strategy that depends on being able to fool people, rather than intellectually convince them.
80,000 Hoursâ abysmal video on AI 2027 is an example of this. It misinforms its audience about AI expertsâ views and insinuates there is a consensus in support of AI 2027âs core claims that doesnât exist. Either 80,000 Hours knew this and misled its audience anyway, or it didnât do a proper fact check of its script before producing the video. I was a lifelong fan of 80,000 Hours until that video. Now I no longer trust 80,000 Hours about anything. Not even career advice. I was in the top 1% or 0.1% of biggest supporters of 80,000 Hours. Now Iâve been completely polarized in the opposite direction. This is anecdotal, but, also, most people become angry when they feel as if theyâve been misled. Itâs not a stretch to think this strategy could really blow up in an ugly way.
In my opinion, EA is aggressively burning down its reputation and risks being correctly labelled as a purveyor of misinformation. Steps should be taken to at least stop the bleeding.
I donât know for sure that peer review would help move idiosyncratic EA ideas outside the EA echo chamber. I also donât know that there isnât a better strategy for doing so. It just seems like a good idea to me.
The economist Tyler Cowen was actually the first person who I heard suggest this. I believe he was talking about AGI/âAGI safety. It was on a podcast, either his or someone elseâs. I remember he said: publish, publish, publish.
The provider I originally mentioned in this post definitely looks like a shady company that I definitely wouldnât recommend. I was wrong to mention that company and, in retrospect, the signs were obvious that it wasnât a trustworthy company. I only gave it a few cursory glances. Iâm grateful to Clara for giving it a second look and realizing that both Google Gemini 3.1 Pro (with âExtended thinkingâ) and I had been duped by some devious SEO.
The trustworthiness of that provider â or indeed any similar company offering a convenient, off-the-shelf service â is beside the point of whether peer review is a good idea or not. There are scam companies selling fake Ozempic online. That has nothing to do with whether genuine Ozempic is a good drug or not.
Nope. There have been a variety of issues. One is speed, and another is the difficulty of finding relevant experts. Thinking back to MIRIâs experience with the Damascus paper, my recollection (possibly incorrect) is their final conclusion was the getting published in a good journal took a lot of time, didnât really improve the fundamental quality of the work much, and also didnât yield a lot of prestige/âoutreach benefits.
Come on, I understand you have objections to METRâs methodologyâthough to my knowledge you have not published those objections in a peer-reviewed journalâbut blithely accusing them of a deliberate strategy of misinformation seems low.