Idea: Pay experts to provide detailed critiques / analyses of EA ideas
For example, could we pay experts to spend time reading and giving careful detailed responses to specific articles or posts on EA-related topics? I would particularly love to get relevant non-EA or EA-adjacent experts to provide point-by-point responses to key claims made by AI risk proponents. (I’m picking AI risk as an example and because it’s one of the topics I spend most of my own research time on, but this idea is certainly not limited to AI risk.)
Note that this is not a proposal to just hire those experts to work on EA problems. Rather it’s a proposal to pay those experts just to spend a few hours or days to do a careful point-by-point analysis of some article or topic. My impression is that organizations hire experts to do analogous things on a regular basis—for example paying them to do short contracting-style work, or paying for their time and travel expenses to attend workshops, or paying for them to spend time writing commissioned articles or books.
Many experts are of course also strongly motivated by prestige factors, especially since their career incentive structures tend to reward prestige. This is part (most?) of why many experts will readily agree to write review articles for prestigious journals, give talks at conferences, or participate as a panelist in a workshop. I am not sure exactly how to use prestige factors here, but I’m guessing it should be possible and I’d be very interested to hear ideas for how to do this. Maybe create dedicated workshops for this at relevant conferences?
For many critics in particular I think this would appeal to their already existing love of critiquing whatever it is they love to critique, so it might be a relatively easy sell for them. I frequently find it frustrating when reading what these critics write, because I often have particular arguments or sets of arguments that I’d really like them to address but they don’t. Even if I attend a Q&A session at a talk they give, I usually only have the chance to ask very short targeted questions, and if I mess up how I phrase the question then they often don’t respond to what I really wanted to ask. In the past I have frequently found myself wanting to show them particular articles and ask them to give point-by-point responses. The proposal in this post is my idea for how to get them to actually give those point-by-point responses. (But note that the idea is not limited to prominent critics. I would also love to ask other relevant experts for their point-by-point analyses of key claims. That might actually turn out to be of higher value than asking already vocal and opinionated critics to do the same.)
I want to run this at some point. Pro statistical consultants could handle many posts, and philosophy grad students many of the rest. I was thinking in terms of one review per important post here.
Riffing further:
Academic peer review generally doesn’t run code, so this is one way we could surpass them.
Also publishing the reviews,
and doing them within a month, rather than within a year.
“And doing them within a month, rather than within a year.”—Might be hard to determine what posts are important after just a month?
I’d be fine with some false positives
I think Open Philanthropy has done some of this. For example:
https://forum.effectivealtruism.org/posts/7JxsXYDuqnKMqa6Eq/ai-timelines-where-the-arguments-and-the-experts-stand
In some cases it might be easier to do this as a structured interview rather than asking for written analyses. For example, I could imagine that it might be possible to create a podcast where guests are given an article or two to read before the interview, and then the interviewer asks them for their responses on a point-by-point basis. This would also allow the interviewer (if they’re particularly good) to do follow-up questions as necessary. On the other hand, my personal impression is that written analyses tend to be more carefully argued and thought through than real-time interviews.
Have you read the transcripts from Vael Gates’s structured interviews with people working in AI about safety stuff, they seem to have done something pretty close to what you’re asking.
Good point! I started reading those a while ago but got distracted and never got back to them. I’ll try looking at them again.