Thanks for posting this. I have lot’s of thoughts about lots of things, that will take longer to think about. So I start with one of the easier questions.
Regarding pear review, you suggest
EAs should place a greater value on scientific rigour
We should use blogposts, Google Docs, and similar works as accessible ways of opening discussions and providing preliminary thoughts, but rely on peer-reviewed research when making important decisions, creating educational materials, and communicating to the public
When citing a blogpost, we should be clear about its scope, be careful to not overstate its claims, and not cite it as if it is comparable to a piece of peer-reviewed research
Have you had any interaction with the academic pear review system? Have you seen some of the stuff that passes though pear review? I’m in favour of scientific rigour, but I don’t think pear review solves that. In reality, my impression is that academia relies as much as name recognition and informal consensus mechanisms, as we (the blogpost community) does. The only reason academia has higher standards (in some fields) is that these fields are older and have developed a consensus around what methods are good enough vs not good enough.
I think pear review have the potential to be good. My impression is that it does really work in math, and that this has a lot to do with that reviewers receive recognition. But in many other fields it mainly serves to slow down research publication, and to gatekeep papers that are not written in the right format, or are not sufficiently interesting.
I did a PhD in theoretical physics and I was not impressed by the pear review responses I got on my paper. It was almost always very shallow comments. Which is not suppressing given that at least in that corner of physics pear review was unpaid and unrecognised work.
EA institutions should commission peer-reviewed research far more often, and be very cautious of basing decisions on shallow-dives by non-experts
For important questions, commission a person/team with relevant expertise to do a study and subject it to peer review
For the most important/central questions, commission a structured expert elicitation
Can one just do that? Isn’t it very hard to find a person who has the right expertise and who you can verify have the right expertise?
I did a PhD in theoretical physics and I was not impressed by the pear review responses I got on my paper. It was almost always very shallow comments. Which is not suppressing given that at least in that corner of physics pear review was unpaid and unrecognised work.
If you already did the work to make a high quality paper, then peer review probably won’t add much. But the point is actually to prevent poor quality, incorrect research from getting through, and to raise the quality of publications as a whole.
My PhD was on computational physics, and yeah, the peer review didn’t add much to my papers. But because I knew it was there, I made sure to put a ton of work to make sure it was error free and every part of it was high quality. If I knew I could get the same reward of publication by being sloppy or lazy, I might be tended to do that. I certainly put orders of magnitude more effort into my papers than I do to my blog posts.
I certainly don’t think peer review is perfect, or that every post should be peer reviewed or anything. But I think that research that has to pass that bar tends to be superior to work that doesn’t.
This comment is pretty long, but TLDR: peer review and academia have their own problems, some similar to EA, some not. Maybe a hybrid approach works, and maybe we should consult with people with expertise in social organisation of science.
To some extent I agree with this. Whilst I’ve been wanting more academic rigour in X-Risk for a while, peer review certainly is no perfect panacea, although I think it is probably better than the current culture of deferring to blog posts as much as we do.
I think you are right that traditional academia really has its problems, and name recognition is also still an issue (eg Nobel Prize winners are 70% more likely to get through peer review etc.). Nonetheless, certainly from the field (solar geoengineering) that I have been in, name recognition and agreement with the ‘thought leaders’ is definitely less incentivised than in EA.
One potential response is to think a balance between peer review, the current EA culture and commissioned reports is a good balance. We could set up an X-Risk journal with some editors and reviewers who are a) dedicated to pluralism and b) will publish things that are methodologically sound irrespective of result. Alternatively, a sort of open peer review system where pre-prints are published publically, with reviewers comments and then responses to these as well. However, for major decisions we could rely on reports written and invetigated by a number of people. Open Phil have done this to an extent, but having broader panels etc. to do these reports may be much more useful. Certainly its something to try.
I do think its really difficult though, but I think the current EA status quo is not working. Perhaps EA consulting with some thinkers on the social organisation of science to better design how we do these things may be good, as there certainly are people with this expertise.
And it is definitely possible to commission structured expert elicitations, and is possible to directly fund specific bits of research.
Moreover, another thing about peer review is that it can sometimes be pretty important for policy. This is certainly the case for the climate change space, where you won’t be incorporated into UN decision making and IPCC reports unless your peer reviewed.
Finally, I think your points about the ‘agreed upon methods’ sort of thing is really good, and this is something I’m trying to work on in XRisk. I talk about this a little in my ‘Beyond Simple Existential Risk’ talk, and am writing a paper with Anders Sandberg, SJ Beard and Adrian Currie on this at present. I’d be keen to hear your thoughts on this if your interested!
Rethink Priorities occasionally pays non-EA subject matter experts (usually academics that are formally recognized by other academics as the relevant and authoritative subject-matter experts, but not always) to review some of our work. I think this is a good way of creating a peer review process without having to publish formally in journals. Though Rethink Priorities also occasionally publishes formally in journals.
Maybe more orgs should try to do that? (I think Open Phil and GiveWell do this as well.)
Finally, I think your points about the ‘agreed upon methods’ sort of thing is really good,
Since you liked that though let me think out loud a bit more.
I think it’s practically impossible to be rigorous without a paradigm.
Old sciences have paradigms and mostly work well but the culture is not nice to people trying to form ideas outside the paradigm, because that is necessarily less rigours. I remember some academic complaining on this on a podcast. They where doing some different approach within cognitive science and had problem with pear review because they where not enough focused on measuring the standard things.
On the other had there is EA/LW style AI Safety research, where everyone talks abut how preparadigmatic we are. Vague speculative ideas, with out inferential depth, get more appreciation and attention. By now there are a few paradigms, the clearest case being Vanessas research, which almost no one understand. I think part of the reason her work is hard to undertand is exactly because it is rigours within a paradigm research. It’s specific proof with in a specific framework. It has both more details and more prerequisites. While reading pre paradigmatic blogposts is like reading the first intro chapter in a text book (which is always less technical), the with in paradigmatic stuff is more like reading chapter 11, and you really have to have read the previous chapters, which makes it less accessible. Especially since no one collected the previous chapters for you, and the person writing it is not selected for their pedagogical skills.
Research has to start as pre paradigmatic. But I think that the dynamic described above makes it hard to move on, to pick some paradigm to explore and start working out the details. Maybe a field at some point needs to develop a culture of looking down at less rigours work, for any rigours work to really take hold? I’m really not sure. And I don’t want to loose the explorative part of EA/LW style AI Safety research either. Possibly rigour will just develop naturally over time?
I think this is pretty interesting and thanks for sharing your thoughys! There’s things here I agree with, things I disagree with, and I might say more when I’m on my computer not phone!. However, I’d love to call about this to talk more, and see
FWIW I don’t think it’s usually very hard to find people with the right expertise. You’d just need to look for author names on a peer reviewed paper / look at who they cite / look at a university’s website / email a prof and ask them to refer you to an expert.
Thanks for posting this. I have lot’s of thoughts about lots of things, that will take longer to think about. So I start with one of the easier questions.
Regarding pear review, you suggest
Have you had any interaction with the academic pear review system? Have you seen some of the stuff that passes though pear review? I’m in favour of scientific rigour, but I don’t think pear review solves that. In reality, my impression is that academia relies as much as name recognition and informal consensus mechanisms, as we (the blogpost community) does. The only reason academia has higher standards (in some fields) is that these fields are older and have developed a consensus around what methods are good enough vs not good enough.
I think pear review have the potential to be good. My impression is that it does really work in math, and that this has a lot to do with that reviewers receive recognition. But in many other fields it mainly serves to slow down research publication, and to gatekeep papers that are not written in the right format, or are not sufficiently interesting.
I did a PhD in theoretical physics and I was not impressed by the pear review responses I got on my paper. It was almost always very shallow comments. Which is not suppressing given that at least in that corner of physics pear review was unpaid and unrecognised work.
Can one just do that? Isn’t it very hard to find a person who has the right expertise and who you can verify have the right expertise?
If you already did the work to make a high quality paper, then peer review probably won’t add much. But the point is actually to prevent poor quality, incorrect research from getting through, and to raise the quality of publications as a whole.
My PhD was on computational physics, and yeah, the peer review didn’t add much to my papers. But because I knew it was there, I made sure to put a ton of work to make sure it was error free and every part of it was high quality. If I knew I could get the same reward of publication by being sloppy or lazy, I might be tended to do that. I certainly put orders of magnitude more effort into my papers than I do to my blog posts.
I certainly don’t think peer review is perfect, or that every post should be peer reviewed or anything. But I think that research that has to pass that bar tends to be superior to work that doesn’t.
This comment is pretty long, but TLDR: peer review and academia have their own problems, some similar to EA, some not. Maybe a hybrid approach works, and maybe we should consult with people with expertise in social organisation of science.
To some extent I agree with this. Whilst I’ve been wanting more academic rigour in X-Risk for a while, peer review certainly is no perfect panacea, although I think it is probably better than the current culture of deferring to blog posts as much as we do.
I think you are right that traditional academia really has its problems, and name recognition is also still an issue (eg Nobel Prize winners are 70% more likely to get through peer review etc.). Nonetheless, certainly from the field (solar geoengineering) that I have been in, name recognition and agreement with the ‘thought leaders’ is definitely less incentivised than in EA.
One potential response is to think a balance between peer review, the current EA culture and commissioned reports is a good balance. We could set up an X-Risk journal with some editors and reviewers who are a) dedicated to pluralism and b) will publish things that are methodologically sound irrespective of result. Alternatively, a sort of open peer review system where pre-prints are published publically, with reviewers comments and then responses to these as well. However, for major decisions we could rely on reports written and invetigated by a number of people. Open Phil have done this to an extent, but having broader panels etc. to do these reports may be much more useful. Certainly its something to try.
I do think its really difficult though, but I think the current EA status quo is not working. Perhaps EA consulting with some thinkers on the social organisation of science to better design how we do these things may be good, as there certainly are people with this expertise.
And it is definitely possible to commission structured expert elicitations, and is possible to directly fund specific bits of research.
Moreover, another thing about peer review is that it can sometimes be pretty important for policy. This is certainly the case for the climate change space, where you won’t be incorporated into UN decision making and IPCC reports unless your peer reviewed.
Finally, I think your points about the ‘agreed upon methods’ sort of thing is really good, and this is something I’m trying to work on in XRisk. I talk about this a little in my ‘Beyond Simple Existential Risk’ talk, and am writing a paper with Anders Sandberg, SJ Beard and Adrian Currie on this at present. I’d be keen to hear your thoughts on this if your interested!
Rethink Priorities occasionally pays non-EA subject matter experts (usually academics that are formally recognized by other academics as the relevant and authoritative subject-matter experts, but not always) to review some of our work. I think this is a good way of creating a peer review process without having to publish formally in journals. Though Rethink Priorities also occasionally publishes formally in journals.
Maybe more orgs should try to do that? (I think Open Phil and GiveWell do this as well.)
Since you liked that though let me think out loud a bit more.
I think it’s practically impossible to be rigorous without a paradigm.
Old sciences have paradigms and mostly work well but the culture is not nice to people trying to form ideas outside the paradigm, because that is necessarily less rigours. I remember some academic complaining on this on a podcast. They where doing some different approach within cognitive science and had problem with pear review because they where not enough focused on measuring the standard things.
On the other had there is EA/LW style AI Safety research, where everyone talks abut how preparadigmatic we are. Vague speculative ideas, with out inferential depth, get more appreciation and attention. By now there are a few paradigms, the clearest case being Vanessas research, which almost no one understand. I think part of the reason her work is hard to undertand is exactly because it is rigours within a paradigm research. It’s specific proof with in a specific framework. It has both more details and more prerequisites. While reading pre paradigmatic blogposts is like reading the first intro chapter in a text book (which is always less technical), the with in paradigmatic stuff is more like reading chapter 11, and you really have to have read the previous chapters, which makes it less accessible. Especially since no one collected the previous chapters for you, and the person writing it is not selected for their pedagogical skills.
Research has to start as pre paradigmatic. But I think that the dynamic described above makes it hard to move on, to pick some paradigm to explore and start working out the details. Maybe a field at some point needs to develop a culture of looking down at less rigours work, for any rigours work to really take hold? I’m really not sure. And I don’t want to loose the explorative part of EA/LW style AI Safety research either. Possibly rigour will just develop naturally over time?
End of speculation
I think this is pretty interesting and thanks for sharing your thoughys! There’s things here I agree with, things I disagree with, and I might say more when I’m on my computer not phone!. However, I’d love to call about this to talk more, and see
Is there a recording?
I’m always happy to offer my opinions.
here’s my email: linda.linsefors@gmail.com
There is, it should be on the cea youtube channel at some point. It is also a forum post:https://forum.effectivealtruism.org/posts/cXH2sG3taM5hKbiva/beyond-simple-existential-risk-survival-in-a-complex#:~:text=It sees the future as,perhaps at least as important.
FWIW I don’t think it’s usually very hard to find people with the right expertise. You’d just need to look for author names on a peer reviewed paper / look at who they cite / look at a university’s website / email a prof and ask them to refer you to an expert.