I don’t think my argument is even that anti-institutionalist. I have issues with how academic publishing works but I still think peer reviewed research is an extremely important and valuable source of information. I just think it has flaws and is much messier than discussions around the topic sometimes make it seem.
My point isn’t to say that we should throw out traditional academic insitutions, it is to say that I feel like the claim that the arguments for short timelines are “non-evidence-based” are critiquing the same messiness that also is present in peer reviewed research. If I read a study whose conclusions I disagree with, I think it would be wrong to say “field X has a replication crisis, therefore we can’t really consider this study to be evidence”. I feel like a similar thing is going on when people say the arguments for short timelines are “non-evidence-based”. To me things like METR’s work definitely are evidence, even if they aren’t necessarily strong or definitive evidence or if that evidence is open to contested interpretations. I don’t think something needs to be peer reviewed to count as “evidence”, is essentially the point I was trying to make.
Generally, the scientific community is not going around arguing that drastic measures should be taken based on singular novel studies. Mainly, what a single novel study will produce is a wave of new studies on the same subject, to ensure that the results are valid and that the assumptions used hold up to scrutiny. Hence why that low-temperature superconductor was so quickly debunked.
I do not see similar efforts in the AI safety community. The studies by METR are great first forays into difficult subjects, but then I see barely any scrutinity or follow-up by other researchers. And people accept much worse scholarship like AI2027 at face-value for seemingly no reason.
I have experience in both academia and EA now, and I believe that the scholarship and skeptical standards in EA are substantially worse.
I agree. EA has a cost-effectiveness problem that conflicts with its truth-seeking attempts. EA’s main driving force is cost-effectiveness, above all else—even above truth itself.
EA is highly incentivised to create and spread apocalyptic doom narratives. This is because apocalyptic doom narratives are good at recruiting people to EA’s “let’s work to decrease the probability of apocalyptic doom (because that has lots of expected value given future population projections)” cause area. And funding-wise, EA community funding (at least in the UK) is pretty much entirely about trying to make more people work in these areas.
EA is also populated by the kinds of people who respond to apocalyptic doom narratives, for the basic reason that if they didn’t they wouldn’t have ended up in EA. So stuff that promotes these narratives does well in EA’s attention economy.
EA just doesn’t have anywhere near as much £$€ to spend as academia does. It’s also very interested in doing stuff and willing to tolerate errors as long as the stuff gets done. Therefore, its academic standards are far lower.
I really don’t know how you’d fix this. I don’t think research into catastrophic risks should be conducted on a shoestring budget and by a pseudoreligion/citizen science community. I think it should be government funded and probably sit within the wider defense and security portfolio.
However I’ll give EA some grace for essentially being a citizen science community, for the same reason I don’t waste effort grumping about the statistical errors made by participants in the Big Garden Birdwatch.
Generally, the scientific community is not going around arguing that drastic measures should be taken based on singular novel studies. Mainly, what a single novel study will produce is a wave of new studies on the same subject, to ensure that the results are valid and that the assumptions used hold up to scrutiny. Hence why that low-temperature superconductor was so quickly debunked.
I agree that on average the scientific community does a great job of this, but I think the process is much much messier in practice than a general description of the process makes it seem. For example, you have the alzheimers research that got huge pick-up and massive funding by major scientific institutions where the original research included doctored images. You have power-posing getting viral attention in science-ajacent media. You have priming where Kahneman wrote in his book that even if it seems wild you have to believe in it largely for similar reasons to what is being suggested here I think, that multiple rigorous scientific studies demonstrate the phenomenon, and yet when the replication crisis came around priming looks a lot more shaky than it seemed when Kahneman wrote that.
None of this means that we should throw out the existing scientific community or declare that most published research is false (although ironically there is a peer reviewed publication with this title!). Instead, my argument is that we should understand that this process is often messy and complicated. Imperfect research still has value and in my view is still “evidence” even if it is imperfect.
The research and arguments around AI risk are not anywhere near as rigorous as a lot of scientific research (and I linked a comment above where I myself criticize AI risk advocates for overestimating the rigor of their arguments). At the same time, this doesn’t mean that these arguments do not contain any evidence or value. There is a huge amount of uncetainty about what will happen with AI. People worried about the risks from AI are trying to muddle through these issues, just like the scientific community has to muddle through figuring things out as well. I think it its completely valid to point of flaws in arguments, lack of rigor, or over confidence (as I have also done). But evidence or argument doesn’t have to appear in a journal or conference to count as “evidence”.
My view is that we have to live with the uncertainty and make decisions based on the information we have, while also trying to get better information. Doing nothing and going with the status quo is itself a decision that can have important consequences. We should use the best evidence we have to make the best decision given uncertainty, not just default to the status quo when we lack ideal, rigorous evidence.
I don’t think my argument is even that anti-institutionalist. I have issues with how academic publishing works but I still think peer reviewed research is an extremely important and valuable source of information. I just think it has flaws and is much messier than discussions around the topic sometimes make it seem.
My point isn’t to say that we should throw out traditional academic insitutions, it is to say that I feel like the claim that the arguments for short timelines are “non-evidence-based” are critiquing the same messiness that also is present in peer reviewed research. If I read a study whose conclusions I disagree with, I think it would be wrong to say “field X has a replication crisis, therefore we can’t really consider this study to be evidence”. I feel like a similar thing is going on when people say the arguments for short timelines are “non-evidence-based”. To me things like METR’s work definitely are evidence, even if they aren’t necessarily strong or definitive evidence or if that evidence is open to contested interpretations. I don’t think something needs to be peer reviewed to count as “evidence”, is essentially the point I was trying to make.
Generally, the scientific community is not going around arguing that drastic measures should be taken based on singular novel studies. Mainly, what a single novel study will produce is a wave of new studies on the same subject, to ensure that the results are valid and that the assumptions used hold up to scrutiny. Hence why that low-temperature superconductor was so quickly debunked.
I do not see similar efforts in the AI safety community. The studies by METR are great first forays into difficult subjects, but then I see barely any scrutinity or follow-up by other researchers. And people accept much worse scholarship like AI2027 at face-value for seemingly no reason.
I have experience in both academia and EA now, and I believe that the scholarship and skeptical standards in EA are substantially worse.
I agree. EA has a cost-effectiveness problem that conflicts with its truth-seeking attempts. EA’s main driving force is cost-effectiveness, above all else—even above truth itself.
EA is highly incentivised to create and spread apocalyptic doom narratives. This is because apocalyptic doom narratives are good at recruiting people to EA’s “let’s work to decrease the probability of apocalyptic doom (because that has lots of expected value given future population projections)” cause area. And funding-wise, EA community funding (at least in the UK) is pretty much entirely about trying to make more people work in these areas.
EA is also populated by the kinds of people who respond to apocalyptic doom narratives, for the basic reason that if they didn’t they wouldn’t have ended up in EA. So stuff that promotes these narratives does well in EA’s attention economy.
EA just doesn’t have anywhere near as much £$€ to spend as academia does. It’s also very interested in doing stuff and willing to tolerate errors as long as the stuff gets done. Therefore, its academic standards are far lower.
I really don’t know how you’d fix this. I don’t think research into catastrophic risks should be conducted on a shoestring budget and by a pseudoreligion/citizen science community. I think it should be government funded and probably sit within the wider defense and security portfolio.
However I’ll give EA some grace for essentially being a citizen science community, for the same reason I don’t waste effort grumping about the statistical errors made by participants in the Big Garden Birdwatch.
I agree that on average the scientific community does a great job of this, but I think the process is much much messier in practice than a general description of the process makes it seem. For example, you have the alzheimers research that got huge pick-up and massive funding by major scientific institutions where the original research included doctored images. You have power-posing getting viral attention in science-ajacent media. You have priming where Kahneman wrote in his book that even if it seems wild you have to believe in it largely for similar reasons to what is being suggested here I think, that multiple rigorous scientific studies demonstrate the phenomenon, and yet when the replication crisis came around priming looks a lot more shaky than it seemed when Kahneman wrote that.
None of this means that we should throw out the existing scientific community or declare that most published research is false (although ironically there is a peer reviewed publication with this title!). Instead, my argument is that we should understand that this process is often messy and complicated. Imperfect research still has value and in my view is still “evidence” even if it is imperfect.
The research and arguments around AI risk are not anywhere near as rigorous as a lot of scientific research (and I linked a comment above where I myself criticize AI risk advocates for overestimating the rigor of their arguments). At the same time, this doesn’t mean that these arguments do not contain any evidence or value. There is a huge amount of uncetainty about what will happen with AI. People worried about the risks from AI are trying to muddle through these issues, just like the scientific community has to muddle through figuring things out as well. I think it its completely valid to point of flaws in arguments, lack of rigor, or over confidence (as I have also done). But evidence or argument doesn’t have to appear in a journal or conference to count as “evidence”.
My view is that we have to live with the uncertainty and make decisions based on the information we have, while also trying to get better information. Doing nothing and going with the status quo is itself a decision that can have important consequences. We should use the best evidence we have to make the best decision given uncertainty, not just default to the status quo when we lack ideal, rigorous evidence.