Ironically, this comment will not be an object-level criticism, and is more a meta-rant.
As someone who believes that the existential risk from AI is significant, and more significant than other existential risks, I am becoming more annoyed that a lot of the arguments for taking AI xrisk less serious are not object-level arguments, but indirect arguments.
If you are worried that EA prioritizes AI xrisk too much, maybe you should provide clear arguments why the chance that advanced AI will kill all of humanity this century is extremely small (eg below 2%).
(or provide other arguments like “actually the risk is 10% but there is nothing you can do to improve it”).
The following are not object-level arguments:
“You are biased”, “You are a homogeneous group”, “You are taking this author of Harry Potter fanfiction too seriously, please listen to the the people we denote as experts™ instead”, “Don’t you think, that as a math/CS person, it aligns suspiciously well with your own self-interest to read a 100-page google doc on eliciting latent knowledge in a rundown hotel in northern england instead of working for google and virtuously donate 10% of your income to Malaria bednets?”
Maybe I am biased, but that does not mean I should completely dismiss my object-level beliefs such as my opinion on deceptive Mesaoptimization.
Argue that the Neural Networks that Google will build in 2039 do not contain any Mesaoptimization at all!
Relay the arguments by domain-experts from academic disciplines such as “Futures Studies”, and “Science & Technology Studies”, so that new EAs can decide by themselves whether they believe the common arguments about orthogonality thesis and instrumental convergence!
Argue that the big tech companies will solve corrigibility on their own, and don’t need any help from a homogeneous group of EA nerds!
To be fair, I have seen arguments that say something like “AGI is very unlikely to be build this century”.
But the fact that some of the readers will have doubts whether these very lines were produced by chatGPT or a human should give you doubt of the position that reaaching human level intelligence with trillions of parameters is impossible in the next 70 years.
I think the problem here is that you are requiring your critics to essentially stay within the EA framework of quantitative thinking and splitting risks, but that framework is exactly what is being criticized
I think something that’s important here is that indirect arguments can show that given other approaches, you may come to different conclusions; not just on prioritisation of ‘risks’(I hate using that word!), but also on techniques to reduce those as well.
For instance, I still think that AI and Biorisk are extremely significant contributors to risk, but probably would take on pretty different approaches to how we deal with this based on trying to consider these more indirect criticisms of the methodologies etc used
Exactly. For example, by looking at vulnerabilities in addition to hazards like AGI and engineered pandemics, we might find a vulnerability that is more pressing to work on than AI risk.
That said, the EA x-risk community has discussed vulnerabilities before: Bostrom’s paper “The Vulnerable World Hypothesis” proposes the semi-anarchic default condition as a societal vulnerability to a broad class of hazards.
To be clear, if you make arguments of the form “X is a more pressing problem then AI risk” or “here is a huge vulnerability X, we should try to fix that” then I would consider that an object-level argument, if you actually name X.
Thank you for extracting these things!
Ironically, this comment will not be an object-level criticism, and is more a meta-rant.
As someone who believes that the existential risk from AI is significant, and more significant than other existential risks, I am becoming more annoyed that a lot of the arguments for taking AI xrisk less serious are not object-level arguments, but indirect arguments.
If you are worried that EA prioritizes AI xrisk too much, maybe you should provide clear arguments why the chance that advanced AI will kill all of humanity this century is extremely small (eg below 2%). (or provide other arguments like “actually the risk is 10% but there is nothing you can do to improve it”).
The following are not object-level arguments: “You are biased”, “You are a homogeneous group”, “You are taking this author of Harry Potter fanfiction too seriously, please listen to the the people we denote as experts™ instead”, “Don’t you think, that as a math/CS person, it aligns suspiciously well with your own self-interest to read a 100-page google doc on eliciting latent knowledge in a rundown hotel in northern england instead of working for google and virtuously donate 10% of your income to Malaria bednets?”
Maybe I am biased, but that does not mean I should completely dismiss my object-level beliefs such as my opinion on deceptive Mesaoptimization.
Argue that the Neural Networks that Google will build in 2039 do not contain any Mesaoptimization at all!
Relay the arguments by domain-experts from academic disciplines such as “Futures Studies”, and “Science & Technology Studies”, so that new EAs can decide by themselves whether they believe the common arguments about orthogonality thesis and instrumental convergence!
Argue that the big tech companies will solve corrigibility on their own, and don’t need any help from a homogeneous group of EA nerds!
To be fair, I have seen arguments that say something like “AGI is very unlikely to be build this century”. But the fact that some of the readers will have doubts whether these very lines were produced by chatGPT or a human should give you doubt of the position that reaaching human level intelligence with trillions of parameters is impossible in the next 70 years.
I think the problem here is that you are requiring your critics to essentially stay within the EA framework of quantitative thinking and splitting risks, but that framework is exactly what is being criticized
I think something that’s important here is that indirect arguments can show that given other approaches, you may come to different conclusions; not just on prioritisation of ‘risks’(I hate using that word!), but also on techniques to reduce those as well. For instance, I still think that AI and Biorisk are extremely significant contributors to risk, but probably would take on pretty different approaches to how we deal with this based on trying to consider these more indirect criticisms of the methodologies etc used
Exactly. For example, by looking at vulnerabilities in addition to hazards like AGI and engineered pandemics, we might find a vulnerability that is more pressing to work on than AI risk.
That said, the EA x-risk community has discussed vulnerabilities before: Bostrom’s paper “The Vulnerable World Hypothesis” proposes the semi-anarchic default condition as a societal vulnerability to a broad class of hazards.
To be clear, if you make arguments of the form “X is a more pressing problem then AI risk” or “here is a huge vulnerability X, we should try to fix that” then I would consider that an object-level argument, if you actually name X.