Arguably, it is effective altruists who are the unusual ones here. The standard EA theory employed to justify extreme levels of caution around AI is strong longtermism.
This suggests people’s expected x-risk levels are really small (‘extreme levels of caution’), which isn’t what people believe.
I think “if you believe the probability that a technology will make humanity go extinct with a probability of 1% or more, be very very cautious” would be endorsed by a large majority of the general population & intellectual ‘elite’. It’s not at all a fringe moral position.
I think “if you believe the probability that a technology will make humanity go extinct with a probability of 1% or more, be very very cautious” would be endorsed by a large majority of the general population & intellectual ‘elite’.
I’m not sure we disagree. A lot seems to depend on what is meant by “very very cautious”. If it means shutting down AI as a field, I’m pretty skeptical. If it means regulating AI, then I agree, but I also think Sam Altman advocates regulation too.
I agree the general population would probably endorse the statement “if a technology will make humanity go extinct with a probability of 1% or more, be very very cautious” if given to them in a survey of some kind, but I think this statement is vague, and somewhat misleading as a frame for how people would think about AI if they were given more facts about the situation.
Firstly, we’re not merely talking about any technology here; we’re talking about a technology that has the potential to both disempower humans, but also make their lives dramatically better. Almost every technology has risks as well as benefits. Probably the most common method people use when deciding whether to adopt a technology themselves is to check whether the risks outweigh the benefits. Just looking at the risks alone gives a misleading picture.
The relevant statistic is the risk to benefit ratio, and here it’s really not obvious that most people would endorse shutting down AI if they were aware of all the facts. Yes, the risks are high, but so are the benefits.
If elites were made aware of both the risks and the benefits from AI development, most of them seem likely to want to proceed cautiously, rather than not proceed at all, or pause AI for many years, as many EAs have suggested. To test this claim empirically, we can just look at what governments are already doing with regards to AI risk policy, after having been advised by experts; and as far as I can tell, all of the relevant governments are substantially interested in both innovation and safety regulation.
Secondly, there’s a persistent and often large gap between what people say through their words (e.g. when answering surveys) and what they actually want as measured by their behavior. For example, plenty of polling has indicated that a large fraction of people are very cautious regarding GMOs, but in practice most people are willing to eat GM foods happily without much concern. People are often largely thoughtless when answering many types of abstract questions posed to them, especially about topics they have little knowledge about. And this makes sense, because their responses typically have almost no impact on anything that might immediately or directly impact them. Bryan Caplan has discussed these issues in surveys and voting systems before.
This suggests people’s expected x-risk levels are really small (‘extreme levels of caution’), which isn’t what people believe.
I think “if you believe the probability that a technology will make humanity go extinct with a probability of 1% or more, be very very cautious” would be endorsed by a large majority of the general population & intellectual ‘elite’. It’s not at all a fringe moral position.
I’m not sure we disagree. A lot seems to depend on what is meant by “very very cautious”. If it means shutting down AI as a field, I’m pretty skeptical. If it means regulating AI, then I agree, but I also think Sam Altman advocates regulation too.
I agree the general population would probably endorse the statement “if a technology will make humanity go extinct with a probability of 1% or more, be very very cautious” if given to them in a survey of some kind, but I think this statement is vague, and somewhat misleading as a frame for how people would think about AI if they were given more facts about the situation.
Firstly, we’re not merely talking about any technology here; we’re talking about a technology that has the potential to both disempower humans, but also make their lives dramatically better. Almost every technology has risks as well as benefits. Probably the most common method people use when deciding whether to adopt a technology themselves is to check whether the risks outweigh the benefits. Just looking at the risks alone gives a misleading picture.
The relevant statistic is the risk to benefit ratio, and here it’s really not obvious that most people would endorse shutting down AI if they were aware of all the facts. Yes, the risks are high, but so are the benefits.
If elites were made aware of both the risks and the benefits from AI development, most of them seem likely to want to proceed cautiously, rather than not proceed at all, or pause AI for many years, as many EAs have suggested. To test this claim empirically, we can just look at what governments are already doing with regards to AI risk policy, after having been advised by experts; and as far as I can tell, all of the relevant governments are substantially interested in both innovation and safety regulation.
Secondly, there’s a persistent and often large gap between what people say through their words (e.g. when answering surveys) and what they actually want as measured by their behavior. For example, plenty of polling has indicated that a large fraction of people are very cautious regarding GMOs, but in practice most people are willing to eat GM foods happily without much concern. People are often largely thoughtless when answering many types of abstract questions posed to them, especially about topics they have little knowledge about. And this makes sense, because their responses typically have almost no impact on anything that might immediately or directly impact them. Bryan Caplan has discussed these issues in surveys and voting systems before.