The term ‘default’ in discussions about AI risk (like ‘doom is the default’) strikes me as an unhelpful rhetorical move. It suggests an unlikely scenario where little-to-no measures are taken to address AI safety. Given the active research and the fact that alignment is likely to be crucial to unlocking the economic value from AI, this seems like a very unnatural baseline to frame discussions around.
It seems like you’re not arguing about the rhetoric of the people you disagree with but rather on the substantive question of the likelihood of disastrous AGI.
The reasons you have given tend to disconfirm the claim that “doom is the default.” But rhetorically, their expression succinctly and well conveys their belief that AGI will be very bad for us unless a very difficult and costly solution is developed.
One issue I have with this is that when someone calls this the ‘default’, I interpret them as implicitly making some prediction about the likelihood of such countermeasures not being taken. The issue then that this is a very vague way to communicate one’s beliefs. How likely does some outcome need to be for it to become the default? 90%? 70%? 50%? Something else?
The second concern is that it’s improbable for minimal or no safety measures to be implemented, making it odd to set this as a key baseline scenario. This belief is supported by substantial evidence indicating that safety precautions are likely to be taken. For instance:
Most of the major AGI labs are investing quite substantially in safety (e.g. OpenAI committing some substantial fraction of its compute budget, a large fraction of Anthropic’s research staff seems dedicated to safety, etc.)
We have received quite a substantial amount of concrete empirical evidence that safety-enhancing innovations are important for unlocking the economic value from AI systems (e.g. RLHF, constitutional AI, etc.)
It seems a priori very likely that alignment is important for unlocking the economic value from AI, because this effectively increases the range of tasks that AI systems can do without substantial human oversight, which is necessary for deriving value from automation
Major governments are interested in AI safety (e.g. the UK’s AI Safety Summit, the White House’s securing commitments around AI safety from AGI labs)
Maybe they think that safety measures taken in a world in which we observe this type of evidence will fall far short from what is neeeded. However, it’s somewhat puzzling be confident enough in this to label it as the ‘default’ scenario at this point.
I take it as a kind of “what do known incentives do and neglect to do” ---- when I say “default” I mean “without philanthropic pressure” or “well-aligned with making someone rich”. Of course, a lot of this depends on my background understanding of public-private partnerships through the history of innovation (something I’m liable to be wrong about).
It’s certainly true that many parts of almost every characterization/definition of “alignment” can simply be offloaded to capitalism, but I think there are a bajillion reasonable and defensible views about which parts those are, if they’re hard, they may be discovered in an inconvenient order, etc.
Well, sure, but if there a way to avoid the doom, then why after 20 years has no one published the plan for how to do it that doesn’t resemble a speculative research project of the type you try when you clearly don’t understand the problem and that doesn’t resemble the vague output of a politician writing about a sensitive issue?
There’s no need for anyone to offer a precise plan for how we can avoid doom. That would be nice, but unnecessary. I interpreted Tamay as saying that the “default” outcomes that people contemplate are improbable to begin with. If the so-called default scenario will not happen, then it’s unclear to me why we need to have a specific plan for how to avoid doom. Humanity will just avoid doom in the way we always have: trudging through until we get out alive. No one had a realistic, detailed plan for how we’d survive the 20th century unscathed by nuclear catastrophe, and yet we did anyway.
I interpreted Tamay as saying that the “default” outcomes that people contemplate are improbable to begin with.
I’m curious about your “to begin with”: do you interpret Tamay as saying that doom is improbable even if little-to-no measures are taken to address AI safety?
While I agree that slogans like “doom is the default” should not take over the discussion in favour of actual engagement, it doesn’t appear that your problem is with the specific phrasing but rather with the content behind this statement.
The term ‘default’ in discussions about AI risk (like ‘doom is the default’) strikes me as an unhelpful rhetorical move. It suggests an unlikely scenario where little-to-no measures are taken to address AI safety. Given the active research and the fact that alignment is likely to be crucial to unlocking the economic value from AI, this seems like a very unnatural baseline to frame discussions around.
It seems like you’re not arguing about the rhetoric of the people you disagree with but rather on the substantive question of the likelihood of disastrous AGI.
The reasons you have given tend to disconfirm the claim that “doom is the default.” But rhetorically, their expression succinctly and well conveys their belief that AGI will be very bad for us unless a very difficult and costly solution is developed.
One issue I have with this is that when someone calls this the ‘default’, I interpret them as implicitly making some prediction about the likelihood of such countermeasures not being taken. The issue then that this is a very vague way to communicate one’s beliefs. How likely does some outcome need to be for it to become the default? 90%? 70%? 50%? Something else?
The second concern is that it’s improbable for minimal or no safety measures to be implemented, making it odd to set this as a key baseline scenario. This belief is supported by substantial evidence indicating that safety precautions are likely to be taken. For instance:
Most of the major AGI labs are investing quite substantially in safety (e.g. OpenAI committing some substantial fraction of its compute budget, a large fraction of Anthropic’s research staff seems dedicated to safety, etc.)
We have received quite a substantial amount of concrete empirical evidence that safety-enhancing innovations are important for unlocking the economic value from AI systems (e.g. RLHF, constitutional AI, etc.)
It seems a priori very likely that alignment is important for unlocking the economic value from AI, because this effectively increases the range of tasks that AI systems can do without substantial human oversight, which is necessary for deriving value from automation
Major governments are interested in AI safety (e.g. the UK’s AI Safety Summit, the White House’s securing commitments around AI safety from AGI labs)
Maybe they think that safety measures taken in a world in which we observe this type of evidence will fall far short from what is neeeded. However, it’s somewhat puzzling be confident enough in this to label it as the ‘default’ scenario at this point.
I take it as a kind of “what do known incentives do and neglect to do” ---- when I say “default” I mean “without philanthropic pressure” or “well-aligned with making someone rich”. Of course, a lot of this depends on my background understanding of public-private partnerships through the history of innovation (something I’m liable to be wrong about).
The standard venn diagram of focused research organizations https://fas.org/publication/focused-research-organizations-a-new-model-for-scientific-research/ gives a more detailed view along the same lines, less clumsy, but still the point is “there are blindspots that we don’t know how to incentivize”.
It’s certainly true that many parts of almost every characterization/definition of “alignment” can simply be offloaded to capitalism, but I think there are a bajillion reasonable and defensible views about which parts those are, if they’re hard, they may be discovered in an inconvenient order, etc.
Well, sure, but if there a way to avoid the doom, then why after 20 years has no one published the plan for how to do it that doesn’t resemble a speculative research project of the type you try when you clearly don’t understand the problem and that doesn’t resemble the vague output of a politician writing about a sensitive issue?
There’s no need for anyone to offer a precise plan for how we can avoid doom. That would be nice, but unnecessary. I interpreted Tamay as saying that the “default” outcomes that people contemplate are improbable to begin with. If the so-called default scenario will not happen, then it’s unclear to me why we need to have a specific plan for how to avoid doom. Humanity will just avoid doom in the way we always have: trudging through until we get out alive. No one had a realistic, detailed plan for how we’d survive the 20th century unscathed by nuclear catastrophe, and yet we did anyway.
I’m curious about your “to begin with”: do you interpret Tamay as saying that doom is improbable even if little-to-no measures are taken to address AI safety?
While I agree that slogans like “doom is the default” should not take over the discussion in favour of actual engagement, it doesn’t appear that your problem is with the specific phrasing but rather with the content behind this statement.