I feel like I detect a missing mood from you where you’re skeptical of pausing (for plausible-to-me reasons), but you’re not conflicted about it like I am and you don’t e.g. look for ways to buy time or ways for regulation to help without the downsides of a pause. (Sorry if this sounds adversarial.) Relatedly, this post is one-sided and so feels soldier-mindset-y. Likely this is just due to the focus on debating AI pause. But I would feel reassured if you said you’re sympathetic to: labs not publishing capabilities research, labs not publishing model weights, dangerous-capability-model-eval-based regulation, US and allies slowing other states and denying them compute, and/or other ways to slow AI or for regulation to help. If you’re unsympathetic to these, I would doubt that the overhang nuances you discuss are your true rejection (but I’d be interested in hearing more about your take on slowing and regulation outside of “pause”).
Edit: man, I wrote this after writing four object-level comments but this got voted to the top. Please note that I’m mostly engaging on the object level and I think object-level discussion is generally more productive—and I think Nora’s post makes several good points.
I think people have started to stretch the “missing mood” concept a bit too far for my taste.
What actual mood is missing here?
If you think that the default path of AI development leads towards eventual x-risk safety, but that rash actions like an AI pause could plausibly push us off that path and into catastrophe, then your default moods would be “fervent desire to dissuade people from doing the potentially disastrous thing”, and “happy that the disastrous thing probably won’t happen”. I think this matches with the mood the OP has provided.
I worry that these sort of meta-critiques can inadvertently be used to pressure people into one side of object-level disagreements. This isn’t a dig at you in particular, and I acknowledge that you made object level points as well, which really should be higher than this comment.
Noticing the irony that this very natural AI safety idea is (in Nora’s view) actually counterproductive and so constructively searching for ways to modify it and for adjacent ideas that don’t have its downsides
Sympathy with the pro-pause position and its proponents
Also feeling more conflicted in general—there are several real considerations in favor of pausing and Nora doesn’t grapple with them. (But this is a debate and maybe Nora is deliberately one-sidedly arguing for a particular position.)
Maybe “missing mood” isn’t exactly the right concept.
So the point of the “missing mood” concept was that it was an indicator for motivated reasoning. If someone reports to you that “lithuanians are genetically bad at chess” with a mood of unrestrained glee, you can rightly get suspicious of their methods. If they weren’t already prejudiced against lithuanians, they would find the result about chess ability sad and unfortunate.
I see no similar indicators here. From nora’s perspective, the AI pause and similar proposals are a bomb that will hurl us much closer to catastrophe. Why, (from their perspective) would there be a requirement to show sympathy for the bomb-throwers, or propose a modified bomb design?
Now of course, as a human being nora will have pre-existing biases towards one side or the other, and you can pick apart the piece if you want to find evidence of that (like using the phrase “heavy handed government regulation”). But having some bias towards one side doesn’t mean your arguments are wrong. The meta can have some uses if it’s truly blatant, but it’s the object level that actually matters.
If you desperately wish we had more time to work on alignment, but also think a pause won’t make that happen or would have larger countervailing costs, then that would lead to an attitude like: “If only we had more time! But alas, a pause would only make things worse. Let’s talk about other ideas…” For my part, I definitely say things like that (see here).
However, Nora has sections claiming “alignment is doing pretty well” and “alignment optimism”, so I think it’s self-consistent for her to not express that kind of mood.
Insofar as Nora discusses nuances of overhang, it would be odd and annoying for that to not actually be cruxy for her (given that she doesn’t say something like this isn’t cruxy for me).
I was reading it as a kinda disjunctive argument. If Nora says that a pause is bad because of A and B, either of which is sufficient on its own from her perspective, then you could say “A isn’t cruxy for her” (because B is sufficient) or you could say “B isn’t cruxy for her” (because A is sufficient). Really, neither of those claims is accurate.
Oh well, whatever, I agree with you that the OP could have been clearer.
Yep it’s all meant to be disjunctive and yep it could have been clearer. FWIW this essay went through multiple major revisions and at one point I was trying to make the disjunctivity of it super clear but then that got de-prioritized relative to other stuff. In the future if/when I write about this I think I’ll be able to organize things significantly better
Hmm, AI safety is probably easy implies that slowing AI is lower-stakes but doesn’t obviously imply much about whether it’s net-positive. It’s not obvious to me what alignment optimism has to do with the pause debate, and I don’t think you discuss this.
It’s not obvious to me what alignment optimism has to do with the pause debate
Sorry, I thought it would be fairly obvious how it’s related. If you’re optimistic about alignment then the expected benefits you might hope to get out of a pause (whether or not you actually do get those benefits) are commensurately smaller, so the unintended consequences should have more relative weight in your EV calculation.
To be clear, I think slowing down AI in general, as opposed to the moratorium proposal in particular, is a more reasonable position that’s a bit harder to argue against. I do still think the overhang concerns apply in non-pause slowdowns but in a less acute manner.
Given alignment optimism, the benefits of pause are smaller—but the unintended consequences for alignment are smaller too. I guess alignment optimism suggests pause-is-bad if e.g. your alignment optimism is super conditional on smooth progress...
“dangerous-capability-model-eval-based regulation” sounds good to me. I’m also in favor of Robin Hanson’s foom liability proposal. These seem like very targeted measures that would plausibly reduce the tail risk of existential catastrophe, and don’t have many negative side effects. I’m also not opposed to the US trying to slow down other states, although it’d depend on the specifics of the proposal.
Where we (partially) disagree:
I think there’s a plausible case to be made that publishing model weights reduces foom risk by making AI capabilities more broadly distributed, and also enhances security-by-transparency. Of course there are concerns about misuse— I do think that’s a real thing to be worried about— but I also think it’s generally exaggerated. I also relatively strongly favor open source on purely normative grounds. So my inclination is to be in favor of it but with reservations. Same goes for labs publishing capabilities research.
I feel like I detect a missing mood from you where you’re skeptical of pausing (for plausible-to-me reasons), but you’re not conflicted about it like I am and you don’t e.g. look for ways to buy time or ways for regulation to help without the downsides of a pause. (Sorry if this sounds adversarial.) Relatedly, this post is one-sided and so feels soldier-mindset-y. Likely this is just due to the focus on debating AI pause. But I would feel reassured if you said you’re sympathetic to: labs not publishing capabilities research, labs not publishing model weights, dangerous-capability-model-eval-based regulation, US and allies slowing other states and denying them compute, and/or other ways to slow AI or for regulation to help. If you’re unsympathetic to these, I would doubt that the overhang nuances you discuss are your true rejection (but I’d be interested in hearing more about your take on slowing and regulation outside of “pause”).
Edit: man, I wrote this after writing four object-level comments but this got voted to the top. Please note that I’m mostly engaging on the object level and I think object-level discussion is generally more productive—and I think Nora’s post makes several good points.
I think people have started to stretch the “missing mood” concept a bit too far for my taste.
What actual mood is missing here?
If you think that the default path of AI development leads towards eventual x-risk safety, but that rash actions like an AI pause could plausibly push us off that path and into catastrophe, then your default moods would be “fervent desire to dissuade people from doing the potentially disastrous thing”, and “happy that the disastrous thing probably won’t happen”. I think this matches with the mood the OP has provided.
I worry that these sort of meta-critiques can inadvertently be used to pressure people into one side of object-level disagreements. This isn’t a dig at you in particular, and I acknowledge that you made object level points as well, which really should be higher than this comment.
Noticing the irony that this very natural AI safety idea is (in Nora’s view) actually counterproductive and so constructively searching for ways to modify it and for adjacent ideas that don’t have its downsides
Sympathy with the pro-pause position and its proponents
Also feeling more conflicted in general—there are several real considerations in favor of pausing and Nora doesn’t grapple with them. (But this is a debate and maybe Nora is deliberately one-sidedly arguing for a particular position.)
Maybe “missing mood” isn’t exactly the right concept.
So the point of the “missing mood” concept was that it was an indicator for motivated reasoning. If someone reports to you that “lithuanians are genetically bad at chess” with a mood of unrestrained glee, you can rightly get suspicious of their methods. If they weren’t already prejudiced against lithuanians, they would find the result about chess ability sad and unfortunate.
I see no similar indicators here. From nora’s perspective, the AI pause and similar proposals are a bomb that will hurl us much closer to catastrophe. Why, (from their perspective) would there be a requirement to show sympathy for the bomb-throwers, or propose a modified bomb design?
Now of course, as a human being nora will have pre-existing biases towards one side or the other, and you can pick apart the piece if you want to find evidence of that (like using the phrase “heavy handed government regulation”). But having some bias towards one side doesn’t mean your arguments are wrong. The meta can have some uses if it’s truly blatant, but it’s the object level that actually matters.
If you desperately wish we had more time to work on alignment, but also think a pause won’t make that happen or would have larger countervailing costs, then that would lead to an attitude like: “If only we had more time! But alas, a pause would only make things worse. Let’s talk about other ideas…” For my part, I definitely say things like that (see here).
However, Nora has sections claiming “alignment is doing pretty well” and “alignment optimism”, so I think it’s self-consistent for her to not express that kind of mood.
Insofar as Nora discusses nuances of overhang, it would be odd and annoying for that to not actually be cruxy for her (given that she doesn’t say something like this isn’t cruxy for me).
I was reading it as a kinda disjunctive argument. If Nora says that a pause is bad because of A and B, either of which is sufficient on its own from her perspective, then you could say “A isn’t cruxy for her” (because B is sufficient) or you could say “B isn’t cruxy for her” (because A is sufficient). Really, neither of those claims is accurate.
Oh well, whatever, I agree with you that the OP could have been clearer.
Yep it’s all meant to be disjunctive and yep it could have been clearer. FWIW this essay went through multiple major revisions and at one point I was trying to make the disjunctivity of it super clear but then that got de-prioritized relative to other stuff. In the future if/when I write about this I think I’ll be able to organize things significantly better
Hmm, AI safety is probably easy implies that slowing AI is lower-stakes but doesn’t obviously imply much about whether it’s net-positive. It’s not obvious to me what alignment optimism has to do with the pause debate, and I don’t think you discuss this.
Sorry, I thought it would be fairly obvious how it’s related. If you’re optimistic about alignment then the expected benefits you might hope to get out of a pause (whether or not you actually do get those benefits) are commensurately smaller, so the unintended consequences should have more relative weight in your EV calculation.
To be clear, I think slowing down AI in general, as opposed to the moratorium proposal in particular, is a more reasonable position that’s a bit harder to argue against. I do still think the overhang concerns apply in non-pause slowdowns but in a less acute manner.
Given alignment optimism, the benefits of pause are smaller—but the unintended consequences for alignment are smaller too. I guess alignment optimism suggests pause-is-bad if e.g. your alignment optimism is super conditional on smooth progress...
Could you say more about what you see as the practical distinction between a “slow down AI in general” proposal vs. a “pause” proposal?
Where we agree:
“dangerous-capability-model-eval-based regulation” sounds good to me. I’m also in favor of Robin Hanson’s foom liability proposal. These seem like very targeted measures that would plausibly reduce the tail risk of existential catastrophe, and don’t have many negative side effects. I’m also not opposed to the US trying to slow down other states, although it’d depend on the specifics of the proposal.
Where we (partially) disagree:
I think there’s a plausible case to be made that publishing model weights reduces foom risk by making AI capabilities more broadly distributed, and also enhances security-by-transparency. Of course there are concerns about misuse— I do think that’s a real thing to be worried about— but I also think it’s generally exaggerated. I also relatively strongly favor open source on purely normative grounds. So my inclination is to be in favor of it but with reservations. Same goes for labs publishing capabilities research.