Yudkowsky’s suggestions seem entirely appropriate if you truly believe, like him, that AI x-risk is probability ~100%.
However, that belief is absurdly high, based on unproven and unlikely assumptions, like that an AI could build nanofactories by ordering proteins to be mixed over email.
In the actual world, where the probability of extinction is signficantly less than 100%, are these proposals valuable? It seems like they will just get everyone else labelled luddites and fearmongerers, especially if years and decades go by with no apocalypse in sight.
Yudkowsky’s suggestions seem entirely appropriate if you truly believe, like him, that AI x-risk is probability ~100%.
These proposals would plausibly be correct (to within an order of magnitude) in terms of the appropriate degree of response with much lower probabilities of doom (i.e. 10-20%). I think you need to actually run the math to say that this doesn’t make sense.
unproven and unlikely assumptions, like that an AI could build nanofactories by ordering proteins to be mixed over email
This is a deeply distorted understanding of Eliezer’s threat model, which is not any specific story that he can tell, but the brute fact that something smarter than you (and him, and everyone else) will come up with something better than that.
In the actual world, where the probability of extinction is signficantly less than 100%, are these proposals valuable?
I do not think it is ever particularly useful to ask “is someone else’s conclusion valid given my premises, which are importantly different from theirs”, if you are attempting to argue against someone’s premises. Obviously “A ⇒ B” & “C” does not imply “B”, and it especially does not imply “~A”.
It seems like they will just get everyone else labelled luddites and fearmongerers, especially if years and decades go by with no apocalypse in sight.
This is an empirical claim about PR, which:
does not seem obviously correct to me
has little to say about the object-level arguments
falls into pattern of suggesting that people should optimize for how others perceive us, rather than optimizing for communicating our true beliefs about the world.
These proposals would plausibly be correct (to within an order of magnitude) in terms of the appropriate degree of response with much lower probabilities of doom (i.e. 10-20%)
I strongly disagree with this. This article includes the following passage:
Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.
This advocates for risking nuclear war for the sake of preventing mere “AI training runs”. I find it highly unlikely that this risk-reward payoff is logical at a 10% x-risk estimate.
You do make a fair point about the structure of my comment, though. In truth I don’t have a problem with yudkowsky writing this article, given his incorrect beliefs about AI-risk. I have a problem with the proposals themselves, because his beliefs are incorrect.
I really don’t like the rhetorical move you’re making here. You (as well as many people on this forum) think his beliefs are incorrect; others on this forum think they are correct. Insofar as there’s no real consensus for which side is correct, I’d strongly prefer people (on both sides) use language like “given his, in my opinion, incorrect beliefs” as opposed to just stating as a matter of fact that he’s incorrect.
Are you expecting me to preface every statement I make with “in my opinion”? Obviously this is my opinion and people are free to disagree. I don’t need to state that explicitly every single time.
Sorry, after rereading my comment, it comes off as more hostile than I was intending (currently sleep-deprived, which sometimes has that effect on me). The intended tone of my comment was more like “this move feels like it could lead to epistemics slipping or to newcomers being confused” and not like “this move violates some important norm of good behavior”.
Regarding your specific question – no, I’m obviously not expecting you to preface every statement with “in my opinion”. Most writing doesn’t include “in my opinion” at the beginning of every statement, yet also most writing doesn’t lead to a flag going off in my head for “huh, this statement is stated as a fact but is actually a matter up for debate” which I did notice here.
This advocates for risking nuclear war for the sake of preventing mere “AI training runs”. I find it highly unlikely that this risk-reward payoff is logical at a 10% x-risk estimate.
All else equal, this depends on what increase in risk of nuclear war you’re trading off against what decrease in x-risk from AI. We may have “increased” risk of nuclear war by providing aid to Ukraine in its war against Russia, but if it was indeed an increase it was probably small and worth the trade-off[1] against our other goals (such as disincentivizing the beginning of wars which might lead to nuclear escalation in the first place). I think approximately the only unusual part of Eliezer’s argument is the fact that he doesn’t beat around the bush in spelling out the implications.
Asserted for the sake of argument; I haven’t actually demonstrated that this is true but my point is more that there are many situations where we behave as if it is obviously a worthwhile trade-off to marginally increase the risk of nuclear war.
He’s not talking about a “marginal increase” in risk of nuclear war. What Eliezer is proposing is nuclear blackmail.
If China, today, told us that “you have 3 months to disband OpenAI or we will nuke you”, what are the chances that the US would comply? I guarantee you they are almost zero, because if the US gives in, then china can demand something else, and then something else, and so on. Instead, the US would probably try to talk them out of their ultimatum, or failing that, do a preemptive strike.
If the deadline does come, China can either launch the nukes and start armageddon, or reveal an empty threat and not be taken seriously, in which case the whole exercise is worthless.
He proposes instituting an international treaty, which seems to be aiming for the reference class of existing treaties around the proliferation of nuclear and biological weapons. He is not proposing that the United States issue unilateral threats of nuclear first strikes.
I do not believe this interpretation is correct. Here is the passage again, including the previous paragraph for added context:
Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for anyone, including governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool. That we all live or die as one, in this, is not a policy but a fact of nature. Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.
He advocates for bombing datacentres and being prepared to start shooting conflicts to destroy GPU clusters, and then advocates for “running some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs”. I cannot see any interpretation other than “threaten to bomb nuclear armed countries that train AI’s”.
To be fair, upon reading it again it’s more likely he means “threaten to conventionally bomb datacentres”. But this is still nuclear brinksmanship: bombing russia or china is an act of war, carrying a high chance of nuclear exchange.
I don’t think the crux here is about nanofactories – I’d imagine that if Eliezer considered a world identical to ours but where nanofactories were impossible, he’d place (almost) as high probability on doom (though he’d presumably expect doom to be somewhat more drawn out).
Yudkowsky’s suggestions seem entirely appropriate if you truly believe, like him, that AI x-risk is probability ~100%.
However, that belief is absurdly high, based on unproven and unlikely assumptions, like that an AI could build nanofactories by ordering proteins to be mixed over email.
In the actual world, where the probability of extinction is signficantly less than 100%, are these proposals valuable? It seems like they will just get everyone else labelled luddites and fearmongerers, especially if years and decades go by with no apocalypse in sight.
Many things about this comment seem wrong to me.
These proposals would plausibly be correct (to within an order of magnitude) in terms of the appropriate degree of response with much lower probabilities of doom (i.e. 10-20%). I think you need to actually run the math to say that this doesn’t make sense.
This is a deeply distorted understanding of Eliezer’s threat model, which is not any specific story that he can tell, but the brute fact that something smarter than you (and him, and everyone else) will come up with something better than that.
I do not think it is ever particularly useful to ask “is someone else’s conclusion valid given my premises, which are importantly different from theirs”, if you are attempting to argue against someone’s premises. Obviously “A ⇒ B” & “C” does not imply “B”, and it especially does not imply “~A”.
This is an empirical claim about PR, which:
does not seem obviously correct to me
has little to say about the object-level arguments
falls into pattern of suggesting that people should optimize for how others perceive us, rather than optimizing for communicating our true beliefs about the world.
I strongly disagree with this. This article includes the following passage:
This advocates for risking nuclear war for the sake of preventing mere “AI training runs”. I find it highly unlikely that this risk-reward payoff is logical at a 10% x-risk estimate.
You do make a fair point about the structure of my comment, though. In truth I don’t have a problem with yudkowsky writing this article, given his incorrect beliefs about AI-risk. I have a problem with the proposals themselves, because his beliefs are incorrect.
I really don’t like the rhetorical move you’re making here. You (as well as many people on this forum) think his beliefs are incorrect; others on this forum think they are correct. Insofar as there’s no real consensus for which side is correct, I’d strongly prefer people (on both sides) use language like “given his, in my opinion, incorrect beliefs” as opposed to just stating as a matter of fact that he’s incorrect.
Are you expecting me to preface every statement I make with “in my opinion”? Obviously this is my opinion and people are free to disagree. I don’t need to state that explicitly every single time.
Sorry, after rereading my comment, it comes off as more hostile than I was intending (currently sleep-deprived, which sometimes has that effect on me). The intended tone of my comment was more like “this move feels like it could lead to epistemics slipping or to newcomers being confused” and not like “this move violates some important norm of good behavior”.
Regarding your specific question – no, I’m obviously not expecting you to preface every statement with “in my opinion”. Most writing doesn’t include “in my opinion” at the beginning of every statement, yet also most writing doesn’t lead to a flag going off in my head for “huh, this statement is stated as a fact but is actually a matter up for debate” which I did notice here.
All else equal, this depends on what increase in risk of nuclear war you’re trading off against what decrease in x-risk from AI. We may have “increased” risk of nuclear war by providing aid to Ukraine in its war against Russia, but if it was indeed an increase it was probably small and worth the trade-off[1] against our other goals (such as disincentivizing the beginning of wars which might lead to nuclear escalation in the first place). I think approximately the only unusual part of Eliezer’s argument is the fact that he doesn’t beat around the bush in spelling out the implications.
Asserted for the sake of argument; I haven’t actually demonstrated that this is true but my point is more that there are many situations where we behave as if it is obviously a worthwhile trade-off to marginally increase the risk of nuclear war.
He’s not talking about a “marginal increase” in risk of nuclear war. What Eliezer is proposing is nuclear blackmail.
If China, today, told us that “you have 3 months to disband OpenAI or we will nuke you”, what are the chances that the US would comply? I guarantee you they are almost zero, because if the US gives in, then china can demand something else, and then something else, and so on. Instead, the US would probably try to talk them out of their ultimatum, or failing that, do a preemptive strike.
If the deadline does come, China can either launch the nukes and start armageddon, or reveal an empty threat and not be taken seriously, in which case the whole exercise is worthless.
He proposes instituting an international treaty, which seems to be aiming for the reference class of existing treaties around the proliferation of nuclear and biological weapons. He is not proposing that the United States issue unilateral threats of nuclear first strikes.
I do not believe this interpretation is correct. Here is the passage again, including the previous paragraph for added context:
He advocates for bombing datacentres and being prepared to start shooting conflicts to destroy GPU clusters, and then advocates for “running some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs”. I cannot see any interpretation other than “threaten to bomb nuclear armed countries that train AI’s”.
To be fair, upon reading it again it’s more likely he means “threaten to conventionally bomb datacentres”. But this is still nuclear brinksmanship: bombing russia or china is an act of war, carrying a high chance of nuclear exchange.
Your post begins with,
And ends with,
If in the writing of a comment you realize that you were wrong, you can just say that.
I don’t think the crux here is about nanofactories – I’d imagine that if Eliezer considered a world identical to ours but where nanofactories were impossible, he’d place (almost) as high probability on doom (though he’d presumably expect doom to be somewhat more drawn out).