because it’s a very complex state of arguments, the vast majority of the world misunderstands it pretty severely… They have very clearly missed some important points of argument.
This seems like an argument from your own authority. I’ve read a number of doomer arguments and personally found them unconvincing, but I’m not asking anyone to take my word for it. Of course you can always say ‘you’ve read the wrong arguments’, but in general, if your argument amounts to ‘you need to read this 10s-of-thousands-of-words-argument’ there’s no reason for an observer to believe that you understand it better than other intelligent individuals who’ve read it and reject it.
Therefore this:
If there’s a counterargument you find convincing, please point me to it!
… sounds like special pleading. You’re trying to simultaneously claim both that a) the arguments for doom are so so complicated that no-one’s anti-doom views have any weight unless they’ve absorbed a nebulous gestalt of pro-doomer literature and b) that the purported absence of a single gestalt-rebutting counterpoint justifies a doomer position.
And to be clear, I don’t think the epistemic burden should be equalised—I think it should be the other way around. Arguments for extinction by AI are necessarily built on a foundation of a priori and partially empirical premises, such that the dissolution of any collapses the whole argument. To give a few examples, such arguments require one to believe:
No causality of intelligence on goals, or too weak causality to outweigh other factors
AI to develop malevolent goals despite the process of developing it inherently involving incremental steps towards making it closer to doing what its developers want
AI to develop malevolent goals despite every developer working on it wanting it not to kill them
Instrumental convergence
Continued exponential progress
Without the (higher) exponential energy demands we’re currently seeing
An ability to adapt to rapidly changing circumstances that’s entirely absent from modern deep learning algorithms
That the ceiling of AI will be sufficiently higher than that of humans to manipulate us on an individual or societal level without anyone noticing and sounding the alarm (or so good at manipulating us that even with the alarm sounding we do its bidding)
That it gets to this level fast enough/stealthily enough that no-one shuts it down
And to specifically believe that AI extinction is the most important thing to work on requires further assumptions, like
nothing else (including AI) is likely to do serious civilisational damage before AI wipes us out
or that civilisational collapse is sufficiently easy to recover from that it’s basically irrelevant to long term expectation
it will be morally bad if AI replaces us
that the thesis of the OP is false, and that survival work is either the same as or higher EV than flourishing work
that there’s anything we can actually do to prevent our destruction, assuming all the above propositions are true
Personally I weakly believe most of these propositions, but even if I weakly believed all of them, that would still leave me with extremely low total concern for Yudkowskian scenarios.
Obviously there are weaker versions of the AI thesis like ‘AI could cause immense harm, perhaps by accident and perhaps by human intent, and so is an important problem to work on’ which it’s a lot more reasonable to believe.
But when you assert that ‘The more people think seriously about this question, the more pessimistic they are’, it sounds like you mean they become something like Yudkowskyesque doomers—and I think that’s basically false outside certain epistemic bubbles.
Inasmuch as it’s true that people who go into the field both tend to be the most pessimistic and don’t tend to exit the field in large numbers after becoming insufficiently pessimistic, I’ll bet you for any specific version of that claim you want to make, something extremely similar is true of biorisk, climate change, s-risks in general, and longterm animal welfare in particular. I’d bet at slightly longer odds the same is true of national defence, global health, pronatalism, antinatalism, nuclear disarmament, conservation, gerontology and many other high-stakes areas.
I think that fact is reflected in the correlation of p(doom) with time-on-task only on alignment specifically
I think most people who’ve put much thought into it agree that the highest probability of human extinction by the end of the century comes from misaligned AI. But that’s not sufficient to justify a strong p(doom) position, let alone a ‘most important cause’ position. I also think it comes from a largely unargued-for (and IMO clearly false) assumption that we’d lose virtually no longterm expected value from civilisational collapse.
This seems like an argument from your own authority. I’ve read a number of doomer arguments and personally found them unconvincing, but I’m not asking anyone to take my word for it. Of course you can always say ‘you’ve read the wrong arguments’, but in general, if your argument amounts to ‘you need to read this 10s-of-thousands-of-words-argument’ there’s no reason for an observer to believe that you understand it better than other intelligent individuals who’ve read it and reject it.
Therefore this:
… sounds like special pleading. You’re trying to simultaneously claim both that a) the arguments for doom are so so complicated that no-one’s anti-doom views have any weight unless they’ve absorbed a nebulous gestalt of pro-doomer literature and b) that the purported absence of a single gestalt-rebutting counterpoint justifies a doomer position.
And to be clear, I don’t think the epistemic burden should be equalised—I think it should be the other way around. Arguments for extinction by AI are necessarily built on a foundation of a priori and partially empirical premises, such that the dissolution of any collapses the whole argument. To give a few examples, such arguments require one to believe:
No causality of intelligence on goals, or too weak causality to outweigh other factors
AI to develop malevolent goals despite the process of developing it inherently involving incremental steps towards making it closer to doing what its developers want
AI to develop malevolent goals despite every developer working on it wanting it not to kill them
Instrumental convergence
Continued exponential progress
Without the (higher) exponential energy demands we’re currently seeing
An ability to adapt to rapidly changing circumstances that’s entirely absent from modern deep learning algorithms
That the ceiling of AI will be sufficiently higher than that of humans to manipulate us on an individual or societal level without anyone noticing and sounding the alarm (or so good at manipulating us that even with the alarm sounding we do its bidding)
That it gets to this level fast enough/stealthily enough that no-one shuts it down
And to specifically believe that AI extinction is the most important thing to work on requires further assumptions, like
nothing else (including AI) is likely to do serious civilisational damage before AI wipes us out
or that civilisational collapse is sufficiently easy to recover from that it’s basically irrelevant to long term expectation
it will be morally bad if AI replaces us
that the thesis of the OP is false, and that survival work is either the same as or higher EV than flourishing work
that there’s anything we can actually do to prevent our destruction, assuming all the above propositions are true
Personally I weakly believe most of these propositions, but even if I weakly believed all of them, that would still leave me with extremely low total concern for Yudkowskian scenarios.
Obviously there are weaker versions of the AI thesis like ‘AI could cause immense harm, perhaps by accident and perhaps by human intent, and so is an important problem to work on’ which it’s a lot more reasonable to believe.
But when you assert that ‘The more people think seriously about this question, the more pessimistic they are’, it sounds like you mean they become something like Yudkowskyesque doomers—and I think that’s basically false outside certain epistemic bubbles.
Inasmuch as it’s true that people who go into the field both tend to be the most pessimistic and don’t tend to exit the field in large numbers after becoming insufficiently pessimistic, I’ll bet you for any specific version of that claim you want to make, something extremely similar is true of biorisk, climate change, s-risks in general, and longterm animal welfare in particular. I’d bet at slightly longer odds the same is true of national defence, global health, pronatalism, antinatalism, nuclear disarmament, conservation, gerontology and many other high-stakes areas.
I think most people who’ve put much thought into it agree that the highest probability of human extinction by the end of the century comes from misaligned AI. But that’s not sufficient to justify a strong p(doom) position, let alone a ‘most important cause’ position. I also think it comes from a largely unargued-for (and IMO clearly false) assumption that we’d lose virtually no longterm expected value from civilisational collapse.