Re p(doom) being high, I don’t think you need to commit to the view that the most likely outcome of AGI is doom. Surveys of AI researchers put the risk from rogue AI at 5%. In the XPT survey professional forecasters put the risk of extinction from AI at 0.03% by 2050 and 0.38% by 2100, whereas domain experts put the chance at 1.1% and 3%, respectively. Though they had longer timelines than you seem to endorse here.
I think your argument goes through even if the risk is 1% conditional on AGI and that seems like an estimate unlikely to upset too many people, so I would just go with that
I still don’t understand where the 95% for non-doom is coming from. I think it’s useful to look at actual mechanisms for why people think this (and so far I’ve found them lacking). The qualifications of the “professional forecasters” in the XPT survey are in doubt (and again, it was pre-GPT-4).
The argument might go through even if the risk is 1%, but people sure aren’t acting like that. At least in EA, broadly speaking (where I imagine the average p(doom|AGI) is closer to 10%). Also, I’d rather just say what I actually believe, even if it sounds “alarmist”. At least I’ve tried to argue for it in some detail. The main reason I am prioritising this so much is because I think it’s the most likely reason I, and everyone I know and love, will die. Unless we stop it. Forget longtermism and EA: people need to understand that this is a threat to their own personal near-term survival.
I think you come across as over-confident, not alarmist, and I think it hurts how you come across quite a lot. (We’ve talked a bit about the object level before.) I’d agree with John’s suggested approach.
Relatedly, I also think that your arguments for “p(doom|AGI)” being high aren’t convincing to people that don’t share your intuitions, and it looks like you’re relying on those (imo weak) arguments, when actually you don’t need to
The issue is that both sides of the debate lack gears-level arguments. The ones you give in this post (like “all the doom flows through the tiniest crack in our defence”) are more like vague intuitions; equally, on the other side, there are vague intuitions like “AGIs will be helping us on a lot of tasks” and “collusion is hard” and “people will get more scared over time” and so on.
I’d say it’s more than a vague intuition. It follows from alignment/control/misuse/coordination not being (close to) solved and ASI being much more powerful than humanity. I think it should be possible to formalise it, even. “AGIs will be helping us on a lot of tasks”, “collusion is hard” and “people will get more scared over time” aren’t anywhere close to overcoming it imo.
It follows from alignment/control/misuse/coordination not being (close to) solved.
“AGIs will be helping us on a lot of tasks”, “collusion is hard” and “people will get more scared over time” aren’t anywhere close to overcoming it imo.
These are what I mean by the vague intuitions.
I think it should be possible to formalise it, even
Nobody has come anywhere near doing this satisfactorily. The most obvious explanation is that they can’t.
To be fair, I think I’m partly making wrong assumptions about what exactly you’re arguing for here.
On a slightly closer read, you don’t actually argue in this piece that it’s as high as 90% - I assumed that because I think you’ve argued for that previously, and I think that’s what “high” p(doom) normally means.
I feel like this is a case of death by epistemic modesty, especially when it isn’t clear how these low p(doom) estimates are arrived at in a technical sense (and a lot seems to me like a kind of “respectability heuristic” cascade). We didn’t do very well with Covid as a society in the UK (and many other countries), following this kind of thinking.
Re p(doom) being high, I don’t think you need to commit to the view that the most likely outcome of AGI is doom. Surveys of AI researchers put the risk from rogue AI at 5%. In the XPT survey professional forecasters put the risk of extinction from AI at 0.03% by 2050 and 0.38% by 2100, whereas domain experts put the chance at 1.1% and 3%, respectively. Though they had longer timelines than you seem to endorse here.
I think your argument goes through even if the risk is 1% conditional on AGI and that seems like an estimate unlikely to upset too many people, so I would just go with that
I still don’t understand where the 95% for non-doom is coming from. I think it’s useful to look at actual mechanisms for why people think this (and so far I’ve found them lacking). The qualifications of the “professional forecasters” in the XPT survey are in doubt (and again, it was pre-GPT-4).
The argument might go through even if the risk is 1%, but people sure aren’t acting like that. At least in EA, broadly speaking (where I imagine the average p(doom|AGI) is closer to 10%). Also, I’d rather just say what I actually believe, even if it sounds “alarmist”. At least I’ve tried to argue for it in some detail. The main reason I am prioritising this so much is because I think it’s the most likely reason I, and everyone I know and love, will die. Unless we stop it. Forget longtermism and EA: people need to understand that this is a threat to their own personal near-term survival.
I think you come across as over-confident, not alarmist, and I think it hurts how you come across quite a lot. (We’ve talked a bit about the object level before.) I’d agree with John’s suggested approach.
Relatedly, I also think that your arguments for “p(doom|AGI)” being high aren’t convincing to people that don’t share your intuitions, and it looks like you’re relying on those (imo weak) arguments, when actually you don’t need to
I’m crying out for convincing gears-level arguments against (even have $1000 bounty on it), please provide some.
The issue is that both sides of the debate lack gears-level arguments. The ones you give in this post (like “all the doom flows through the tiniest crack in our defence”) are more like vague intuitions; equally, on the other side, there are vague intuitions like “AGIs will be helping us on a lot of tasks” and “collusion is hard” and “people will get more scared over time” and so on.
I’d say it’s more than a vague intuition. It follows from alignment/control/misuse/coordination not being (close to) solved and ASI being much more powerful than humanity. I think it should be possible to formalise it, even. “AGIs will be helping us on a lot of tasks”, “collusion is hard” and “people will get more scared over time” aren’t anywhere close to overcoming it imo.
These are what I mean by the vague intuitions.
Nobody has come anywhere near doing this satisfactorily. The most obvious explanation is that they can’t.
To be fair, I think I’m partly making wrong assumptions about what exactly you’re arguing for here.
On a slightly closer read, you don’t actually argue in this piece that it’s as high as 90% - I assumed that because I think you’ve argued for that previously, and I think that’s what “high” p(doom) normally means.
I do think it is basically ~90%, but I’m arguing here for doom being the default outcome of AGI; I think “high” can reasonably be interpreted as >50%.
I feel like this is a case of death by epistemic modesty, especially when it isn’t clear how these low p(doom) estimates are arrived at in a technical sense (and a lot seems to me like a kind of “respectability heuristic” cascade). We didn’t do very well with Covid as a society in the UK (and many other countries), following this kind of thinking.
What part of Greg writing comes across as over confident?
I suppose one solution might be to say that your personal view is that pdoom is >50%, but a range of estimates suggest >1% is plausible