I really appreciate you writing this. Getting clear on one’s own reasoning about AI seems really valuable, but for many people, myself included, it’s too daunting to actually do.
If you think it’s relevant to your overall point, I would suggest moving the first two footnotes (clarifying what you mean by short timelines and high risk) into the main text. Short timelines sometimes means <10 years and high risk sometimesmeans >95%.
I think you’re expressing your attitude to the general cluster of EA/rationalist views around AI risk typified by eg. Holden and Ajeya’s views (and maybe Paul Christiano’s, I don’t know) rather than a subset of those views typified by eg. Eliezer (and maybe other MIRI people and Daniel Kokotajlo, I don’t know). To me, the main text implies you’re thinking about the second kind of view, but the footnotes are about the first.
And different arguments in the post apply more strongly to different views. Eg
Fewer ‘smart people disagree’ about the numbers in your footnote than about the more extreme view.
I’m not sure Eliezer having occasionally been overconfident, but got the general shape of things right is any evidence at all against >50% AGI in 30 years or >15% chance of catastrophe this century (though it could be evidence against Eliezer’s very high risk view).
The Carlsmith post you say you roughly endorse seems to have 65% on AGI in 50 years, with a 10% chance of existential catastophe overall. So I’m not sure if that means your conclusion is
‘I agree with this view I’ve been critically examining’
‘I’m still skeptical of 30 year timelines with >15% risk, but I roughly endorse 50 year timelines with 10% risk’
‘I’m skeptical of 10 year timelines with >50% risk, but I roughly endorse 30-50 year timelines with 5-20% risk’
Thanks for pointing this out. I agree that I wasn’t clear about this in the post.
My hesitations have been around adopting views with timelines and risk level that are at least as concerning as the OpenPhil cluster (Holden, Ajeya, etc.) that you’re pointing at; essentially views that seem to imply that AI and things that feed into it are clearly the most important cause area.
I’m not sure Eliezer having occasionally been overconfident, but got the general shape of things right is any evidence at all against >50% AGI in 30 years or >15% chance of catastrophe this century (though it could be evidence against Eliezer’s very high risk view).
I wouldn’t go as far as no evidence at all given that my understanding is Eliezer (+ MIRI) was heavily involved in influencing the OpenPhil’s cluster’s views so it’s not entirely independent, but I agree it’s much weaker evidence for less extreme views.
Fewer ‘smart people disagree’ about the numbers in your footnote than about the more extreme view.′
I was going to say that it seems like a big difference within our community, but both clusters of views are very far away from the median pretty reasonable person and the median AI researcher. Though I suppose the latter actually isn’t far away on timelines (potentially depending on the framing?). It definitely seems to be in significant tension with how AI researchers and the general public / markets / etc. act, regardless of stated beliefs (e.g. I found it interesting how short the American public’s timelines are, compared to their actions).
Anyway, overall I think you’re right that it makes a difference but it seems like a substantive concern for both clusters of views.
The Carlsmith post you say you roughly endorse seems to have 65% on AGI in 50 years, with a 10% chance of existential catastophe overall. So I’m not sure if that means your conclusion is
[...]
The conclusion I intend to convey is something like “I’m no longer as hesitant about adopting views which are at least as concerning as >50% of AGI/TAI/APS-AI within 30 years, and >15% chance of existential catastrophe this century” which as I referred to above seem to make AI clearly the most important cause area.
Copying my current state on the object level views from another recent post:
I’m now at ~20% by 2036; my median is now ~2050 though still with a fat right tail.
My timelines shortening [due to reflecting on MATH breakthrough] should also increase my p(AI doom by 2100) a bit, though I’m still working out my views here. I’m guessing I’ll land somewhere between 20 and 60% [TBC, most of the variance is coming from working out my views and not the MATH breakthrough].
I really appreciate you writing this. Getting clear on one’s own reasoning about AI seems really valuable, but for many people, myself included, it’s too daunting to actually do.
If you think it’s relevant to your overall point, I would suggest moving the first two footnotes (clarifying what you mean by short timelines and high risk) into the main text. Short timelines sometimes means <10 years and high risk sometimes means >95%.
I think you’re expressing your attitude to the general cluster of EA/rationalist views around AI risk typified by eg. Holden and Ajeya’s views (and maybe Paul Christiano’s, I don’t know) rather than a subset of those views typified by eg. Eliezer (and maybe other MIRI people and Daniel Kokotajlo, I don’t know). To me, the main text implies you’re thinking about the second kind of view, but the footnotes are about the first.
And different arguments in the post apply more strongly to different views. Eg
Fewer ‘smart people disagree’ about the numbers in your footnote than about the more extreme view.
I’m not sure Eliezer having occasionally been overconfident, but got the general shape of things right is any evidence at all against >50% AGI in 30 years or >15% chance of catastrophe this century (though it could be evidence against Eliezer’s very high risk view).
The Carlsmith post you say you roughly endorse seems to have 65% on AGI in 50 years, with a 10% chance of existential catastophe overall. So I’m not sure if that means your conclusion is
‘I agree with this view I’ve been critically examining’
‘I’m still skeptical of 30 year timelines with >15% risk, but I roughly endorse 50 year timelines with 10% risk’
‘I’m skeptical of 10 year timelines with >50% risk, but I roughly endorse 30-50 year timelines with 5-20% risk’
Or something else
Thanks for pointing this out. I agree that I wasn’t clear about this in the post.
My hesitations have been around adopting views with timelines and risk level that are at least as concerning as the OpenPhil cluster (Holden, Ajeya, etc.) that you’re pointing at; essentially views that seem to imply that AI and things that feed into it are clearly the most important cause area.
I wouldn’t go as far as no evidence at all given that my understanding is Eliezer (+ MIRI) was heavily involved in influencing the OpenPhil’s cluster’s views so it’s not entirely independent, but I agree it’s much weaker evidence for less extreme views.
I was going to say that it seems like a big difference within our community, but both clusters of views are very far away from the median pretty reasonable person and the median AI researcher. Though I suppose the latter actually isn’t far away on timelines (potentially depending on the framing?). It definitely seems to be in significant tension with how AI researchers and the general public / markets / etc. act, regardless of stated beliefs (e.g. I found it interesting how short the American public’s timelines are, compared to their actions).
Anyway, overall I think you’re right that it makes a difference but it seems like a substantive concern for both clusters of views.
The conclusion I intend to convey is something like “I’m no longer as hesitant about adopting views which are at least as concerning as >50% of AGI/TAI/APS-AI within 30 years, and >15% chance of existential catastrophe this century” which as I referred to above seem to make AI clearly the most important cause area.
Copying my current state on the object level views from another recent post: