Presumably this will differ a fair bit for different members of the LTFF, but speaking personally, my p(doom) is around 30%,[1] and my median timelines are ~15 years (though with high uncertainty). I haven’t thought as much about 10% timelines, but it would be some single-digit number of years.
Though a large chunk of the remainder includes outcomes that are much “better” than today but which are also very suboptimal – e.g., due to “good-enough” alignment + ~shard theory + etc, AI turns most of the reachable universe into paperclips but leaves humans + our descendants to do what we want with the Milky Way. This is arguably an existential catastrophe in terms of opportunity cost, but wouldn’t represent human extinction or disempowerment of humanity in the same way as “doom.”
Interesting that you give significant weight to non-extinction existential catastrophes (such as the AI leaving us the Milky Way). By what mechanism would that happen? Naively, all or (especially) nothing seem much more likely. It doesn’t seem like we’d have much bargaining power with not perfectly-aligned ASI. If it’s something analogous to us preserving other species, then I’m not optimistic that we’d get anything close to a flourishing civilisation confined to one galaxy. A small population in a “zoo”; or grossly distorted “pet” versions of humans; or merely being kept, overwhelmingly inactive, in digital storage, seem more likely.
So I’m imagining, for instance, AGIs with some shards of caring about human ~autonomy, but also other (stronger) shards that are for caring about (say) paperclips (also this was just meant as an example). I was also thinking that this might be what “a small population in a ‘zoo’” would look like – the Milky Way is small compared to the reachable universe! (Though before writing out my response, I almost wrote it as “our solar system” instead of “the Milky Way,” so I was imagining a relatively expansive set within this category; I’m not sure if distorted “pet” versions of humans would qualify or not.)
FWIW, I think specific changes here are unlikely to be cruxy for the decisions we make.
[Edited to add: I think if we could know with certainty that AGI was coming in 202X for a specific X, then that would be decision-relevant for certain decisions we’d face. But a shift of a few years for the 10% mark seems less decision relevant]
I think it’s super decision-relevant if the shift leads you to 10%(+) in 2023 or 2024. Basically I think we can no longer rely on having enough time for alignment research to bear fruit, so we should be shifting the bulk of resources toward directly buying more time (i.e. pushing for a global moratorium on AGI).
Do you have specific examples of mistakes you think we’re making, eg (with permission from the applicants) grants we didn’t make that we would if we have shorter 10% timelines, or grants that we made that we shouldn’t?
Presumably this will differ a fair bit for different members of the LTFF, but speaking personally, my p(doom) is around 30%,[1] and my median timelines are ~15 years (though with high uncertainty). I haven’t thought as much about 10% timelines, but it would be some single-digit number of years.
Though a large chunk of the remainder includes outcomes that are much “better” than today but which are also very suboptimal – e.g., due to “good-enough” alignment + ~shard theory + etc, AI turns most of the reachable universe into paperclips but leaves humans + our descendants to do what we want with the Milky Way. This is arguably an existential catastrophe in terms of opportunity cost, but wouldn’t represent human extinction or disempowerment of humanity in the same way as “doom.”
Interesting that you give significant weight to non-extinction existential catastrophes (such as the AI leaving us the Milky Way). By what mechanism would that happen? Naively, all or (especially) nothing seem much more likely. It doesn’t seem like we’d have much bargaining power with not perfectly-aligned ASI. If it’s something analogous to us preserving other species, then I’m not optimistic that we’d get anything close to a flourishing civilisation confined to one galaxy. A small population in a “zoo”; or grossly distorted “pet” versions of humans; or merely being kept, overwhelmingly inactive, in digital storage, seem more likely.
So I’m imagining, for instance, AGIs with some shards of caring about human ~autonomy, but also other (stronger) shards that are for caring about (say) paperclips (also this was just meant as an example). I was also thinking that this might be what “a small population in a ‘zoo’” would look like – the Milky Way is small compared to the reachable universe! (Though before writing out my response, I almost wrote it as “our solar system” instead of “the Milky Way,” so I was imagining a relatively expansive set within this category; I’m not sure if distorted “pet” versions of humans would qualify or not.)
Why wouldn’t the stronger shards just overpower the weaker shards?
Please keep this in mind in your grantmaking.
FWIW, I think specific changes here are unlikely to be cruxy for the decisions we make.
[Edited to add: I think if we could know with certainty that AGI was coming in 202X for a specific X, then that would be decision-relevant for certain decisions we’d face. But a shift of a few years for the 10% mark seems less decision relevant]
I think it’s super decision-relevant if the shift leads you to 10%(+) in 2023 or 2024. Basically I think we can no longer rely on having enough time for alignment research to bear fruit, so we should be shifting the bulk of resources toward directly buying more time (i.e. pushing for a global moratorium on AGI).
Do you have specific examples of mistakes you think we’re making, eg (with permission from the applicants) grants we didn’t make that we would if we have shorter 10% timelines, or grants that we made that we shouldn’t?
I don’t know specifics on who has applied to LTFF, but I think you should be funding orgs and people like these:
(Maybe there is a bottleneck on applications too.)