(f) not something I know enough about to comment on but also something I don’t think I have a reason to prioritize looking into further (as I can’t look into everything) -- #1, #8, #17
Though I guess any good list would include a combination of all six. And of course I could be the wrong one!
I’d particularly really like to hear more about “nuclear safety being as important as AI alignment and plausibly contributing to AI risk via overhang” as I think this could change my priorities if true. Is the idea just that nuclear weapons are a particularly viable attack vector for a hostile AI?
Gavin (the OP) and I agree on #5, #10, #13, #18, and #20 are important, neglected, or at least intriguing. We also agree on #6, #7, and #15 being wrong/nonsensical/correctly ignored.
Gavin thinks #12 is wrong / correctly ignored but I think it is correct + taken into account just fine by EAs. I’m happy to switch to Gavin’s position on this.
So where Gavin and I disagree is #2 and #14. Gavin finds these intriguing but I think #2 is wrong / correctly ignored and I think #14 is correct but taken into account just fine by EAs / not very neglected on the margin.
Note that as of the time of writing Gavin has not opined on #1 (IMO no comment), #3 (IMO wrong), #4 (IMO nonsensical), #8 (IMO no comment), #9 (IMO wrong), #11 (IMO intriguing), #16 (IMO intriguing), #17 (IMO no comment), #19 (IMO intriguing), #21 (IMO nonsensical), or #22 (IMO taken into account just fine by EAs).
Similarly, I don’t think societal safety will be much of a priority as long as we’re a society that is ravaged by mental health problems. It might be that solving mental health is a necessary pre-cursor to the sort of safety spending/efforts we would need to achieve existential security.
Hey Peter, on your last point, I believe the clearest paths from AI to x-risk run directly through either nuclear weapons or bioweapons. Not sure if the author believes the same, but here’s some thoughts I wrote up on the topic:
11. EA correctly identifies improving institutional decision-making as important but hasn’t yet grappled with the radical political implications of doing that
This one feels like it requires substantial unpacking; I’ll probably expand on it further at some point.
Essentially the existing power structure is composed of organizations (mostly large bureaucracies) and all of these organizations have (formal and informal) immunological responses that activate when someone tries to change them. (Here’s some flavor to pump intuition on this.)
To improve something is to change it. There are few Pareto improvements available on the current margin, and those that exist are often not perceived as Pareto by all who would be touched by the change. So attempts to improve institutional decision-making trigger organizational immune responses by default.
These immune responses are often opaque and informal, especially in the first volleys. And they can arise emergently: top-down coordination isn’t required to generate them, only incentive gradients.
The New York Times’ assault on Scott Alexander (a) is an example to build some intuition of what this can look like: the ascendant power of Slate Star Codex began to feel threatening to the Times and so the Times moved against SSC.
16. taking dharma seriously a la @RomeoStevens76′s current research direction
I’ve since realized that this would be best accomplished by generalizing (and modernizing) to a broader category, which we’ve taken to referring to as valence studies.
19. worldview drift of elite EA orgs (e.g. @CSETGeorgetown, @open_phil) via mimesis being real and concerning
It’s hard to ground things objectively, so social structures tend to become more like the other social structures around them.
CSET is surrounded by and intercourses with DC-style think tanks, so it is becoming more like a DC-style think tank (e.g. suiting up starts to seem like a good idea).
Thanks for this—a good mix of ideas that are:
(a) well-taken and important IMO and indeed neglected by other EAs IMO (though I wouldn’t say they’re literally unhearable) -- #5, #13, #18, #20
(b) intriguing IMO and I want to hear more -- #10, #11, #16, #19
(c) actually I think taken into account just fine by EAs -- #12, #14, #22
(d) just wrong / correctly ignored IMO -- #2, #3, #6, #7, #9
(e) nonsensical …at least to me -- #4, #15, #21
(f) not something I know enough about to comment on but also something I don’t think I have a reason to prioritize looking into further (as I can’t look into everything) -- #1, #8, #17
Though I guess any good list would include a combination of all six. And of course I could be the wrong one!
I’d particularly really like to hear more about “nuclear safety being as important as AI alignment and plausibly contributing to AI risk via overhang” as I think this could change my priorities if true. Is the idea just that nuclear weapons are a particularly viable attack vector for a hostile AI?
Gavin (the OP) and I agree on #5, #10, #13, #18, and #20 are important, neglected, or at least intriguing. We also agree on #6, #7, and #15 being wrong/nonsensical/correctly ignored.
Gavin thinks #12 is wrong / correctly ignored but I think it is correct + taken into account just fine by EAs. I’m happy to switch to Gavin’s position on this.
So where Gavin and I disagree is #2 and #14. Gavin finds these intriguing but I think #2 is wrong / correctly ignored and I think #14 is correct but taken into account just fine by EAs / not very neglected on the margin.
Note that as of the time of writing Gavin has not opined on #1 (IMO no comment), #3 (IMO wrong), #4 (IMO nonsensical), #8 (IMO no comment), #9 (IMO wrong), #11 (IMO intriguing), #16 (IMO intriguing), #17 (IMO no comment), #19 (IMO intriguing), #21 (IMO nonsensical), or #22 (IMO taken into account just fine by EAs).
At first I thought this was incorrect but I think there might be a kernel of truth here—although I have a different framing.
It has been suggested that boosting economic growth can lower existential risk as, if we’re richer, we’ll want to spend more on safety. On the other hand when you’re poor, you just want to get richer.
Similarly, I don’t think societal safety will be much of a priority as long as we’re a society that is ravaged by mental health problems. It might be that solving mental health is a necessary pre-cursor to the sort of safety spending/efforts we would need to achieve existential security.
I’ve thought about this a bit and don’t think #2 is incorrect, although I could quibble with it as an “important” factor.
I think broadly improving mental health could reduce catastrophic risk if:
A. Catastrophic technologies (i.e., Big Red Buttons) will become cheaper to access. B. Someone unhinged is likelier to press a Big Red Button.
The connection here doesn’t seem mysterious at all to me. Sane people are less likely to end the world.
However, this may be more about reducing variance in mental health than increasing its average level.
Now I wish there were numbers in the OP to make referencing easier
Edit: thanks
Looks like the OP added numbers. Thanks OP!
Hey Peter, on your last point, I believe the clearest paths from AI to x-risk run directly through either nuclear weapons or bioweapons. Not sure if the author believes the same, but here’s some thoughts I wrote up on the topic:
https://forum.effectivealtruism.org/posts/7ZZpWPq5iqkLMmt25/aidan-o-gara-s-shortform?commentId=rnM3FAHtBpymBsdT7
Yes, I have a similar position that early-AGI risk runs through nuclear mostly. I wrote my thoughts on this here: When Bits Split Atoms
Thanks I’ll take a look!
(b) intriguing IMO and I want to hear more -- #10, #11, #16, #19
10. nuclear safety being as important as AI alignment and plausibly contributing to AI risk via overhang
See discussion in this thread
11. EA correctly identifies improving institutional decision-making as important but hasn’t yet grappled with the radical political implications of doing that
This one feels like it requires substantial unpacking; I’ll probably expand on it further at some point.
Essentially the existing power structure is composed of organizations (mostly large bureaucracies) and all of these organizations have (formal and informal) immunological responses that activate when someone tries to change them. (Here’s some flavor to pump intuition on this.)
To improve something is to change it. There are few Pareto improvements available on the current margin, and those that exist are often not perceived as Pareto by all who would be touched by the change. So attempts to improve institutional decision-making trigger organizational immune responses by default.
These immune responses are often opaque and informal, especially in the first volleys. And they can arise emergently: top-down coordination isn’t required to generate them, only incentive gradients.
The New York Times’ assault on Scott Alexander (a) is an example to build some intuition of what this can look like: the ascendant power of Slate Star Codex began to feel threatening to the Times and so the Times moved against SSC.
16. taking dharma seriously a la @RomeoStevens76′s current research direction
I’ve since realized that this would be best accomplished by generalizing (and modernizing) to a broader category, which we’ve taken to referring to as valence studies.
19. worldview drift of elite EA orgs (e.g. @CSETGeorgetown, @open_phil) via mimesis being real and concerning
I’m basically saying that mimesis is a thing.
It’s hard to ground things objectively, so social structures tend to become more like the other social structures around them.
CSET is surrounded by and intercourses with DC-style think tanks, so it is becoming more like a DC-style think tank (e.g. suiting up starts to seem like a good idea).
Open Phil interfaces with a lot of mainstream philanthropy, and it’s starting to give away money in more mainstream ways.