Yep, that’s roughly correct as a statement of my position. Thanks. I guess I’d put it slightly differently in some respects—I’d say something like “A good test for whether to do some EA project is how likely it is that it’s within a few orders of magnitude as good as AI safety work. There will be several projects for which we can tell a not-too-implausible story for how they are close to as good or better than AI safety work, and then we can let tractibility/neglectedness/fit considerations convince us to do them. But if we can’t even tell such a story in the first place, that’s a pretty bad sign.” The general thought is: AI safety is the “gold standard” to compare against, since it’s currently No. 1 priority in my book. (If something else was No. 1, it would be my gold standard.)I
think QRI actually can tell such a story, I just haven’t heard it yet. In the comments it seems that a story like this was sketched. I would be interested to hear it in more detail. I don’t think the very abstract story of “we are trying to make good experiences but we don’t know what experiences are” is plausible enough as a story for why this is close to as good as AI safety. (But I might be wrong about that too.)re
: A: Hmmm, fair enough that you disagree, but I have the opposite intuition.re
: B: Yeah I think even the EA community underweights AI safety. I have loads of respect for people doing animal welfare stuff and global poverty stuff, but it just doesn’t seem nearly as important as preventing everyone from being killed or worse in the near future. It also seems much less neglected—most of the quality-adjusted AI safety work is being done by EA-adjacent people, whereas that’s not true (I think?) for animal welfare or global poverty stuff. As for traceability, I’m less sure how to make the comparison—it’s obviously much more tractable to make SOME improvement to animal welfare or the lives of the global poor, but if we compare helping ALL the animals / ALL the global poor to AI safety, it actually seems less tractable (while still being less important and less neglected.) There’s a lot more to say about this topic obviously, I worry I come across as callous or ignorant of various nuances… so let me just say I’d love to discuss with you further and hear your thoughts.re
: D: I’m certainly pretty uncertain about the improving collective sanity thing. One reason I’m more optimistic about it than QRI is that I see how it plugs in to AI safety: If we improve collective sanity, that massively helps with AI safety, whereas if we succeed at understanding consciousness better, how does that help with AI safety? (QRI seems to think it does, I just don’t see it yet) Therefore sanity-improvement can be thought of as similarly important to AI safety (or alternatively as a kind of AI safety intervention) and the remaining question is how tractable and neglected it is. I’m unsure, but one thing that makes me optimistic about tractability is that we don’t need to improve sanity of the entire world, just a few small parts of the world—most importantly, our community, but also certain AI companies and (maybe) governments. And even if all we do is improve sanity of our own community, that has a substantially positive effect on AI safety already, since so much of AI safety work comes from our community. As for neglectedness, yeah IDK. Within our community there is a lot of focus on good epistemology and stuff already, so maybe the low-hanging fruit has been picked already. But subjectively I get the impression that there are still good things to be doing—e.g. trying to forecast how collective epistemology in the relevant communities could change in the coming years, building up new tools (such as Guesstimate or Metaculus) …
Thanks for the detailed engagement!
Yep, that’s roughly correct as a statement of my position. Thanks. I guess I’d put it slightly differently in some respects—I’d say something like “A good test for whether to do some EA project is how likely it is that it’s within a few orders of magnitude as good as AI safety work. There will be several projects for which we can tell a not-too-implausible story for how they are close to as good or better than AI safety work, and then we can let tractibility/neglectedness/fit considerations convince us to do them. But if we can’t even tell such a story in the first place, that’s a pretty bad sign.” The general thought is: AI safety is the “gold standard” to compare against, since it’s currently No. 1 priority in my book. (If something else was No. 1, it would be my gold standard.)I
think QRI actually can tell such a story, I just haven’t heard it yet. In the comments it seems that a story like this was sketched. I would be interested to hear it in more detail. I don’t think the very abstract story of “we are trying to make good experiences but we don’t know what experiences are” is plausible enough as a story for why this is close to as good as AI safety. (But I might be wrong about that too.)re
: A: Hmmm, fair enough that you disagree, but I have the opposite intuition.re
: B: Yeah I think even the EA community underweights AI safety. I have loads of respect for people doing animal welfare stuff and global poverty stuff, but it just doesn’t seem nearly as important as preventing everyone from being killed or worse in the near future. It also seems much less neglected—most of the quality-adjusted AI safety work is being done by EA-adjacent people, whereas that’s not true (I think?) for animal welfare or global poverty stuff. As for traceability, I’m less sure how to make the comparison—it’s obviously much more tractable to make SOME improvement to animal welfare or the lives of the global poor, but if we compare helping ALL the animals / ALL the global poor to AI safety, it actually seems less tractable (while still being less important and less neglected.) There’s a lot more to say about this topic obviously, I worry I come across as callous or ignorant of various nuances… so let me just say I’d love to discuss with you further and hear your thoughts.re
: D: I’m certainly pretty uncertain about the improving collective sanity thing. One reason I’m more optimistic about it than QRI is that I see how it plugs in to AI safety: If we improve collective sanity, that massively helps with AI safety, whereas if we succeed at understanding consciousness better, how does that help with AI safety? (QRI seems to think it does, I just don’t see it yet) Therefore sanity-improvement can be thought of as similarly important to AI safety (or alternatively as a kind of AI safety intervention) and the remaining question is how tractable and neglected it is. I’m unsure, but one thing that makes me optimistic about tractability is that we don’t need to improve sanity of the entire world, just a few small parts of the world—most importantly, our community, but also certain AI companies and (maybe) governments. And even if all we do is improve sanity of our own community, that has a substantially positive effect on AI safety already, since so much of AI safety work comes from our community. As for neglectedness, yeah IDK. Within our community there is a lot of focus on good epistemology and stuff already, so maybe the low-hanging fruit has been picked already. But subjectively I get the impression that there are still good things to be doing—e.g. trying to forecast how collective epistemology in the relevant communities could change in the coming years, building up new tools (such as Guesstimate or Metaculus) …