I don’t think I would say the same thing to every project discussed on the EA forum. I think for every non-AI-focused project I’d say something similar (why not focus instead on AI?) but the bit about how I didn’t find QRI’s positive pitch compelling was specific to QRI. (I’m a philosopher, I love thinking about what things mean, but I think we’ve got to have a better story than “We are trying to make more good and less bad experiences, therefore we should try to objectively quantify and measure experience.” Compare: Suppose it were WW2, 1939. We are thinking of various ways to help the allied war effort. An institute designed to study “what does war even mean anyway? What does it mean to win a war? let’s try to objectively quantify this so we can measure how much we are winning and optimize that metric” is not obviously a good idea. Like, it’s definitely not harmful, but it wouldn’t be top priority, especially if there are various other projects that seem super important, tractable, and neglected, such as preventing the Axis from getting atom bombs. (I think of the EA community’s position with respect to AI as analogous to the position re atom bombs held by the small cohort of people in 1939 “in the know” about the possibility. It would be silly for someone who knew about atom bombs in 1939 to instead focus on objectively defining war and winning.)
But yeah, I would say to every non-AI-related project something like “Will your project be useful for making AI go well? How?” And I think that insofar as one could do good work on both AI safety stuff and something else, one should probably choose AI safety stuff. This isn’t because I think AI safety stuff is DEFINITELY the most important, merely that I think it probably is. (Also I think it’s more neglected AND tractable than many, though not all, of the alternatives people typically consider)
Some projects I think are still worth pursuing even if they don’t help make AI go well. For example, bio risk, preventing nuclear war, improving collective sanity/rationality/decision-making, … (lots of other things would be added, it all depends on tractibility + neglectedness + personal fit.) After all, maybe AI won’t happen for many decades or even centuries. Or maybe one of those other risks is more likely to happen soon than it appears.
Anyhow, to sum it all up: I agree that we shouldn’t be super confident that AI is the most important thing. Depending on how broadly you define AI, I’m probably about 80-90% confident. And I agree that this means our community should explore a portfolio of ideas rather than just one. Nevertheless, I think even our community is currently less focused on AI than it should be, and I think AI is the “gold standard” so to speak that projects should compare themselves to, and moreover I think QRI in particular has not done much to argue for their case. (Compare with, say, ALLFED which has a pretty good case IMO: There’s at least a 1% chance of some sort of global agricultural shortfall prior to AI getting crazy, and by default this will mean terrible collapse and famine, but if we prepare for this possibility it could instead mean much better things (people and institutions surviving, maybe learning)).
My criticism is not directly of QRI but of their argument as presented here. I expect that if I talked with them and heard more of their views, I’d hear a better, more expanded version of the argument that would be much more convincing. In fact I’d say 40% chance QRI ends up seeming better than ALLFED to me after such a conversation. For example, I myself used to think that consciousness research was really important for making AI go well. It might not be so hard to convince me to switch back to that old position.
Thanks for the detailed response kokotajlod, I appreciate it.
Let me summarize your viewpoint back to you to check I’ve understood correctly. It sounds as though you are saying that AI (broadly defined) is likely to be extremely important and the EA community currently underweights AI safety relative to its importance. Therefore, while you do think that not everyone will be suited to AI safety work and that the EA community should take a portfolio approach across problems, you think it’s important to highlight where projects do not seem as important as working on AI safety since that will help nudge the EA community towards a better-balanced portfolio. Outside of AI safety, there are a few other things that you think are also important, mostly in the existential risk kind of category but also including improving collective sanity/rationality/decision-making and maybe others. Therefore, the critique of QRI is mostly part of the activity to keep the portfolio properly balanced, however, you do have some additional skepticism that learning about what we mean by happiness and suffering is useful.
Is that roughly right?
If that is approximately your view, I think I have a couple of disagreements/things I’m confused about.
A. Firstly, I don’t think the WW2 example is quite right for this case. I think in the case of war, we understand the concept well enough to take the relevant actions and we don’t predict defining the concept to change that. I don’t think we understand the concepts of suffering or happiness well enough to take similar actions as in the WW2 case.
B. Secondly, I would have guessed that the EA community overweights AI safety so I’m curious as to why you think that is not the case. It could be that my intuitions are wrong about the focus it actually receives (vs the hype in the community) or it could be that I think it should receive less focus than you do. Not so much compared to its importance, more like its tractability when factoring in safety and the challenges of coordination. I worry that perhaps we overly focus on the technical side such that there’s a risk that we just speed up development more than we increase safety.
C. While I don’t know much about QRI’s research, in particular, my concerns from point B make me more inclined to support research in areas related to social sciences that might improve our understanding of and ability to coordinate.
D. And finally, why include “improving collective sanity/rationality/decision-making” in the list of other important things but exclude QRI? Here I’m not necessarily disagreeing, I just don’t quite get the underlying model that generates existential threats as the most important but then includes something like this and then excludes something like QRI.
To be clear, these are not confident viewpoints, they are simply areas where I notice my views seem to differ from many in the EA community and I expect I’d learn something useful from understanding why that is.
Yep, that’s roughly correct as a statement of my position. Thanks. I guess I’d put it slightly differently in some respects—I’d say something like “A good test for whether to do some EA project is how likely it is that it’s within a few orders of magnitude as good as AI safety work. There will be several projects for which we can tell a not-too-implausible story for how they are close to as good or better than AI safety work, and then we can let tractibility/neglectedness/fit considerations convince us to do them. But if we can’t even tell such a story in the first place, that’s a pretty bad sign.” The general thought is: AI safety is the “gold standard” to compare against, since it’s currently No. 1 priority in my book. (If something else was No. 1, it would be my gold standard.)I
think QRI actually can tell such a story, I just haven’t heard it yet. In the comments it seems that a story like this was sketched. I would be interested to hear it in more detail. I don’t think the very abstract story of “we are trying to make good experiences but we don’t know what experiences are” is plausible enough as a story for why this is close to as good as AI safety. (But I might be wrong about that too.)re
: A: Hmmm, fair enough that you disagree, but I have the opposite intuition.re
: B: Yeah I think even the EA community underweights AI safety. I have loads of respect for people doing animal welfare stuff and global poverty stuff, but it just doesn’t seem nearly as important as preventing everyone from being killed or worse in the near future. It also seems much less neglected—most of the quality-adjusted AI safety work is being done by EA-adjacent people, whereas that’s not true (I think?) for animal welfare or global poverty stuff. As for traceability, I’m less sure how to make the comparison—it’s obviously much more tractable to make SOME improvement to animal welfare or the lives of the global poor, but if we compare helping ALL the animals / ALL the global poor to AI safety, it actually seems less tractable (while still being less important and less neglected.) There’s a lot more to say about this topic obviously, I worry I come across as callous or ignorant of various nuances… so let me just say I’d love to discuss with you further and hear your thoughts.re
: D: I’m certainly pretty uncertain about the improving collective sanity thing. One reason I’m more optimistic about it than QRI is that I see how it plugs in to AI safety: If we improve collective sanity, that massively helps with AI safety, whereas if we succeed at understanding consciousness better, how does that help with AI safety? (QRI seems to think it does, I just don’t see it yet) Therefore sanity-improvement can be thought of as similarly important to AI safety (or alternatively as a kind of AI safety intervention) and the remaining question is how tractable and neglected it is. I’m unsure, but one thing that makes me optimistic about tractability is that we don’t need to improve sanity of the entire world, just a few small parts of the world—most importantly, our community, but also certain AI companies and (maybe) governments. And even if all we do is improve sanity of our own community, that has a substantially positive effect on AI safety already, since so much of AI safety work comes from our community. As for neglectedness, yeah IDK. Within our community there is a lot of focus on good epistemology and stuff already, so maybe the low-hanging fruit has been picked already. But subjectively I get the impression that there are still good things to be doing—e.g. trying to forecast how collective epistemology in the relevant communities could change in the coming years, building up new tools (such as Guesstimate or Metaculus) …
Good question. Here are my answers:
I don’t think I would say the same thing to every project discussed on the EA forum. I think for every non-AI-focused project I’d say something similar (why not focus instead on AI?) but the bit about how I didn’t find QRI’s positive pitch compelling was specific to QRI. (I’m a philosopher, I love thinking about what things mean, but I think we’ve got to have a better story than “We are trying to make more good and less bad experiences, therefore we should try to objectively quantify and measure experience.” Compare: Suppose it were WW2, 1939. We are thinking of various ways to help the allied war effort. An institute designed to study “what does war even mean anyway? What does it mean to win a war? let’s try to objectively quantify this so we can measure how much we are winning and optimize that metric” is not obviously a good idea. Like, it’s definitely not harmful, but it wouldn’t be top priority, especially if there are various other projects that seem super important, tractable, and neglected, such as preventing the Axis from getting atom bombs. (I think of the EA community’s position with respect to AI as analogous to the position re atom bombs held by the small cohort of people in 1939 “in the know” about the possibility. It would be silly for someone who knew about atom bombs in 1939 to instead focus on objectively defining war and winning.)
But yeah, I would say to every non-AI-related project something like “Will your project be useful for making AI go well? How?” And I think that insofar as one could do good work on both AI safety stuff and something else, one should probably choose AI safety stuff. This isn’t because I think AI safety stuff is DEFINITELY the most important, merely that I think it probably is. (Also I think it’s more neglected AND tractable than many, though not all, of the alternatives people typically consider)
Some projects I think are still worth pursuing even if they don’t help make AI go well. For example, bio risk, preventing nuclear war, improving collective sanity/rationality/decision-making, … (lots of other things would be added, it all depends on tractibility + neglectedness + personal fit.) After all, maybe AI won’t happen for many decades or even centuries. Or maybe one of those other risks is more likely to happen soon than it appears.
Anyhow, to sum it all up: I agree that we shouldn’t be super confident that AI is the most important thing. Depending on how broadly you define AI, I’m probably about 80-90% confident. And I agree that this means our community should explore a portfolio of ideas rather than just one. Nevertheless, I think even our community is currently less focused on AI than it should be, and I think AI is the “gold standard” so to speak that projects should compare themselves to, and moreover I think QRI in particular has not done much to argue for their case. (Compare with, say, ALLFED which has a pretty good case IMO: There’s at least a 1% chance of some sort of global agricultural shortfall prior to AI getting crazy, and by default this will mean terrible collapse and famine, but if we prepare for this possibility it could instead mean much better things (people and institutions surviving, maybe learning)).
My criticism is not directly of QRI but of their argument as presented here. I expect that if I talked with them and heard more of their views, I’d hear a better, more expanded version of the argument that would be much more convincing. In fact I’d say 40% chance QRI ends up seeming better than ALLFED to me after such a conversation. For example, I myself used to think that consciousness research was really important for making AI go well. It might not be so hard to convince me to switch back to that old position.
Thanks for the detailed response kokotajlod, I appreciate it.
Let me summarize your viewpoint back to you to check I’ve understood correctly. It sounds as though you are saying that AI (broadly defined) is likely to be extremely important and the EA community currently underweights AI safety relative to its importance. Therefore, while you do think that not everyone will be suited to AI safety work and that the EA community should take a portfolio approach across problems, you think it’s important to highlight where projects do not seem as important as working on AI safety since that will help nudge the EA community towards a better-balanced portfolio. Outside of AI safety, there are a few other things that you think are also important, mostly in the existential risk kind of category but also including improving collective sanity/rationality/decision-making and maybe others. Therefore, the critique of QRI is mostly part of the activity to keep the portfolio properly balanced, however, you do have some additional skepticism that learning about what we mean by happiness and suffering is useful.
Is that roughly right?
If that is approximately your view, I think I have a couple of disagreements/things I’m confused about.
A. Firstly, I don’t think the WW2 example is quite right for this case. I think in the case of war, we understand the concept well enough to take the relevant actions and we don’t predict defining the concept to change that. I don’t think we understand the concepts of suffering or happiness well enough to take similar actions as in the WW2 case.
B. Secondly, I would have guessed that the EA community overweights AI safety so I’m curious as to why you think that is not the case. It could be that my intuitions are wrong about the focus it actually receives (vs the hype in the community) or it could be that I think it should receive less focus than you do. Not so much compared to its importance, more like its tractability when factoring in safety and the challenges of coordination. I worry that perhaps we overly focus on the technical side such that there’s a risk that we just speed up development more than we increase safety.
C. While I don’t know much about QRI’s research, in particular, my concerns from point B make me more inclined to support research in areas related to social sciences that might improve our understanding of and ability to coordinate.
D. And finally, why include “improving collective sanity/rationality/decision-making” in the list of other important things but exclude QRI? Here I’m not necessarily disagreeing, I just don’t quite get the underlying model that generates existential threats as the most important but then includes something like this and then excludes something like QRI.
To be clear, these are not confident viewpoints, they are simply areas where I notice my views seem to differ from many in the EA community and I expect I’d learn something useful from understanding why that is.
Thanks for the detailed engagement!
Yep, that’s roughly correct as a statement of my position. Thanks. I guess I’d put it slightly differently in some respects—I’d say something like “A good test for whether to do some EA project is how likely it is that it’s within a few orders of magnitude as good as AI safety work. There will be several projects for which we can tell a not-too-implausible story for how they are close to as good or better than AI safety work, and then we can let tractibility/neglectedness/fit considerations convince us to do them. But if we can’t even tell such a story in the first place, that’s a pretty bad sign.” The general thought is: AI safety is the “gold standard” to compare against, since it’s currently No. 1 priority in my book. (If something else was No. 1, it would be my gold standard.)I
think QRI actually can tell such a story, I just haven’t heard it yet. In the comments it seems that a story like this was sketched. I would be interested to hear it in more detail. I don’t think the very abstract story of “we are trying to make good experiences but we don’t know what experiences are” is plausible enough as a story for why this is close to as good as AI safety. (But I might be wrong about that too.)re
: A: Hmmm, fair enough that you disagree, but I have the opposite intuition.re
: B: Yeah I think even the EA community underweights AI safety. I have loads of respect for people doing animal welfare stuff and global poverty stuff, but it just doesn’t seem nearly as important as preventing everyone from being killed or worse in the near future. It also seems much less neglected—most of the quality-adjusted AI safety work is being done by EA-adjacent people, whereas that’s not true (I think?) for animal welfare or global poverty stuff. As for traceability, I’m less sure how to make the comparison—it’s obviously much more tractable to make SOME improvement to animal welfare or the lives of the global poor, but if we compare helping ALL the animals / ALL the global poor to AI safety, it actually seems less tractable (while still being less important and less neglected.) There’s a lot more to say about this topic obviously, I worry I come across as callous or ignorant of various nuances… so let me just say I’d love to discuss with you further and hear your thoughts.re
: D: I’m certainly pretty uncertain about the improving collective sanity thing. One reason I’m more optimistic about it than QRI is that I see how it plugs in to AI safety: If we improve collective sanity, that massively helps with AI safety, whereas if we succeed at understanding consciousness better, how does that help with AI safety? (QRI seems to think it does, I just don’t see it yet) Therefore sanity-improvement can be thought of as similarly important to AI safety (or alternatively as a kind of AI safety intervention) and the remaining question is how tractable and neglected it is. I’m unsure, but one thing that makes me optimistic about tractability is that we don’t need to improve sanity of the entire world, just a few small parts of the world—most importantly, our community, but also certain AI companies and (maybe) governments. And even if all we do is improve sanity of our own community, that has a substantially positive effect on AI safety already, since so much of AI safety work comes from our community. As for neglectedness, yeah IDK. Within our community there is a lot of focus on good epistemology and stuff already, so maybe the low-hanging fruit has been picked already. But subjectively I get the impression that there are still good things to be doing—e.g. trying to forecast how collective epistemology in the relevant communities could change in the coming years, building up new tools (such as Guesstimate or Metaculus) …