I can understand many of these points, though I disagree with most of them. I think the speculativeness point worries me most though, and I see it pretty frequently. I totally agree that AI risks are currently very uncertain and speculative, but I guess I think the relevance of this comes down to a few points:
Is it highly plausible that when AI as smart as or smarter than humans arrives, this will be a huge, world changing threat?
Around how long do we need to address this threat properly?
How soon before this threat materializes do we think our understanding of the risks will cross your threshold of rigor?
You might disagree on any of this, but for my own part I think it is fairly intuitive that the answers to these are “yes”, “decades at least”, and “years at most” respectively when you think about it. Taken together, this means that the speculativeness objection will by default sleepwalk us into the worst defaults of this risk, and that we should really start taking this risk as seriously as we ever plan to when it is still uncertain and speculative.
I think this on its own doesn’t answer whether it is a good cause area right now, alien invasion, the expansion of the sun, and the heat death of the universe all look like similarly big and hard problems, but they are arguably less urgent, we expect them much longer from now. A final assumption needed to worry about AI risks now, which you seem to disagree on, is that this is coming pretty darn soon.
I want to emphasize this as much as possible, this is super unclear and all of the arguments about when this is coming are sort of pretty terrible, but all of the most systematic, least pretty terrible ones I’m aware of converge on “around a century or sooner, probably sooner, possibly much sooner”, like the partially informative priors study, Ajeya Cotra’s biological anchors report (which Cotra herself thinks estimates too late an arrival date), expert surveys, and metaculus.
Again, all of this could very easily be wrong, but I don’t see a good enough reason to default to that assumption, so I think it just is the case that, not only should we take this risk as seriously as we ever plan to while it’s still speculative, but we should take this risk as seriously as we ever plan to as soon as possible. I would recommend reading Holden Karnofsky’s most important century series for a more spelled out version of similar points, especially about timelines, if you’re interested, but that’s my basic view on this issue and how to react to the speculativeness.
I do agree that there is some risk, and it’s certainly worth some thought and research. However, in the EA context, the cause areas should have effective interventions. Due to all this uncertainty, AI risk seems a very low-priority cause, since we cannot be sure if the research and other projects funded have any real impact. It would seem more beneficial to use the money for interventions that have been proved effective. That is why I think that EA is a wrong platform for AI risk discussion.
On the standard “importance, tractability, neglectedness” framework, I agree that tractability is AI risk’s worst feature if that’s what you mean. I think there is some consensus on this amongst people worried about the issue, as stated in 80k’s recently updated profile on the issue:
“Making progress on preventing an AI-related catastrophe seems hard, but there are a lot of avenues for more research and the field is very young. So we think it’s moderately tractable, though we’re highly uncertain — again, assessments of the tractability of making AI safe vary enormously.”
I think these other two aspects, importance and neglectedness, just matter a great deal and it would be a bad idea to disqualify cause areas just for moderately weak tractability. In terms of importance, transformative AI seems like it could easily be the most powerful technology we’ve ever made, for roughly the same reasons that humans are the most transformative “technology” on Earth right now. But even if you think this is overrated, consider the relatively meager funds and tiny field as it exists today. I think many people who find the risk a bit out there would at least agree with you that it’s “worth some thought and research”, but because of the rarity of the type of marginal thinking about good and willingness to take weird-sounding ideas seriously found in EA, practically no one else is ensuring that there is some thought and research. The field would, arguably, almost entirely dry up if EA stopped routing resources and people towards it.
Again though, I think maybe some of the disagreement is bound up in the “some risk” idea. My vague impression, and correct me if this doesn’t describe you, is that people who are weirded out by EA working on this as a cause area think that it’s a bit like if EA was getting people, right now, to work on risks from alien invasions (and then a big question is why isn’t it?), whereas people like me who are worried about it think that it is closer to working on risks from alien invasions if NASA discovered an alien spaceship parked five lightyears away from us. The risks here would still be very uncertain, the timelines, what we might be able to do to help, what sorts of things these aliens would be able to or want to do, but I think it would still look crazy if almost no one was looking into it, and I would be very wary of telling one of the only groups that was trying to look into it that they should let someone else handle it.
If you would like I would be happy to chat more about this, either by DMs, or email, or voice/video call. I’m probably not the most qualified person since I’m not in the field, but in a way that might give you a better sense of why the typical EA who is worried about this is. I guess I would like to make this an open invitation for anyone this post resonates with. Feel absolutely no pressure to though, and if you prefer I could just link some resources I think are helpful.
I’m just in the awkward position of both being very worried about this risk, and being very worried about how EA talking about this risk might put potential EAs off. I think it would be a real shame if you felt unwelcome or uncomfortable in the movement because you disagree about this risk, and if there’s something I can do to try to at least persuade you that those of us who are worried are worth sharing the movement with at least, I would like to try to do that.
I can understand many of these points, though I disagree with most of them. I think the speculativeness point worries me most though, and I see it pretty frequently. I totally agree that AI risks are currently very uncertain and speculative, but I guess I think the relevance of this comes down to a few points:
Is it highly plausible that when AI as smart as or smarter than humans arrives, this will be a huge, world changing threat?
Around how long do we need to address this threat properly?
How soon before this threat materializes do we think our understanding of the risks will cross your threshold of rigor?
You might disagree on any of this, but for my own part I think it is fairly intuitive that the answers to these are “yes”, “decades at least”, and “years at most” respectively when you think about it. Taken together, this means that the speculativeness objection will by default sleepwalk us into the worst defaults of this risk, and that we should really start taking this risk as seriously as we ever plan to when it is still uncertain and speculative.
I think this on its own doesn’t answer whether it is a good cause area right now, alien invasion, the expansion of the sun, and the heat death of the universe all look like similarly big and hard problems, but they are arguably less urgent, we expect them much longer from now. A final assumption needed to worry about AI risks now, which you seem to disagree on, is that this is coming pretty darn soon.
I want to emphasize this as much as possible, this is super unclear and all of the arguments about when this is coming are sort of pretty terrible, but all of the most systematic, least pretty terrible ones I’m aware of converge on “around a century or sooner, probably sooner, possibly much sooner”, like the partially informative priors study, Ajeya Cotra’s biological anchors report (which Cotra herself thinks estimates too late an arrival date), expert surveys, and metaculus.
Again, all of this could very easily be wrong, but I don’t see a good enough reason to default to that assumption, so I think it just is the case that, not only should we take this risk as seriously as we ever plan to while it’s still speculative, but we should take this risk as seriously as we ever plan to as soon as possible. I would recommend reading Holden Karnofsky’s most important century series for a more spelled out version of similar points, especially about timelines, if you’re interested, but that’s my basic view on this issue and how to react to the speculativeness.
I do agree that there is some risk, and it’s certainly worth some thought and research. However, in the EA context, the cause areas should have effective interventions. Due to all this uncertainty, AI risk seems a very low-priority cause, since we cannot be sure if the research and other projects funded have any real impact. It would seem more beneficial to use the money for interventions that have been proved effective. That is why I think that EA is a wrong platform for AI risk discussion.
On the standard “importance, tractability, neglectedness” framework, I agree that tractability is AI risk’s worst feature if that’s what you mean. I think there is some consensus on this amongst people worried about the issue, as stated in 80k’s recently updated profile on the issue:
“Making progress on preventing an AI-related catastrophe seems hard, but there are a lot of avenues for more research and the field is very young. So we think it’s moderately tractable, though we’re highly uncertain — again, assessments of the tractability of making AI safe vary enormously.”
I think these other two aspects, importance and neglectedness, just matter a great deal and it would be a bad idea to disqualify cause areas just for moderately weak tractability. In terms of importance, transformative AI seems like it could easily be the most powerful technology we’ve ever made, for roughly the same reasons that humans are the most transformative “technology” on Earth right now. But even if you think this is overrated, consider the relatively meager funds and tiny field as it exists today. I think many people who find the risk a bit out there would at least agree with you that it’s “worth some thought and research”, but because of the rarity of the type of marginal thinking about good and willingness to take weird-sounding ideas seriously found in EA, practically no one else is ensuring that there is some thought and research. The field would, arguably, almost entirely dry up if EA stopped routing resources and people towards it.
Again though, I think maybe some of the disagreement is bound up in the “some risk” idea. My vague impression, and correct me if this doesn’t describe you, is that people who are weirded out by EA working on this as a cause area think that it’s a bit like if EA was getting people, right now, to work on risks from alien invasions (and then a big question is why isn’t it?), whereas people like me who are worried about it think that it is closer to working on risks from alien invasions if NASA discovered an alien spaceship parked five lightyears away from us. The risks here would still be very uncertain, the timelines, what we might be able to do to help, what sorts of things these aliens would be able to or want to do, but I think it would still look crazy if almost no one was looking into it, and I would be very wary of telling one of the only groups that was trying to look into it that they should let someone else handle it.
If you would like I would be happy to chat more about this, either by DMs, or email, or voice/video call. I’m probably not the most qualified person since I’m not in the field, but in a way that might give you a better sense of why the typical EA who is worried about this is. I guess I would like to make this an open invitation for anyone this post resonates with. Feel absolutely no pressure to though, and if you prefer I could just link some resources I think are helpful.
I’m just in the awkward position of both being very worried about this risk, and being very worried about how EA talking about this risk might put potential EAs off. I think it would be a real shame if you felt unwelcome or uncomfortable in the movement because you disagree about this risk, and if there’s something I can do to try to at least persuade you that those of us who are worried are worth sharing the movement with at least, I would like to try to do that.