I think this argument is pretty wrong for a few reasons:
It generalizes way too far… for example, you could say “Before trying to shape the far future, why don’t we solve [insert other big problem]? Isn’t the fact that we haven’t solved [other big problem] bad news about our ability to shape the far future positively?” Of course, our prospects would look more impressive if we had solved many other big problems. But I think it’s an unfair and unhelpful test to pick a specific big problem, notice that we haven’t solved it, and infer that we need to solve it first.
Many, if not most, longtermists believe we’re living near a hinge of history and might have very little time remaining to try to influence it. Waiting until we first ended factory farming would inherently forgo a huge fraction of the time remaining on those views to make a difference.
You say “It is a stirring vision, but it rests on a fragile assumption: that humanity is capable of aligning on a mission, coordinating across cultures and centuries, and acting with compassion at scale.” but that’s not true/exactly; I don’t think longtermism rests on the assuption that the best thing to do is try to directly cause that right now (see the hinge of history link above). For example, I’m not sure how we would end factory farming, but it might require, as you allude to, massive global coordination. In contrast, creating techniques to align AIs might require only a relatively small group of researchers, and a small group of AI companies adopting research that is in their best interests to use. To be clear, there are longtermist-relevant interventions that might also require global and widespread coordination, but they don’t all require it (and the ones I’m most optimistic about don’t require it, because global coordination is very difficult).
Related to the above, the problems are just different, and require different skills and resources (and shaping the far future isn’t necessarily harder than ending factory farming; for example, I wouldn’t be surprised if cutting bio x-risk in half ends up being much easier than ending factory farming). Succeeding at one is unlikely to be the best practice for succeeding at the other.
(I think factory farming is a moral abomination of gigantic proportions, I feel deep gratitude for people who are trying to end it, and dearly hope they succeed.)
Many, if not most, longtermists believe we’re living near a hinge of history
Right but this requires believing the future will be better if humans survive. I take ops point as saying she doesn’t agree or is at least skeptical.
and a small group of AI companies adopting research that is in their best interests to use.
I think again, the point of OP is trying to make is we have very little proof of concept of getting people to go against their best interests. And so if doing what’s right isn’t in the ai companies best interest op wouldn’t believe we can get them to do what we think they should.
Right but this requires believing the future will be better if humans survive. I take ops point as saying she doesn’t agree or is at least skeptical
I think the post isn’t clear between the stances “it would make the far future better to end factory farming now” and “the only path by which the far future is net positive requires ending factory farming”, or generally how much of the claim that we should try to end factory farming now is motivated by the “if we can’t do that, we shouldn’t attempt to do longtermist interventions because they will probably fail” vs. “if we can’t do that, we shouldn’t attempt to do longtermist interventions because they are less valuable becuase the EV of the future is worse”
Anyway, working to cause humans to survive requires (or at least, is probably motivated by) thinking the future will be better that way. Not all longtermism is about that (see e.g. s-risk mitigation), and those parts are also relevant to the hinge of history question.
I think again, the point of OP is trying to make is we have very little proof of concept of getting people to go against their best interests. And so if doing what’s right isn’t in the ai companies best interest op wouldn’t believe we can get them to do what we think they should.
I am saying aligning AI is in the best interests of AI companies, unlike the situation with ending factory farming and animal ag companies, which is a relevant difference. Any AI company that could align their AIs once and for all for $10M would do it in a heartbeat. I don’t think they will do nearly enough to align their AIs (so in that sense, their incentives are not humanity’s incentives), given the stakes, but they do want to at least a little
Yea my original framing was a little confused wrt the “vs” dichotomy you present in paragraph one, good shout. I guess I actually meant a little bit of each, though. My interpretation of the post is basically, (1) in so forth as we need to defeat powerful people or thought patterns we (ea or humans) haven’t proven it (2) it’s somewhat likely we will need to do this to create the world we want.
I.e. Given that future s-risk efforts are probably not going to be successful, current extinction-risk efforts are therefore also less useful.
I am saying aligning AI is in the best interests of AI companies
If you define it in a specifically narrow AI Takeover way yes. Making sure it doesn’t allow a dictator to take power or gradually disempowerment scenarios, not really. Or to the extent that ensuring alignment requires slowing down progress.
Anyway mostly in agreement with your points/world, I definitely think we should be focusing on AI right now and I think that our goals and the AI companies/US gov are sufficiently aligned atm that we aren’t swimming up stream, but I resonante with OP that it would allievate some concerns if we actually racked up some hard fought politically unpopular battles before trying to steer the whole future.
It certainly seems possible (>1%) that in the next 2 US admins (current plus next) AI safety becomes so toxic that all the EA -adj ai safety people in the gov get purged and they stop listeing to most ai safety researchers. If this co-occurs with some sort of AI nationalization most of our TOC is cooked.
I think this argument is pretty wrong for a few reasons:
It generalizes way too far… for example, you could say “Before trying to shape the far future, why don’t we solve [insert other big problem]? Isn’t the fact that we haven’t solved [other big problem] bad news about our ability to shape the far future positively?” Of course, our prospects would look more impressive if we had solved many other big problems. But I think it’s an unfair and unhelpful test to pick a specific big problem, notice that we haven’t solved it, and infer that we need to solve it first.
Many, if not most, longtermists believe we’re living near a hinge of history and might have very little time remaining to try to influence it. Waiting until we first ended factory farming would inherently forgo a huge fraction of the time remaining on those views to make a difference.
You say “It is a stirring vision, but it rests on a fragile assumption: that humanity is capable of aligning on a mission, coordinating across cultures and centuries, and acting with compassion at scale.” but that’s not true/exactly; I don’t think longtermism rests on the assuption that the best thing to do is try to directly cause that right now (see the hinge of history link above). For example, I’m not sure how we would end factory farming, but it might require, as you allude to, massive global coordination. In contrast, creating techniques to align AIs might require only a relatively small group of researchers, and a small group of AI companies adopting research that is in their best interests to use. To be clear, there are longtermist-relevant interventions that might also require global and widespread coordination, but they don’t all require it (and the ones I’m most optimistic about don’t require it, because global coordination is very difficult).
Related to the above, the problems are just different, and require different skills and resources (and shaping the far future isn’t necessarily harder than ending factory farming; for example, I wouldn’t be surprised if cutting bio x-risk in half ends up being much easier than ending factory farming). Succeeding at one is unlikely to be the best practice for succeeding at the other.
(I think factory farming is a moral abomination of gigantic proportions, I feel deep gratitude for people who are trying to end it, and dearly hope they succeed.)
Right but this requires believing the future will be better if humans survive. I take ops point as saying she doesn’t agree or is at least skeptical.
I think again, the point of OP is trying to make is we have very little proof of concept of getting people to go against their best interests. And so if doing what’s right isn’t in the ai companies best interest op wouldn’t believe we can get them to do what we think they should.
I think the post isn’t clear between the stances “it would make the far future better to end factory farming now” and “the only path by which the far future is net positive requires ending factory farming”, or generally how much of the claim that we should try to end factory farming now is motivated by the “if we can’t do that, we shouldn’t attempt to do longtermist interventions because they will probably fail” vs. “if we can’t do that, we shouldn’t attempt to do longtermist interventions because they are less valuable becuase the EV of the future is worse”
Anyway, working to cause humans to survive requires (or at least, is probably motivated by) thinking the future will be better that way. Not all longtermism is about that (see e.g. s-risk mitigation), and those parts are also relevant to the hinge of history question.
I am saying aligning AI is in the best interests of AI companies, unlike the situation with ending factory farming and animal ag companies, which is a relevant difference. Any AI company that could align their AIs once and for all for $10M would do it in a heartbeat. I don’t think they will do nearly enough to align their AIs (so in that sense, their incentives are not humanity’s incentives), given the stakes, but they do want to at least a little
Yea my original framing was a little confused wrt the “vs” dichotomy you present in paragraph one, good shout. I guess I actually meant a little bit of each, though. My interpretation of the post is basically, (1) in so forth as we need to defeat powerful people or thought patterns we (ea or humans) haven’t proven it (2) it’s somewhat likely we will need to do this to create the world we want.
I.e. Given that future s-risk efforts are probably not going to be successful, current extinction-risk efforts are therefore also less useful.
If you define it in a specifically narrow AI Takeover way yes. Making sure it doesn’t allow a dictator to take power or gradually disempowerment scenarios, not really. Or to the extent that ensuring alignment requires slowing down progress.
Anyway mostly in agreement with your points/world, I definitely think we should be focusing on AI right now and I think that our goals and the AI companies/US gov are sufficiently aligned atm that we aren’t swimming up stream, but I resonante with OP that it would allievate some concerns if we actually racked up some hard fought politically unpopular battles before trying to steer the whole future.
It certainly seems possible (>1%) that in the next 2 US admins (current plus next) AI safety becomes so toxic that all the EA -adj ai safety people in the gov get purged and they stop listeing to most ai safety researchers. If this co-occurs with some sort of AI nationalization most of our TOC is cooked.