I want to flesh out my impressions about what the crucial considerations for strong longtermist EA priorities are. I think I may want to put some of these into a fuller writeup eventually. I think there’s still a pressing need to evaluate the stronger claims about the value of AI alignment research more, so I wanted to put out my thoughts below about why I currently think the value of a pretty wide range of what I call medium-term interventions rest on those strong claims, especially as I’m pretty personally interested in a lot of more medium-term interventions.
I think there’s a rough consensus among longtermists that (a) we currently live in a time of perils due to anthropogenic x-risk and that (b) most of the moral value of humanity lies in scenarios where we escape the time of perils and that (c) we need to get out of the time of perils by developing and implementing “safety technologies” (broadly construed).
I think that a broad range of potential interventions can be characterized as aiming to broadly increase the ability of humanity to act in the medium term future. I think these include:
Most interventions under the label of “patient philanthropy:”
Global priorities research
Movement building
Saving to give later
Many broad interventions currently considered lower priorities:
Various forms of governance and institutional reform.
Building infrastructure to speed up scientific progress and economic growth.
I think a crucial consideration driving the value of these medium-term interventions relative to more targeted attempts at achieving differential technological progress is whether risk-reductions from safety technologies are diffuse or concentrated.
If we can identify a small number of technologies that have very large expected x-risk reductions, then it makes sense to try to identify them and begin work on them immediately.
On the other hand, if possible x-risk reduction is spread out over a large number of potential safety technologies, then picking out technologies to start working on now becomes a much more difficult and less fruitful task. We are likely better off with broad interventions empowering future people with more information and resources to work on these problems, thus medium-term interventions become much more valuable.
I think our default view should be in favor of x-risk reduction being fairly diffuse across safety technologies.
I think that the case for x-risk reduction being concentrated largely rests on what I want to call the “strong case for technical AGI-alignment research:” (a) the existential risk posed by misaligned AGI is very large and that (b) we can start research alignment technologies now that can permanently eliminate a large portion of that risk.
Eg, if keeping AGI risk low requires constantly coming up with new safety solutions for a long time, then broadly improving our medium-term capabilities might be a lot more useful than going all in on AGI safety research now.
I think we do not currently believe that there’s a small number of discrete safety technologies that can eliminate a large portion of biorisk.
I think that this means that the case for medium-term interventions largely rests on whether or not the strong claim for technical AGI-alignment research is true. I think this means is that if we think the strong case is largely untrue, we should probably be prioritizing
I am generally skeptical of the strong case, but I think there are a lot of uncertainties that could change my mind. I think the main factors affecting my beliefs about the strong case right now are (1) I think the strong case probably requires the Bostrom-Yudkowsky model of AGI development being true (2) I think that the Bostrom-Yudkowsky model is probably false and (3) I think that if the Bostrom-Yudkowsky model were true, technical alignment is probably largely intractable. I don’t have a huge level of confidence in any of these three claims though. (I think I want to flesh out this last point a lot more later).
I want to flesh out my impressions about what the crucial considerations for strong longtermist EA priorities are. I think I may want to put some of these into a fuller writeup eventually. I think there’s still a pressing need to evaluate the stronger claims about the value of AI alignment research more, so I wanted to put out my thoughts below about why I currently think the value of a pretty wide range of what I call medium-term interventions rest on those strong claims, especially as I’m pretty personally interested in a lot of more medium-term interventions.
I think there’s a rough consensus among longtermists that (a) we currently live in a time of perils due to anthropogenic x-risk and that (b) most of the moral value of humanity lies in scenarios where we escape the time of perils and that (c) we need to get out of the time of perils by developing and implementing “safety technologies” (broadly construed).
I think that a broad range of potential interventions can be characterized as aiming to broadly increase the ability of humanity to act in the medium term future. I think these include:
Most interventions under the label of “patient philanthropy:”
Global priorities research
Movement building
Saving to give later
Many broad interventions currently considered lower priorities:
Various forms of governance and institutional reform.
Building infrastructure to speed up scientific progress and economic growth.
I think a crucial consideration driving the value of these medium-term interventions relative to more targeted attempts at achieving differential technological progress is whether risk-reductions from safety technologies are diffuse or concentrated.
If we can identify a small number of technologies that have very large expected x-risk reductions, then it makes sense to try to identify them and begin work on them immediately.
On the other hand, if possible x-risk reduction is spread out over a large number of potential safety technologies, then picking out technologies to start working on now becomes a much more difficult and less fruitful task. We are likely better off with broad interventions empowering future people with more information and resources to work on these problems, thus medium-term interventions become much more valuable.
I think our default view should be in favor of x-risk reduction being fairly diffuse across safety technologies.
I think that the case for x-risk reduction being concentrated largely rests on what I want to call the “strong case for technical AGI-alignment research:” (a) the existential risk posed by misaligned AGI is very large and that (b) we can start research alignment technologies now that can permanently eliminate a large portion of that risk.
Eg, if keeping AGI risk low requires constantly coming up with new safety solutions for a long time, then broadly improving our medium-term capabilities might be a lot more useful than going all in on AGI safety research now.
I think we do not currently believe that there’s a small number of discrete safety technologies that can eliminate a large portion of biorisk.
I think that this means that the case for medium-term interventions largely rests on whether or not the strong claim for technical AGI-alignment research is true. I think this means is that if we think the strong case is largely untrue, we should probably be prioritizing
I am generally skeptical of the strong case, but I think there are a lot of uncertainties that could change my mind. I think the main factors affecting my beliefs about the strong case right now are (1) I think the strong case probably requires the Bostrom-Yudkowsky model of AGI development being true (2) I think that the Bostrom-Yudkowsky model is probably false and (3) I think that if the Bostrom-Yudkowsky model were true, technical alignment is probably largely intractable. I don’t have a huge level of confidence in any of these three claims though. (I think I want to flesh out this last point a lot more later).