Thanks for writing this post! It is important to think about the implications of cluelessness and moral uncertainty for AI safety. To clarify the value of working on AI safety, it helps to decompose the problem into two subquestions:
Is the outcome that we are aiming for robust to cluelessness, and moral uncertainty?
Do we know of an intervention that is robustly good to achieve that outcome? (i.e. an intervention that is at least better than doing nothing to achieve that outcome)
An outcome could be reducing X-risks from AI which could at least happen in two different ways: value lock-in from a human-aligned AI or from a non-human AI controlled future. Reducing value lock-in seems robustly good, and i won’t argue for that here.
If the outcome we are thinking about is reducing extinction from AI, then the near-term case from reducing extinction from AI seems more robust to cluelessness, and I feel that the post could have emphasised it a bit more. Indeed, reducing the risk of extinction from AI, for all the people alive today and in the next few generations, looks good from a range of moral perspectives (it is at least determinate good for humans) even though it is indeterminate in the long-term. But then, one has to compare short term AI X-risks with other interventions that makes things determinately good in the short term from an impartial perspective, like working on animal welfare, or reducing extreme poverty.
AI seems high stakes, even though we don’t know which way it will go in the short/long term, which might suggest focusing more on capacity building instead of a more direct intervention (I would put capacity building in some of the paths that you suggested, as a more general category to put careers like earning to give). This could hold as long as the capacity building (for ex. putting ourselves in a position to make things go well w.r.t AI when we have more information) has low risk of backfiring (don’t make things worse) that is.
If we grant that reducing X-risks from AI seems robustly good, and better than alternative short-term causes (which is a higher bar than ``better than doing nothing″), then we still need to figure out interventions that are robust to reduce X-risks from AI (i.e. so that we don’t make things worse). I already mentioned some non-backfire capacity building (if we find such kind of capacity building). Beyond capacity building; it’s not completely clear to me that there are robustly good interventions in AI safety, and I think more work is needed to prioritize interventions.
It seems useful to think of one’s career as being part of a portfolio, and work on things where one could plausibly be in a position to do excellent work, unless the intervention that one is working on is not determinately better than doing nothing.
Thanks for writing this post! It is important to think about the implications of cluelessness and moral uncertainty for AI safety. To clarify the value of working on AI safety, it helps to decompose the problem into two subquestions:
Is the outcome that we are aiming for robust to cluelessness, and moral uncertainty?
Do we know of an intervention that is robustly good to achieve that outcome? (i.e. an intervention that is at least better than doing nothing to achieve that outcome)
An outcome could be reducing X-risks from AI which could at least happen in two different ways: value lock-in from a human-aligned AI or from a non-human AI controlled future. Reducing value lock-in seems robustly good, and i won’t argue for that here.
If the outcome we are thinking about is reducing extinction from AI, then the near-term case from reducing extinction from AI seems more robust to cluelessness, and I feel that the post could have emphasised it a bit more. Indeed, reducing the risk of extinction from AI, for all the people alive today and in the next few generations, looks good from a range of moral perspectives (it is at least determinate good for humans) even though it is indeterminate in the long-term. But then, one has to compare short term AI X-risks with other interventions that makes things determinately good in the short term from an impartial perspective, like working on animal welfare, or reducing extreme poverty.
AI seems high stakes, even though we don’t know which way it will go in the short/long term, which might suggest focusing more on capacity building instead of a more direct intervention (I would put capacity building in some of the paths that you suggested, as a more general category to put careers like earning to give). This could hold as long as the capacity building (for ex. putting ourselves in a position to make things go well w.r.t AI when we have more information) has low risk of backfiring (don’t make things worse) that is.
If we grant that reducing X-risks from AI seems robustly good, and better than alternative short-term causes (which is a higher bar than ``better than doing nothing″), then we still need to figure out interventions that are robust to reduce X-risks from AI (i.e. so that we don’t make things worse). I already mentioned some non-backfire capacity building (if we find such kind of capacity building). Beyond capacity building; it’s not completely clear to me that there are robustly good interventions in AI safety, and I think more work is needed to prioritize interventions.
It seems useful to think of one’s career as being part of a portfolio, and work on things where one could plausibly be in a position to do excellent work, unless the intervention that one is working on is not determinately better than doing nothing.