Thanks so much for writing this Will, I especially like the ideas:
It is much more clear now than it was 10 years ago that AI will be a major issue of our time, affecting many aspects of our world (and our future). So it isn’t just relevant as a cause, but instead as something that affects how we pursue many causes, including things like global health, global development, pandemics, animal welfare etc.
Previously EA work on AI was tightly focused around technical safety work, but expansion of this to include governance work has been successful and we will need to further expand it, such that there are multiple distinct AI areas of focus within EA.
If you’ve got a very high probability of AI takeover (obligatory reference!), then my first two arguments, at least, might seem very weak because essentially the only thing that matters is reducing the risk of AI takeover.
I’m not even sure your arguments would be weak in that scenario.
e.g. if there were a 90% chance we fall at the first hurdle with an unaligned AI taking over, but also a 90% chance that even if we avoid this, we fall at the second hurdle with a post-AGI world that squanders most of the value of the future, then this would be symmetrical between the problems (it doesn’t matter formally which one comes a little earlier). In this case we’d only have a 1% chance of a future that is close to the best we could achieve. Completely solving either problem would increase that to 10%. Halving the chance of the bad outcome for both of them would instead increase that to 30.25% (and would probably be easier than completely solving one). So in this case there would be active reason to work on both at once (even if work on each had a linear effect on its probability).
One needs to add substantive additional assumptions on top of the very high probability of AI takeover to get it to argue against allocating some substantial effort to ensuring that even with aligned AI things go well. e.g. that if AI doesn’t takeover, it solves our other problems or that the chance of near-best futures is already very high, etc.
I’m not even sure your arguments would be weak in that scenario.
Thanks—classic Toby point! I agree entirely that you need additional assumptions.
I was imagining someone who thinks that, say, there’s a 90% risk of unaligned AI takeover, and a 50% loss of EV of the future from other non-alignment issues that we can influence. So EV of the future is 5%.
If so, completely solving AI risk would increase the EV of the future to 50%; halving both would increase it only to 41%.
But, even so, it’s probably easier to halve both than to completely eliminate AI takeover risk, and more generally the case for a mixed strategy seems strong.
I was imagining someone who thinks that, say, there’s a 90% risk of unaligned AI takeover, and a 50% loss of EV of the future from other non-alignment issues that we can influence. So EV of the future is 25%.
I’m not understanding—if there’s no value in the 90%, and then 50% value in the remaining 10%, wouldn’t the EV of the future be 5%?
Thanks so much for writing this Will, I especially like the ideas:
It is much more clear now than it was 10 years ago that AI will be a major issue of our time, affecting many aspects of our world (and our future). So it isn’t just relevant as a cause, but instead as something that affects how we pursue many causes, including things like global health, global development, pandemics, animal welfare etc.
Previously EA work on AI was tightly focused around technical safety work, but expansion of this to include governance work has been successful and we will need to further expand it, such that there are multiple distinct AI areas of focus within EA.
I’m not even sure your arguments would be weak in that scenario.
e.g. if there were a 90% chance we fall at the first hurdle with an unaligned AI taking over, but also a 90% chance that even if we avoid this, we fall at the second hurdle with a post-AGI world that squanders most of the value of the future, then this would be symmetrical between the problems (it doesn’t matter formally which one comes a little earlier). In this case we’d only have a 1% chance of a future that is close to the best we could achieve. Completely solving either problem would increase that to 10%. Halving the chance of the bad outcome for both of them would instead increase that to 30.25% (and would probably be easier than completely solving one). So in this case there would be active reason to work on both at once (even if work on each had a linear effect on its probability).
One needs to add substantive additional assumptions on top of the very high probability of AI takeover to get it to argue against allocating some substantial effort to ensuring that even with aligned AI things go well. e.g. that if AI doesn’t takeover, it solves our other problems or that the chance of near-best futures is already very high, etc.
Thanks—classic Toby point! I agree entirely that you need additional assumptions.
I was imagining someone who thinks that, say, there’s a 90% risk of unaligned AI takeover, and a 50% loss of EV of the future from other non-alignment issues that we can influence. So EV of the future is 5%.
If so, completely solving AI risk would increase the EV of the future to 50%; halving both would increase it only to 41%.
But, even so, it’s probably easier to halve both than to completely eliminate AI takeover risk, and more generally the case for a mixed strategy seems strong.
I’m not understanding—if there’s no value in the 90%, and then 50% value in the remaining 10%, wouldn’t the EV of the future be 5%?
Argh, thanks for catching that! Edited now.