I agree with most of this, albeit I have 2 big disagreements with the article:
I think alignment is still important and net-positive, but yeah I’ve come to think it’s no longer the number 1 priority, for the reasons you raise.
2. I think with the exception of biotech and maybe nanotech, no plausible technology in the physical world can actually become a recipe for ruin, unless we are deeply wrong about how the physical laws of the universe work, so we can just defer that question to AI superintelligences.
The basic reason for this is that once you are able to build dyson swarms, the fact that space is big and the speed of light is a huge barrier means it’s very, very easy to close off a network to prevent issues from spreading, and I think that conditional on building aligned ASI, dyson swarms are likely to be built within 100 years.
And even nanotech has been argued to be way less powerful than people think it is, and @Muireall has argued against nanotech being powerful here:
I agree with most of this, albeit I have 2 big disagreements with the article:
I think alignment is still important and net-positive, but yeah I’ve come to think it’s no longer the number 1 priority, for the reasons you raise.
2. I think with the exception of biotech and maybe nanotech, no plausible technology in the physical world can actually become a recipe for ruin, unless we are deeply wrong about how the physical laws of the universe work, so we can just defer that question to AI superintelligences.
The basic reason for this is that once you are able to build dyson swarms, the fact that space is big and the speed of light is a huge barrier means it’s very, very easy to close off a network to prevent issues from spreading, and I think that conditional on building aligned ASI, dyson swarms are likely to be built within 100 years.
And even nanotech has been argued to be way less powerful than people think it is, and @Muireall has argued against nanotech being powerful here:
https://muireall.space/pdf/considerations.pdf#page=17
https://forum.effectivealtruism.org/posts/oqBJk2Ae3RBegtFfn/my-thoughts-on-nanotechnology-strategy-research-as-an-ea?commentId=WQn4nEH24oFuY7pZy
https://muireall.space/nanosystems/