I’m still quite uncertain on my beliefs but I don’t think you got them quite right. Maybe a better summary is that I am generally pessimistic about both humans being ever able to create AGI and especially about humans being able to create safe AGI (it is a special case so it should probably be harder than any AGI). I also think that relying a lot on strong unsafe systems (AI powered or not) can be an x-risk. This is why it is easier to me to understand why AI governance is a way to try to reduce x-risk (at least if actors in the world want to rely on unsafe systems, I don’t know how much this happens but I would not find it very surprising).
I wish I had a better understanding on how x-risk probabilities are estimated (as I said I will try to look into that) but I don’t directly understand why x-risk from AI would be a lot more probable than, say, biorisk (that I don’t understand in detail at all).
Ah, yeah I misread your opinion of the likelihood that humans will ever create AGI. I believe it will happen eventually unless AI research stops due to some exogenous reason (civilizational collapse, a ban on development, etc.). Important assumptions I am making:
General Intelligence is all computation, so it isn’t substrate-dependent
The more powerful an AI is the more economically valuable it is to the creators
Moore’s Law will continue so more computing will be available.
If other approaches fail, we will be able to simulate brains with sufficient compute.
Fully simulated brains will be AGI.
I’m not saying that I think this would be the best, easiest, or only way to create AGI, just that if every other attempt fails, I don’t see what would prevent this from happening. Particularly since we are already to simulate portions of a mouse brain. I am also not claiming here that this implies short timelines for AGI. I don’t have a good estimate of how long this approach would take.
I’m still quite uncertain on my beliefs but I don’t think you got them quite right. Maybe a better summary is that I am generally pessimistic about both humans being ever able to create AGI and especially about humans being able to create safe AGI (it is a special case so it should probably be harder than any AGI). I also think that relying a lot on strong unsafe systems (AI powered or not) can be an x-risk. This is why it is easier to me to understand why AI governance is a way to try to reduce x-risk (at least if actors in the world want to rely on unsafe systems, I don’t know how much this happens but I would not find it very surprising).
I wish I had a better understanding on how x-risk probabilities are estimated (as I said I will try to look into that) but I don’t directly understand why x-risk from AI would be a lot more probable than, say, biorisk (that I don’t understand in detail at all).
Ah, yeah I misread your opinion of the likelihood that humans will ever create AGI. I believe it will happen eventually unless AI research stops due to some exogenous reason (civilizational collapse, a ban on development, etc.). Important assumptions I am making:
General Intelligence is all computation, so it isn’t substrate-dependent
The more powerful an AI is the more economically valuable it is to the creators
Moore’s Law will continue so more computing will be available.
If other approaches fail, we will be able to simulate brains with sufficient compute.
Fully simulated brains will be AGI.
I’m not saying that I think this would be the best, easiest, or only way to create AGI, just that if every other attempt fails, I don’t see what would prevent this from happening. Particularly since we are already to simulate portions of a mouse brain. I am also not claiming here that this implies short timelines for AGI. I don’t have a good estimate of how long this approach would take.