I’m going to attempt to summarize what I think part of your current beliefs are (please correct me if I am wrong!)
Current ML techniques are not sufficient to develop AGI
But someday humans will be able to create AGI
It is possible (likely?) that it will be difficult to ensure that the AGI is safe
It is possible that humans will give enough control to an unsafe AGI that it is an X risk.
If I got that right I would describe that as both having (appropriately loosely held) beliefs about AI Safety and agreement that AI Safety is a risk with some unspecified probability and magnitude.
What you don’t have a view on, but you believe people in AI safety do have strong views on is (again not trying to put words in your mouth just my best attempt at understanding):
Is AI safety actually possible?
What work would be useful to increase AI Safety if that is possible?
How important is AI safety compared to other cause areas?
My (fairly uninformed view) is that people working on AI safety don’t know the answer to that first or second question. Rather, they think that the probability and magnitude of the problem are high enough that it swamps those questions in calculating the importance of the cause area. Some of these people have tried to model out this reasoning, while others are leaning more on intuition. I think reducing the uncertainty of any of these three questions is useful in itself, so I think it would be great if you wanted to work on that.
I’m still quite uncertain on my beliefs but I don’t think you got them quite right. Maybe a better summary is that I am generally pessimistic about both humans being ever able to create AGI and especially about humans being able to create safe AGI (it is a special case so it should probably be harder than any AGI). I also think that relying a lot on strong unsafe systems (AI powered or not) can be an x-risk. This is why it is easier to me to understand why AI governance is a way to try to reduce x-risk (at least if actors in the world want to rely on unsafe systems, I don’t know how much this happens but I would not find it very surprising).
I wish I had a better understanding on how x-risk probabilities are estimated (as I said I will try to look into that) but I don’t directly understand why x-risk from AI would be a lot more probable than, say, biorisk (that I don’t understand in detail at all).
Ah, yeah I misread your opinion of the likelihood that humans will ever create AGI. I believe it will happen eventually unless AI research stops due to some exogenous reason (civilizational collapse, a ban on development, etc.). Important assumptions I am making:
General Intelligence is all computation, so it isn’t substrate-dependent
The more powerful an AI is the more economically valuable it is to the creators
Moore’s Law will continue so more computing will be available.
If other approaches fail, we will be able to simulate brains with sufficient compute.
Fully simulated brains will be AGI.
I’m not saying that I think this would be the best, easiest, or only way to create AGI, just that if every other attempt fails, I don’t see what would prevent this from happening. Particularly since we are already to simulate portions of a mouse brain. I am also not claiming here that this implies short timelines for AGI. I don’t have a good estimate of how long this approach would take.
I’m going to attempt to summarize what I think part of your current beliefs are (please correct me if I am wrong!)
Current ML techniques are not sufficient to develop AGI
But someday humans will be able to create AGI
It is possible (likely?) that it will be difficult to ensure that the AGI is safe
It is possible that humans will give enough control to an unsafe AGI that it is an X risk.
If I got that right I would describe that as both having (appropriately loosely held) beliefs about AI Safety and agreement that AI Safety is a risk with some unspecified probability and magnitude.
What you don’t have a view on, but you believe people in AI safety do have strong views on is (again not trying to put words in your mouth just my best attempt at understanding):
Is AI safety actually possible?
What work would be useful to increase AI Safety if that is possible?
How important is AI safety compared to other cause areas?
My (fairly uninformed view) is that people working on AI safety don’t know the answer to that first or second question. Rather, they think that the probability and magnitude of the problem are high enough that it swamps those questions in calculating the importance of the cause area. Some of these people have tried to model out this reasoning, while others are leaning more on intuition. I think reducing the uncertainty of any of these three questions is useful in itself, so I think it would be great if you wanted to work on that.
I’m still quite uncertain on my beliefs but I don’t think you got them quite right. Maybe a better summary is that I am generally pessimistic about both humans being ever able to create AGI and especially about humans being able to create safe AGI (it is a special case so it should probably be harder than any AGI). I also think that relying a lot on strong unsafe systems (AI powered or not) can be an x-risk. This is why it is easier to me to understand why AI governance is a way to try to reduce x-risk (at least if actors in the world want to rely on unsafe systems, I don’t know how much this happens but I would not find it very surprising).
I wish I had a better understanding on how x-risk probabilities are estimated (as I said I will try to look into that) but I don’t directly understand why x-risk from AI would be a lot more probable than, say, biorisk (that I don’t understand in detail at all).
Ah, yeah I misread your opinion of the likelihood that humans will ever create AGI. I believe it will happen eventually unless AI research stops due to some exogenous reason (civilizational collapse, a ban on development, etc.). Important assumptions I am making:
General Intelligence is all computation, so it isn’t substrate-dependent
The more powerful an AI is the more economically valuable it is to the creators
Moore’s Law will continue so more computing will be available.
If other approaches fail, we will be able to simulate brains with sufficient compute.
Fully simulated brains will be AGI.
I’m not saying that I think this would be the best, easiest, or only way to create AGI, just that if every other attempt fails, I don’t see what would prevent this from happening. Particularly since we are already to simulate portions of a mouse brain. I am also not claiming here that this implies short timelines for AGI. I don’t have a good estimate of how long this approach would take.