Taken as intended and not as a question for me (I am personally quite concerned abouts-risks, but think working on them involves similar insights to working on AI x-risks), I think the most common reason is people seeing these scenarios as quite unlikely. Roughly: astronomically bad outcomes are, like astronomically good outcomes, only a very tiny slice of possible outcomes of randomly selected optimization functions, but unlike astronomically good outcomes don’t have existing powerful intelligences trying to aim for them. I think there are reasonable counters to this, but that’s my impression of the most common thought here.
Could you expound on this or maybe point me in the right direction to learn why this might be?
I tend to agree with the intuition that s-risks are unlikely because they are a small part of possibility space and that nobody is really aiming for them. I can see a risk that systems trained to produce eudaimonia will instead produce −1 x eudaimonia, but I can’t see how that justifies thinking that astronomic bad is more likely than astronomic good. Surely a random sign flip is less likely than not.
Taken as intended and not as a question for me (I am personally quite concerned abouts-risks, but think working on them involves similar insights to working on AI x-risks), I think the most common reason is people seeing these scenarios as quite unlikely. Roughly: astronomically bad outcomes are, like astronomically good outcomes, only a very tiny slice of possible outcomes of randomly selected optimization functions, but unlike astronomically good outcomes don’t have existing powerful intelligences trying to aim for them. I think there are reasonable counters to this, but that’s my impression of the most common thought here.
Yeah, that seems likely. Astronomically bad seems much more likely than astronomically good to me though.
Could you expound on this or maybe point me in the right direction to learn why this might be?
I tend to agree with the intuition that s-risks are unlikely because they are a small part of possibility space and that nobody is really aiming for them. I can see a risk that systems trained to produce eudaimonia will instead produce −1 x eudaimonia, but I can’t see how that justifies thinking that astronomic bad is more likely than astronomic good. Surely a random sign flip is less likely than not.