I feel on board with essentially everything in this article. I’m pretty confused by the popularity of the Long Reflection idea—it seems utterly impractical and unrealistic without major changes to how humans act, for eg the reasons outlined here. I feel like I must be misunderstanding or missing something?
One argument for the long reflection that I think has been missed in a lot of this discussion is that it’s a proposal for taking Nick’s Astronomical Waste argument (AWA) seriously. Nick argues that it’s worth spending millennia to reduce existential risk by a couple percent. But launching for example, a superintelligence with the values of humanity in 2025 could itself constitute an existential risk, in light of future human values. So AWA implies that a sufficiently wise and capable society would be prepared to wait millennia before jumping in to such an action.
Now we may practically never be capable enough to coordinate to do so, but the theory makes sense.
Just spitballing, but I spontaneously don’t find it completely unrealistic. Would a decades long moratorium on transformative AI count as a long reflection, which is prolonged until some consensus forms among world governments? That currently seems unthinkable, but if the right people in the US and Chinese governments would be convinced, and we would have lived through a global catastrophe due to misaligned AI systems, I wouldn’t be surprised to see a global coordination to control advances in AI and scaling up alignment and ethical reflection research, research which would have significant sway over the future of humanity over decades.
I’m spontaneosly also not convinced that this moratorium would pose so significant costs on many relevant actors. Progress would continue, just slower. I.e. we would still eradicate poverty in the coming decades, make significant progress on fighting most major diseases, maybe even have just-not-quite transformative personal AI assistants, etc. Right?
I largely agree with Neel, and fwiw to me there isn’t really ‘an alternative’. Humanity, if it continues to exist as something resembling autonomous individuals, is going to bumble along as it always has and grand proposals to socially engineer it seem unlikely to work and incredibly dangerous if they do.
On the other hand, developing a social movement of concerned individuals with realistic goals that people new to it can empathise with, so that over time their concerns start to become mainstream, seems like a good marginal way of nudging long term experiences to be more positive.
I think “Humanity is going to bumble along as it always has” is not a realistic alternative; the Long Reflection is motivated by the worry that that won’t happen by default. Instead, we’ll all die, or end up in one of the various dystopian scenarios people talk about, e.g. the hardscrapple frontier, the disneyland with no children, some of the darker Age of Em stuff… (I could elaborate if you like). If we want humanity to continue bumbling on, we need to do something to make that happen, and the Long Reflection is a proposal for how to do that.
Well hence the caveat. It may not continue to exist, and encouraging it to do so seems valuable.
But the timescale Toby gave for the long reflection seems to take us to well over to the far side of most foreseeable x-risks, meaning a) it won’t have helped us solve them, but rather will only be possible as a consequence of having done so and b) it might well exacerbate them if it turns out that the majority of risks are local, and we’ve forced ourselves to sit in one spot contemplating them rather than spread out to other star systems.
Hanson’s concern seems to be an extension of b), where it ends up causing them directly, which also seems plausible.
Btw, I object to using flowery jargon like ‘the hardscrapple frontier, the disneyland with no children’ that map to easily expressible concepts like ‘subsistence living’ and ‘the extinction of consciousness’. It seems like virtue signalling at the expense of communication.
I strongly agree with the general sentiment about jargon and flowery language, though I think “disneyland with no children” is not equivalent to “extinction of consciousness” because (1) Bostrom wants to remain non-committal about the question of which things constitute a person’s welfare and how these things relate to consciousness and (2) he is focused on cases in which, from the outside, it appears that people are enjoying very high welfare levels, when in fact they do not experience any welfare at all.
Ok, but if you were optimising for communicating that concept, is ‘Disneyland with no children’ really the phrase you’d use? You could spell it out in full or come up with a more literal pithy phrase.
Hmmm, point taken. I do think this particular case was intended to serve, and does serve, a communicative purpose: If I just said “subsistence living or the extinction of consciousness” then you wouldn’t have keywords to search for, whereas instead by giving these scenarios the names their authors chose, you can easily go read about them. I guess I didn’t think things through enough; after all, it’s annoying to have to go look things up and by name-dropping the scenarios I force you do do that. My apologies!
It seems unlikely but not impossible given how strong status quo bias is among humans. NIMBY movement, reactionary and conservative politics in general, lots of examples of politics that call for less or no change.
Humans have had periods of tens or hundreds of thousands of years where we stagnate and technology doesn’t seem to change much, as far as we can tell from the archaeological record, so this isn’t unprecedented.
I feel on board with essentially everything in this article. I’m pretty confused by the popularity of the Long Reflection idea—it seems utterly impractical and unrealistic without major changes to how humans act, for eg the reasons outlined here. I feel like I must be misunderstanding or missing something?
One argument for the long reflection that I think has been missed in a lot of this discussion is that it’s a proposal for taking Nick’s Astronomical Waste argument (AWA) seriously. Nick argues that it’s worth spending millennia to reduce existential risk by a couple percent. But launching for example, a superintelligence with the values of humanity in 2025 could itself constitute an existential risk, in light of future human values. So AWA implies that a sufficiently wise and capable society would be prepared to wait millennia before jumping in to such an action.
Now we may practically never be capable enough to coordinate to do so, but the theory makes sense.
Just spitballing, but I spontaneously don’t find it completely unrealistic. Would a decades long moratorium on transformative AI count as a long reflection, which is prolonged until some consensus forms among world governments? That currently seems unthinkable, but if the right people in the US and Chinese governments would be convinced, and we would have lived through a global catastrophe due to misaligned AI systems, I wouldn’t be surprised to see a global coordination to control advances in AI and scaling up alignment and ethical reflection research, research which would have significant sway over the future of humanity over decades.
I’m spontaneosly also not convinced that this moratorium would pose so significant costs on many relevant actors. Progress would continue, just slower. I.e. we would still eradicate poverty in the coming decades, make significant progress on fighting most major diseases, maybe even have just-not-quite transformative personal AI assistants, etc. Right?
What’s your preferred alternative?
I largely agree with Neel, and fwiw to me there isn’t really ‘an alternative’. Humanity, if it continues to exist as something resembling autonomous individuals, is going to bumble along as it always has and grand proposals to socially engineer it seem unlikely to work and incredibly dangerous if they do.
On the other hand, developing a social movement of concerned individuals with realistic goals that people new to it can empathise with, so that over time their concerns start to become mainstream, seems like a good marginal way of nudging long term experiences to be more positive.
I think “Humanity is going to bumble along as it always has” is not a realistic alternative; the Long Reflection is motivated by the worry that that won’t happen by default. Instead, we’ll all die, or end up in one of the various dystopian scenarios people talk about, e.g. the hardscrapple frontier, the disneyland with no children, some of the darker Age of Em stuff… (I could elaborate if you like). If we want humanity to continue bumbling on, we need to do something to make that happen, and the Long Reflection is a proposal for how to do that.
Well hence the caveat. It may not continue to exist, and encouraging it to do so seems valuable.
But the timescale Toby gave for the long reflection seems to take us to well over to the far side of most foreseeable x-risks, meaning a) it won’t have helped us solve them, but rather will only be possible as a consequence of having done so and b) it might well exacerbate them if it turns out that the majority of risks are local, and we’ve forced ourselves to sit in one spot contemplating them rather than spread out to other star systems.
Hanson’s concern seems to be an extension of b), where it ends up causing them directly, which also seems plausible.
Btw, I object to using flowery jargon like ‘the hardscrapple frontier, the disneyland with no children’ that map to easily expressible concepts like ‘subsistence living’ and ‘the extinction of consciousness’. It seems like virtue signalling at the expense of communication.
I strongly agree with the general sentiment about jargon and flowery language, though I think “disneyland with no children” is not equivalent to “extinction of consciousness” because (1) Bostrom wants to remain non-committal about the question of which things constitute a person’s welfare and how these things relate to consciousness and (2) he is focused on cases in which, from the outside, it appears that people are enjoying very high welfare levels, when in fact they do not experience any welfare at all.
Ok, but if you were optimising for communicating that concept, is ‘Disneyland with no children’ really the phrase you’d use? You could spell it out in full or come up with a more literal pithy phrase.
Sorry, what virtue do you think is being signaled here?
Mainly EA ingroupiness.
Hmmm, point taken. I do think this particular case was intended to serve, and does serve, a communicative purpose: If I just said “subsistence living or the extinction of consciousness” then you wouldn’t have keywords to search for, whereas instead by giving these scenarios the names their authors chose, you can easily go read about them. I guess I didn’t think things through enough; after all, it’s annoying to have to go look things up and by name-dropping the scenarios I force you do do that. My apologies!
I appreciated the link to the hardscrapple frontier, which I had not heard of, FWIW.
It seems unlikely but not impossible given how strong status quo bias is among humans. NIMBY movement, reactionary and conservative politics in general, lots of examples of politics that call for less or no change.
Humans have had periods of tens or hundreds of thousands of years where we stagnate and technology doesn’t seem to change much, as far as we can tell from the archaeological record, so this isn’t unprecedented.