The Michael Neilsen critique seems thoughtful, constructive, and well-balanced on first read, but I have some serious reservations about the underlying ethos and its implications.
Look, any compelling new world-view that is outside the mainstream cultures’ Overton window can be pathologized as an information hazard that makes its believers feel unhappy, inadequate, and even mentally ill by mainstream standards. Nielsen seems to view ‘strong EA’ as that kind of information hazard, and critiques it as such.
Trouble is, if you understand that most normies are delusional about some important issue, and you you develop some genuinely deeper insights into that issue, the psychologically predictable result is some degree of alienation and frustration. This is true for everyone who has a religious conversion experience. It’s true for everyone who really takes onboard the implications of any intellectually compelling science—whether cosmology, evolutionary biology, neuroscience, signaling theory, game theory, behavior genetics, etc. It’s true for everyone who learns about any branch of moral philosophy and takes it seriously as a guide to action.
I’ve seen this over, and over, and over in my own field of evolutionary psychology. The usual ‘character arc’ of ev psych insight is that (1) you read Dawkins or Pinker or Buss, you get filled with curiosity about the origins of human nature, (2) you learn some more and you feel overwhelming intellectual awe and excitement about the grandeur of evolutionary theory, (3) you gradually come to understand that every human perception, preference, value, desire, emotion, and motivation has deep evolutionary roots beyond your control, and you start to feel uneasy, (4) you ruminate about how you’re nothing but an evolved robot chasing reproductive success through adaptively self-deceived channels, and you feel some personal despair, (5) you look around at a society full of other self-deceived humans unaware of their biological programming, and you feel black-pilled civilizational despair, (6) you live with the Darwinian nihilism for a few years, adapt to the new normal, and gradually find some way to live with the new insights, climbing your way back into some semblance of normie-adjacent happiness. I’ve seen these six phases many times in my own colleagues, grad students, and collaborators.
And that’s just with a new descriptive world-view about how the human world works. EA’s challenge can be even more profound, because it’s not just descriptive, but normative, or at least prescriptive. So there’s a painful gap between what we could be doing, and what we are doing. And so there should be, if you take the world in a morally serious way.
I think the deeper problem is that given 20th century history, there’s a general dubiousness about any group of people who do take the world in a morally serious way that deviates from the usual forms of mild political virtue signaling encouraged in our current system of credentialism, careerism, and consumerism.
I think even on EA’s own terms (apart from any effects from EA being fringe) there’s a good reason for EAs to be OK with being more stressed and unhappy than people with other philosophies.
On the scale of human history we’re likely in an emergency situation when we have an opportunity to trade off the happiness of EAs for enormous gains in total well-being. Similar to how during a bear attack you’d accept that you won’t feel relaxed and happy while you try to mitigate the attack, but this period of stress is worth it overall. This is especially true if you believe we’re in the hinge of history.
Eh this logic can be used to justify a lot of extreme action in the name of progress. Communists and Marxists have had a lot of thoughts about the “hinge of history” and used that to unleash terrible destruction on the rest of humanity.
In contrast to a bear attack, you don’t expect to know that the “period of stress” has ended during your lifetime. Which raises a few questions, like “Is it worth it?” and “How sure can we be that this really is a stress period?”. The thought that we especially are in a position to trade our happiness for enormous gains for society—while not impossible—is dangerous in that it’s very appealing, regardless whether it’s true or not.
The thought that we especially are in a position to trade our happiness for enormous gains for society [...] is dangerous in that it’s very appealing,
I’m not denying that what you say is true, but on the face of it, “the appeal of this ideology is that you have to sacrifice a lot for others’ gain” is not an intuitively compelling message.
In contrast to a bear attack, you don’t expect to know that the “period of stress” has ended during your lifetime.
I expect to know this. Either AI will go well and we’ll get the glorious transhuman future, or it’ll go poorly and we’ll have a brief moment of realization before we are killed etc. (or more realistically, a longer moment of awareness where we realize all is truly and thoroughly lost, before eventually the nanobots orwhatever come for us).
By many numbers AI risk being solved would only reduce total probability of X-risk by 1⁄3, 2⁄3, or maybe 9⁄10 if you are very heavy on AI-risk probability.
Personally I think humanity’s “period of stress” will take at least 1000s of years to be solved but I might be being quite pessimistic. Of course situations will get better but I think world will still be “burning” for quite some time.
Here’s a common belief in these circles (which I share):
If AI risk is solved through means other than “we collectively coordinate to not build TAI”(a solution which I think is unlikely both because that level of global coordination is very hard and because the opportunity costs are massive), then soon after, whether human civilization flourishes or not is mostly a question that’s out of human hands.
This is completely unrelated to the great point you made with the comment but I felt I had to share a classic? EA tip that worked well for me. (uncertain how much this counts as a classic.) I got to the nice nihilistic bottom of realising that my moral system is essentially based on evolution but I reversed that within a year by reading a bunch of Buddhist philosophy and by meditating. Now it’s all nirvana over here! (try it out now...)
The Michael Neilsen critique seems thoughtful, constructive, and well-balanced on first read, but I have some serious reservations about the underlying ethos and its implications.
Look, any compelling new world-view that is outside the mainstream cultures’ Overton window can be pathologized as an information hazard that makes its believers feel unhappy, inadequate, and even mentally ill by mainstream standards. Nielsen seems to view ‘strong EA’ as that kind of information hazard, and critiques it as such.
Trouble is, if you understand that most normies are delusional about some important issue, and you you develop some genuinely deeper insights into that issue, the psychologically predictable result is some degree of alienation and frustration. This is true for everyone who has a religious conversion experience. It’s true for everyone who really takes onboard the implications of any intellectually compelling science—whether cosmology, evolutionary biology, neuroscience, signaling theory, game theory, behavior genetics, etc. It’s true for everyone who learns about any branch of moral philosophy and takes it seriously as a guide to action.
I’ve seen this over, and over, and over in my own field of evolutionary psychology. The usual ‘character arc’ of ev psych insight is that (1) you read Dawkins or Pinker or Buss, you get filled with curiosity about the origins of human nature, (2) you learn some more and you feel overwhelming intellectual awe and excitement about the grandeur of evolutionary theory, (3) you gradually come to understand that every human perception, preference, value, desire, emotion, and motivation has deep evolutionary roots beyond your control, and you start to feel uneasy, (4) you ruminate about how you’re nothing but an evolved robot chasing reproductive success through adaptively self-deceived channels, and you feel some personal despair, (5) you look around at a society full of other self-deceived humans unaware of their biological programming, and you feel black-pilled civilizational despair, (6) you live with the Darwinian nihilism for a few years, adapt to the new normal, and gradually find some way to live with the new insights, climbing your way back into some semblance of normie-adjacent happiness. I’ve seen these six phases many times in my own colleagues, grad students, and collaborators.
And that’s just with a new descriptive world-view about how the human world works. EA’s challenge can be even more profound, because it’s not just descriptive, but normative, or at least prescriptive. So there’s a painful gap between what we could be doing, and what we are doing. And so there should be, if you take the world in a morally serious way.
I think the deeper problem is that given 20th century history, there’s a general dubiousness about any group of people who do take the world in a morally serious way that deviates from the usual forms of mild political virtue signaling encouraged in our current system of credentialism, careerism, and consumerism.
I think even on EA’s own terms (apart from any effects from EA being fringe) there’s a good reason for EAs to be OK with being more stressed and unhappy than people with other philosophies.
On the scale of human history we’re likely in an emergency situation when we have an opportunity to trade off the happiness of EAs for enormous gains in total well-being. Similar to how during a bear attack you’d accept that you won’t feel relaxed and happy while you try to mitigate the attack, but this period of stress is worth it overall. This is especially true if you believe we’re in the hinge of history.
Eh this logic can be used to justify a lot of extreme action in the name of progress. Communists and Marxists have had a lot of thoughts about the “hinge of history” and used that to unleash terrible destruction on the rest of humanity.
In contrast to a bear attack, you don’t expect to know that the “period of stress” has ended during your lifetime. Which raises a few questions, like “Is it worth it?” and “How sure can we be that this really is a stress period?”. The thought that we especially are in a position to trade our happiness for enormous gains for society—while not impossible—is dangerous in that it’s very appealing, regardless whether it’s true or not.
I’m not denying that what you say is true, but on the face of it, “the appeal of this ideology is that you have to sacrifice a lot for others’ gain” is not an intuitively compelling message.
I expect to know this. Either AI will go well and we’ll get the glorious transhuman future, or it’ll go poorly and we’ll have a brief moment of realization before we are killed etc. (or more realistically, a longer moment of awareness where we realize all is truly and thoroughly lost, before eventually the nanobots orwhatever come for us).
By many numbers AI risk being solved would only reduce total probability of X-risk by 1⁄3, 2⁄3, or maybe 9⁄10 if you are very heavy on AI-risk probability.
Personally I think humanity’s “period of stress” will take at least 1000s of years to be solved but I might be being quite pessimistic. Of course situations will get better but I think world will still be “burning” for quite some time.
Here’s a common belief in these circles (which I share):
If AI risk is solved through means other than “we collectively coordinate to not build TAI”(a solution which I think is unlikely both because that level of global coordination is very hard and because the opportunity costs are massive), then soon after, whether human civilization flourishes or not is mostly a question that’s out of human hands.
This is completely unrelated to the great point you made with the comment but I felt I had to share a classic? EA tip that worked well for me. (uncertain how much this counts as a classic.) I got to the nice nihilistic bottom of realising that my moral system is essentially based on evolution but I reversed that within a year by reading a bunch of Buddhist philosophy and by meditating. Now it’s all nirvana over here! (try it out now...)
https://www.lesswrong.com/posts/Mf2MCkYgSZSJRz5nM/a-non-mystical-explanation-of-insight-meditation-and-the
https://www.lesswrong.com/posts/WYmmC3W6ZNhEgAmWG/a-mechanistic-model-of-meditation
https://www.lesswrong.com/s/ZbmRyDN8TCpBTZSip