I work at MIRI, but as usual, this comment is me speaking for myself, and I haven’t heard from Eliezer or anyone else on whether they’d agree with the following.
My general thoughts:
The primary things I like about this post are that (1) it focuses on specific points of disagreement, encouraging us to then hash out a bunch of object-level questions; and (2) it might help wake some people from their dream if they hero-worship Eliezer, or if they generally think that leaders in this space can do no wrong.
By “hero-worshipping” I mean a cognitive algorithm, not a set of empirical conclusions. I’m generally opposed to faux egalitarianism and the Modest-Epistemology reasoning discussed in Inadequate Equilibria: if your generalized anti-hero-worship defenses force the conclusion that there just aren’t big gaps in skills or knowledge (or that skills and knowledge always correspond to mainstream prestige and authority), then your defenses are ruling out reality a priori. In saying “people need to hero-worship Eliezer less”, I’m opposing a certain kind of reasoning process and mindset, not a specific factual belief like “Eliezer is the clearest thinker about AI risk”.
In a sense, I want to promote the idea that the latter is a boring claim, to be evaluated like any other claim about the world; flinching away from it (e.g., because Eliezer is weird and says sci-fi-sounding stuff) and flinching toward it (e.g., because you have a bunch of your identity invested in the idea that the Sequences are awesome and rationalists are great) are both errors of process.
The main thing I dislike about this post is that it introduces a bunch of not-obviously-false Eliezer-claims — claims that EAs either widely disagree about, or haven’t discussed — and then dives straight into ‘therefore Eliezer has a bad track record’.
E.g., I disagree that molecular nanotech isn’t a big deal (if that’s a claim you’re making?), that Robin better predicted deep learning than Eliezer did, and that your counter-arguments against Eliezer and Bostrom are generally strong. Certainly I don’t think these points have been well-established enough that it makes sense to cite them in the mode ‘look at these self-evident ways Yudkowsky got stuff wrong; let us proceed straight to psychoanalysis, without dwelling on the case for why I think he’s wrong about this stuff’. At this stage of the debate on those topics, it would be more appropriate to talk in terms of cruxes like ‘I think the history of tech shows it’s ~always continuous in technological change and impact’, so it’s clear why you disagree with Eliezer in the first place.
I generally think that EA’s core bottlenecks right now are related to ‘willingness to be candid and weird enough to make intellectual progress (especially on AI alignment), and to quickly converge on our models of the world’.
My own models suggest to me that EA’s path to impact is almost entirely as a research community and a community that helps produce other research communities, rather than via ‘changing the culture of the world at large’ or going into politics or what-have-you. In that respect, rigor and skepticism is good, but singling out Eliezer because he’s unusually weird and candid is bad, because it discourages others from expressing weird/novel/minority views and from blurting out their true thought processes. (I recognize that this isn’t the only reason you’re singling Eliezer out, but it’s obviously a contributing factor.)
I am a big fan of Ben’s follow-up comment. Especially the part where he outlines the thought process that led to him generating the post’s contents. I think this is an absolutely wonderful thing to include in a variety of posts, or to add in the comment sections for a lot of posts.
Some specific thoughts on Ben’s follow-up comment:
1. I agree with Ben on this: “If a lot of people in the community believe AI is probably going to kill everyone soon, then (if they’re wrong) this can have really important negative effects”.
I think they’re not wrong, and I think the benefits of discussing this openly strongly outweigh the costs. But the negative effects are no less real for that.
(Separately, I think the “death with dignity” post was a suboptimal way to introduce various people to the view that p(doom) is very high. I’m much more confident that we should discuss this at all, than that Eliezer or I or others have been discussing this optimally.)
2. “Directly and indirectly, deference to Yudkowsky has a significant influence on a lot of people’s views”
Agreed.
Roughly speaking, my own view is:
EAs currently do a very high amount of deferring to others (both within EA and outside of EA) on topics like AI, global development, moral philosophy, economics, cause prioritization, organizational norms, personal career development, etc.
On the whole, EAs currently do a low amount of model-building and developing their own inside views.
EAs should switch to doing a medium amount of deference on topics like the ones I listed, and a very high amount of personal model-building.
Note that model-building can be useful even if you think all your conclusions will be strictly worse than the models of some other person you’ve identified. I’m pretty radical on this topic, and think that nearly all EAs should spend a nontrivial fraction of their time developing their own inside-view models of EA-relevant stuff, in spite of the obvious reasons (like gains from specialization) that this would normally not make sense.
Happy to say more about my views here, and I’ll probably write a post explaining why I think this.
I think the Alignment Research Field Guide, in spite of nominally being about “alignment”, is the best current intro resource for “how should I go about developing my own models on EA stuff?” A lot of the core advice is important and generalizes extremely well, IMO.
Insofar as EAs should do deference at all, Eliezer is in the top tier of people it makes sense to defer to.
But I’d guess the current amount of Eliezer-deference is way too high, because the current amount of deference overall is way too high. Eliezer should get a relatively high fraction of the deference pie IMO, but the overall pie should shrink a lot.
3. I also agree with Ben on “The track records of influential intellectuals (including Yudkowsky) should be publicly discussed.”
I don’t like the execution of the OP, but I strongly disagree with the people in the comments who have said “let us never publicly talk about individuals’ epistemic track records at all”—both because I think ‘how good is EY’s reasoning’ is a genuine crux for lots of people, and because I think this is a very common topic people think about, both in more pro-Eliezer and in more anti-Eliezer camps.
Discussing cruxes is obviously good, but even if this weren’t a crux for anyone, I’m strongly in favor of EAs doing a lot more “sharing their actual thoughts out loud”, including the more awkward and potentially inflammatory ones. (I’m happy to say more about why I think this.)
I do think it’s worth talking about what the best way is to discuss individuals’ epistemic track records, without making EA feel hostile/unpleasant/scary. I think EAs are currently way too timid (on average) about sharing their thoughts, so I worry about any big norm shifts that might make that problem even worse.
But Eliezer’s views are influential enough (and cover a topic, AGI, that is complicated and difficult enough to reason about) that this just seems like an important topic to me (similar to ‘how much should we defer to Paul?’, etc.). I’d rather see crappy discussion of this in the community than zero discussion whatsoever.
This is in large part Eliezer’s fault for picking such a bad post title, but I should still note that this is a very misleading summary. “Dying with dignity” often refers to giving up on taking any actions to keep yourself alive.
Eliezer’s version of “dying with dignity” is exactly the opposite: he’s advocating for doing whatever it takes to maximize the probability that humanity survives.
It’s true that he thinks we’ll probably fail (and I agree), and he thinks we should emotionally reconcile ourselves with that fact (because he thinks this emotional reconciliation will itself increase our probability of surviving!!), but he doesn’t advocate giving up.
Quoting the post:
“Q1: Does ‘dying with dignity’ in this context mean accepting the certainty of your death, and not childishly regretting that or trying to fight a hopeless battle?
“Don’t be ridiculous. How would that increase the log odds of Earth’s survival?”
At least up until 1999, admittedly when he was still only about 20 years old, Yudkowsky argued that transformative nanotechnology would probably emerge suddenly and soon (“no later than 2010”) and result in human extinction by default.
I think the “no later than 2010” prediction is from when Eliezer was 20, but the bulk of the linked essay was written when he was 17. The quotation here is: “As of ’95, Drexler was giving the ballpark figure of 2015. I suspect the timetable has been accelerated a bit since then. My own guess would be no later than 2010.”
The argument for worrying about extinction via molecular nanotech to some non-small degree seems pretty straightforward and correct: molecular nanotech lets you build arbitrary structures, including dangerous ones, and some humans would want to destroy the world given the power to do so.
Eliezer was overconfident about nanotech timelines (though roughly to the same degree as Drexler, the world’s main authority on nanotech).
Eliezer may have also been overconfident about nanotech’s riskiness, but the specific thing he said when he was 17 is that he considered it important for humanity to achieve AGI “before nanotechnology, given the virtual certainty of deliberate misuse—misuse of a purely material (and thus, amoral) ultratechnology, one powerful enough to destroy the planet”.
It’s not clear to me whether this is saying that human-extinction-scale misuse from nanotech is ‘virtually certain’, versus the more moderate claim that some misuse is ‘virtually certain’ if nanotech sees wide usage (and any misuse is pretty terrifying in EV terms). The latter seems reasonable to me, given how powerful molecular nanotechnology would be.
Eliezer denies that he has a general tendency toward alarmism:
[Ngo][18:19]] (As a side note, I think that if Eliezer had been around in the 1930s, and you described to him what actually happened with nukes over the next 80 years, he would have called that “insanely optimistic”.)
[Yudkowsky][18:21] Mmmmmmaybe. Do note that I tend to be more optimistic than the average human about, say, global warming, or everything in transhumanism outside of AGI.
Nukes have going for them that, in fact, nobody has an incentive to start a global thermonuclear war. Eliezer is not in fact pessimistic about everything and views his AGI pessimism as generalizing to very few other things, which are not, in fact, as bad as AGI.
[Ngo][18:27] [...] So yeah, I picture 1930s-Eliezer pointing to technological trends and being like “by default, 30 years after the first nukes are built, you’ll be able to build one in your back yard. And governments aren’t competent enough to stop that happening.”
And I don’t think I could have come up with a compelling counterargument back then.
[Yudkowsky][18:29] So, I mean, in fact, I don’t prophesize doom from very many trends at all! It’s literally just AGI that is anywhere near that unmanageable! Many people in EA are more worried about biotech than I am, for example.
It seems fair to note that nanotech is a second example of Eliezer raising alarm bells. But this remains a pretty small number of data points, and in neither of those cases does it actually look unreasonable to worry a fair bit—those are genuinely some of the main ways we could destroy ourselves.
I think ‘Eliezer predicted nanotech way too early’ is a better data point here, as evidence for ‘maybe Eliezer tends to have overly aggressive tech forecasts’.
If Eliezer was deferring to Drexler to some extent, that makes the data a bit less relevant, but ‘I was deferring to someone else who was also wrong’ is not in fact a general-purpose excuse for getting the wrong answer.
In 2001, and possibly later, Yudkowsky apparently believed that his small team would be able to develop a “final stage AI” that would “reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010.”
In the first half of the 2000s, he produced a fair amount of technical and conceptual work related to this goal. It hasn’t ultimately had much clear usefulness for AI development, and, partly on the basis, my impression is that it has not held up well—but that he was very confident in the value of this work at the time.
That view seems very dumb to me — specifically the belief that SingInst’s very first unvetted idea would pan out and result in them building AGI, more so than the timelines per se.
I don’t fault 21-year-old Eliezer for trying (except insofar as he was totally wrong about the probability of Unfriendly AI at the time!), because the best way to learn that a weird new path is unviable is often to just take a stab at it. But insofar as 2001-Eliezer thought his very first idea was very likely to work, this seems like a totally fair criticism of the quality of his reasoning at the time.
Looking at the source text, I notice that the actual text is much more hedged than Ben’s summary (though it still sounds foreseeably overconfident to me, to the extent I can glean likely implicit probabilities from tone):
[...] The Singularity Institute is fully aware that creating true intelligence will not be easy. In addition to the enormous power deficit between modern computers and the human brain, there is an even more severe software deficit. The software of the human brain is the result of millions of years of evolution and contains perhaps tens of thousands of complex functional adaptations. The human brain itself is not a homogenous lump but a highly modular supersystem; the cerebral cortex is divided into two hemispheres, each containing 52 areas, each area subdivided into a half-dozen distinguishable maps. Cortical neurons group into minicolumns of perhaps a hundred neurons and macrocolumns of a few hundred minicolumns, with perhaps 1,000 macrocolumns to a cortical map. Of the 750 megabytes of human DNA, the vast majority is believed to be junk and 98% is identical to chimpanzee DNA, with perhaps 1% being concerned with intelligence—leaving 7.5 megabytes to specify, not the actual wiring of the brain, but the neuroanatomy of areas and maps and pathways, and the initial tiling patterns and learning algorithms for neurons and minicolumns and macrocolumns.
The Singularity Institute seriously intends to build a true general intelligence, possessed of all the key subsystems of human intelligence, plus design features unique to AI. We do not hold that all the complex features of the human mind are “emergent”, or that intelligence is the result of some simple architectural principle, or that general intelligence will appear if we simply add enough data or computing power. We are willing to do the work required to duplicate the massive complexity of human intelligence; to explore the functionality and behavior of each system and subsystem until we have a complete blueprint for a mind. For more about our Artificial Intelligence plans, see the document General Intelligence and Seed AI.
Our specific cognitive architecture and development plan forms our basis for answering questions such as “Will transhumans be friendly to humanity?” and “When will the Singularity occur?” At the Singularity Institute, we believe that the answer to the first question is “Yes” with respect to our proposed AI design—if we didn’t believe that, the Singularity Institute would not exist. Our best guess for the timescale is that our final-stage AI will reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010. As always with basic research, this is only a guess, and heavily contingent on funding levels. [...]
Note that this paper was written much earlier than its publication date. Description from yudkowsky.net: “Book chapter I wrote in 2002 for an edited volume, Artificial General Intelligence, which is now supposed to come out in late 2006. I no longer consider LOGI’s theory useful for building de novo AI. However, it still stands as a decent hypothesis about the evolutionary psychology of human general intelligence.”
Although Hanson very clearly wasn’t envisioning something like deep learning either, his side of the argument seems to fit better with what AI progress has looked like over the past decade.
I agree that Eliezer loses Bayes points (e.g., relative to Shane Legg and Dario Amodei) for not predicting the enormous success of deep learning. See also Nate’s recent post about this.
I disagree that Robin Hanson scored Bayes points off of Eliezer, on net, from the deep learning revolution, or that Hanson’s side of the Foom debate looks good (compared to Eliezer’s) with the benefit of hindsight. I side with Gwern here; I think Robin’s predictions and arguments on this topic have been terrible, as a rule.
I think that Yudkowsky’s prediction—that a small amount of code, run using only a small amount of computing power, was likely to abruptly jump economic output upward by more than a dozen orders-of-magnitude—was extreme enough to require very strong justifications.
I think Eliezer assigned too high a probability to ‘it’s easy to find relatively clean, understandable approaches to AGI’, and too low a probability to ‘it’s easy to find relatively messy, brute-forced approaches to AGI’. A consequence of the latter is that he (IMO) underestimated how compute-intensive AGI was likely to be, and overestimated how important recursive self-improvement was likely to be.
I otherwise broadly agree with his picture. E.g.:
I expect AGI to represent a large, sharp capabilities jump. (I think this is unlikely to require a bunch of recursive self-improvement.)
I think AGI is mainly bottlenecked on software, rather than hardware. (E.g., I think GPT-3 is impressive, but isn’t a baby AGI; rather than AGI just being ‘current systems but bigger’, I expect at least one more key insight lies on the shortest likely path to AGI.)
And I expect AGI to be much more efficient than current systems at utilizing small amounts of data. Though (because it’s likely to come from a relatively brute-forced, unalignable approach) I still expect it to be more compute-intensive than 2009-Eliezer was imagining.
However, later analysis has suggested that coherence arguments have either no or very limited implications for how we should expect future AI systems to behave.
I think that part of why Eliezer’s early stuff sounds weird is:
He generally had a lower opinion of the competence of elites in business, science, etc. (Which he later updated about.)
He had a lower opinion of the field of AI in particular, as it existed in the 1990s and 2000s. Maybe more like nutrition science or continental philosophy than like chemistry, on the scale of ‘field rigor and intellectual output’.
If you think of A(G)I as a weird, neglected, pre-paradigmatic field that gets very little attention outside of science fiction writing, then it’s less surprising to think it’s possible to make big, fast strides in the field. Outperforming a competitive market is very different from outperforming a small, niche market where very little high-quality effort is going into trying new things.
Similarly, if you have a lower opinion of elites, you should be more willing to endorse weird, fringe ideas, because you should be less confident that the mainstream is efficient relative to you. (And I think Eliezer still has a low opinion of elites on some very important dimensions, compared to a lot of EAs. But not to the same degrees as teenaged Eliezer.)
I used to think—not from experience, but from the general memetic atmosphere I grew up in—that executives were just people who, by dint of superior charisma and butt-kissing, had managed to work their way to the top positions at the corporate hog trough.
No, that was just a more comfortable meme, at least when it comes to what people put down in writing and pass around. The story of the horrible boss gets passed around more than the story of the boss who is, not just competent, but more competent than you.
[...]
But the business world is not the only venue where I’ve encountered the upper echelons and discovered that, amazingly, they actually are better at what they do.
Case in point: Professor Rodney Brooks, CTO of iRobot and former director of the MIT AI Lab, who spoke at the 2007 Singularity Summit. I had previously known “Rodney Brooks” primarily as the promoter of yet another dreadful nouvelle paradigm in AI—the embodiment of AIs in robots, and the forsaking of deliberation for complicated reflexes that didn’t involve modeling. Definitely not a friend to the Bayesian faction. Yet somehow Brooks had managed to become a major mainstream name, a household brand in AI...
And by golly, Brooks sounded intelligent and original. He gave off a visible aura of competence.
At one of the first conferences organized around the tiny little subfield of Artificial General Intelligence, I met someone who was heading up a funded research project specifically declaring AGI as a goal, within a major corporation. I believe he had people under him on his project. He was probably paid at least three times as much as I was paid (at that time). His academic credentials were superior to mine (what a surprise) and he had many more years of experience. He had access to lots and lots of computing power.
And like nearly everyone in the field of AGI, he was rushing forward to write code immediately—not holding off and searching for a sufficiently precise theory to permit stable self-improvement.
In short, he was just the sort of fellow that… Well, many people, when they hear about Friendly AI, say: “Oh, it doesn’t matter what you do, because [someone like this guy] will create AI first.” He’s the sort of person about whom journalists ask me, “You say that this isn’t the time to be talking about regulation, but don’t we need laws to stop people like this from creating AI?”
“I suppose,” you say, your voice heavy with irony, “that you’re about to tell us, that this person doesn’t really have so much of an advantage over you as it might seem. Because your theory—whenever you actually come up with a theory—is going to be so much better than his. Or,” your voice becoming even more ironic, “that he’s too mired in boring mainstream methodology—”
No. I’m about to tell you that I happened to be seated at the same table as this guy at lunch, and I made some kind of comment about evolutionary psychology, and he turned out to be...
...a creationist.
This was the point at which I really got, on a gut level, that there was no test you needed to pass in order to start your own AGI project.
One of the failure modes I’ve come to better understand in myself since observing it in others, is what I call, “living in the should-universe”. The universe where everything works the way it common-sensically ought to, as opposed to the actual is-universe we live in. There’s more than one way to live in the should-universe, and outright delusional optimism is only the least subtle. Treating the should-universe as your point of departure—describing the real universe as the should-universe plus a diff—can also be dangerous.
Up until the moment when yonder AGI researcher explained to me that he didn’t believe in evolution because that’s not what the Bible said, I’d been living in the should-universe. In the sense that I was organizing my understanding of other AGI researchers as should-plus-diff. I saw them, not as themselves, not as their probable causal histories, but as their departures from what I thought they should be.
[...] When Scott Aaronson was 12 years old, he: “set myself the modest goal of writing a BASIC program that would pass the Turing Test by learning from experience and following Asimov’s Three Laws of Robotics. I coded up a really nice tokenizer and user interface, and only got stuck on the subroutine that was supposed to understand the user’s question and output an intelligent, Three-Laws-obeying response.” It would be pointless to try and construct a diff between Aaronson12 and what an AGI researcher should be. You’ve got to explain Aaronson12 in forward-extrapolation mode: He thought it would be cool to make an AI and didn’t quite understand why the problem was difficult.
It was yonder creationist who let me see AGI researchers for themselves, and not as departures from my ideal.
[...]
The really striking fact about the researchers who show up at AGI conferences, is that they’re so… I don’t know how else to put it...
More like… around, say, the level of above-average scientists, which I yesterday compared to the level of partners at a non-big-name venture capital firm. Some of whom might well be Christians, or even creationists if they don’t work in evolutionary biology.
The attendees at AGI conferences aren’t literally average mortals, or even average scientists. The average attendee at an AGI conference is visibly one level up from the average attendee at that random mainstream AI conference I talked about yesterday.
[...] But even if you just poke around on Norvig or McCarthy’s website, and you’ve achieved sufficient level yourself to discriminate what you see, you’ll get a sense of a formidable mind. Not in terms of accomplishments—that’s not a fair comparison with someone younger or tackling a more difficult problem—but just in terms of the way they talk. If you then look at the website of a typical AGI-seeker, even one heading up their own project, you won’t get an equivalent sense of formidability.
[...] If you forget the should-universe, and think of the selection effect in the is-universe, it’s not difficult to understand. Today, AGI attracts people who fail to comprehend the difficulty of AGI. Back in the earliest days, a bright mind like John McCarthy would tackle AGI because no one knew the problem was difficult. In time and with regret, he realized he couldn’t do it. Today, someone on the level of Peter Norvig knows their own competencies, what they can do and what they can’t; and they go on to achieve fame and fortune (and Research Directorship of Google) within mainstream AI.
And then...
Then there are the completely hopeless ordinary programmers who wander onto the AGI mailing list wanting to build a really big semantic net.
Or the postdocs moved by some (non-Singularity) dream of themselves presenting the first “human-level” AI to the world, who also dream an AI design, and can’t let go of that.
Just normal people with no notion that it’s wrong for an AGI researcher to be normal.
Indeed, like most normal people who don’t spend their lives making a desperate effort to reach up toward an impossible ideal, they will be offended if you suggest to them that someone in their position needs to be a little less imperfect.
This misled the living daylights out of me when I was young, because I compared myself to other people who declared their intentions to build AGI, and ended up way too impressed with myself; when I should have been comparing myself to Peter Norvig, or reaching up toward E. T. Jaynes. (For I did not then perceive the sheer, blank, towering wall of Nature.)
I don’t mean to bash normal AGI researchers into the ground. They are not evil. They are not ill-intentioned. They are not even dangerous, as individuals. Only the mob of them is dangerous, that can learn from each other’s partial successes and accumulate hacks as a community.
And that’s why I’m discussing all this—because it is a fact without which it is not possible to understand the overall strategic situation in which humanity finds itself, the present state of the gameboard. It is, for example, the reason why I don’t panic when yet another AGI project announces they’re going to have general intelligence in five years. It also says that you can’t necessarily extrapolate the FAI-theory comprehension of future researchers from present researchers, if a breakthrough occurs that repopulates the field with Norvig-class minds.
Even an average human engineer is at least six levels higher than the blind idiot god, natural selection, that managed to cough up the Artificial Intelligence called humans, by retaining its lucky successes and compounding them. And the mob, if it retains its lucky successes and shares them, may also cough up an Artificial Intelligence, with around the same degree of precise control. But it is only the collective that I worry about as dangerous—the individuals don’t seem that formidable.
If you yourself speak fluent Bayesian, and you distinguish a person-concerned-with-AGI as speaking fluent Bayesian, then you should consider that person as excepted from this whole discussion.
Of course, among people who declare that they want to solve the AGI problem, the supermajority don’t speak fluent Bayesian.
Why would they? Most people don’t.
I think this, plus Eliezer’s general ‘fuck it, I’m gonna call it like I see it rather than be reflexively respectful to authority’ attitude, explains most of Ben’s ‘holy shit, your views were so weird!!’ thing.
“I don’t fault 21-year-old Eliezer for trying (except insofar as he was totally wrong about the probability of Unfriendly AI at the time!), because the best way to learn that a weird new path is unviable is often to just take a stab at it”
It was only weird in that involved technologies and methods that were unlikely to work, and EY could have figured that out theoretically by learning more about AI and software development.
I work at MIRI, but as usual, this comment is me speaking for myself, and I haven’t heard from Eliezer or anyone else on whether they’d agree with the following.
My general thoughts:
The primary things I like about this post are that (1) it focuses on specific points of disagreement, encouraging us to then hash out a bunch of object-level questions; and (2) it might help wake some people from their dream if they hero-worship Eliezer, or if they generally think that leaders in this space can do no wrong.
By “hero-worshipping” I mean a cognitive algorithm, not a set of empirical conclusions. I’m generally opposed to faux egalitarianism and the Modest-Epistemology reasoning discussed in Inadequate Equilibria: if your generalized anti-hero-worship defenses force the conclusion that there just aren’t big gaps in skills or knowledge (or that skills and knowledge always correspond to mainstream prestige and authority), then your defenses are ruling out reality a priori. In saying “people need to hero-worship Eliezer less”, I’m opposing a certain kind of reasoning process and mindset, not a specific factual belief like “Eliezer is the clearest thinker about AI risk”.
In a sense, I want to promote the idea that the latter is a boring claim, to be evaluated like any other claim about the world; flinching away from it (e.g., because Eliezer is weird and says sci-fi-sounding stuff) and flinching toward it (e.g., because you have a bunch of your identity invested in the idea that the Sequences are awesome and rationalists are great) are both errors of process.
The main thing I dislike about this post is that it introduces a bunch of not-obviously-false Eliezer-claims — claims that EAs either widely disagree about, or haven’t discussed — and then dives straight into ‘therefore Eliezer has a bad track record’.
E.g., I disagree that molecular nanotech isn’t a big deal (if that’s a claim you’re making?), that Robin better predicted deep learning than Eliezer did, and that your counter-arguments against Eliezer and Bostrom are generally strong. Certainly I don’t think these points have been well-established enough that it makes sense to cite them in the mode ‘look at these self-evident ways Yudkowsky got stuff wrong; let us proceed straight to psychoanalysis, without dwelling on the case for why I think he’s wrong about this stuff’. At this stage of the debate on those topics, it would be more appropriate to talk in terms of cruxes like ‘I think the history of tech shows it’s ~always continuous in technological change and impact’, so it’s clear why you disagree with Eliezer in the first place.
I generally think that EA’s core bottlenecks right now are related to ‘willingness to be candid and weird enough to make intellectual progress (especially on AI alignment), and to quickly converge on our models of the world’.
My own models suggest to me that EA’s path to impact is almost entirely as a research community and a community that helps produce other research communities, rather than via ‘changing the culture of the world at large’ or going into politics or what-have-you. In that respect, rigor and skepticism is good, but singling out Eliezer because he’s unusually weird and candid is bad, because it discourages others from expressing weird/novel/minority views and from blurting out their true thought processes. (I recognize that this isn’t the only reason you’re singling Eliezer out, but it’s obviously a contributing factor.)
I am a big fan of Ben’s follow-up comment. Especially the part where he outlines the thought process that led to him generating the post’s contents. I think this is an absolutely wonderful thing to include in a variety of posts, or to add in the comment sections for a lot of posts.
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
Some specific thoughts on Ben’s follow-up comment:
1. I agree with Ben on this: “If a lot of people in the community believe AI is probably going to kill everyone soon, then (if they’re wrong) this can have really important negative effects”.
I think they’re not wrong, and I think the benefits of discussing this openly strongly outweigh the costs. But the negative effects are no less real for that.
(Separately, I think the “death with dignity” post was a suboptimal way to introduce various people to the view that p(doom) is very high. I’m much more confident that we should discuss this at all, than that Eliezer or I or others have been discussing this optimally.)
2. “Directly and indirectly, deference to Yudkowsky has a significant influence on a lot of people’s views”
Agreed.
Roughly speaking, my own view is:
EAs currently do a very high amount of deferring to others (both within EA and outside of EA) on topics like AI, global development, moral philosophy, economics, cause prioritization, organizational norms, personal career development, etc.
On the whole, EAs currently do a low amount of model-building and developing their own inside views.
EAs should switch to doing a medium amount of deference on topics like the ones I listed, and a very high amount of personal model-building.
Note that model-building can be useful even if you think all your conclusions will be strictly worse than the models of some other person you’ve identified. I’m pretty radical on this topic, and think that nearly all EAs should spend a nontrivial fraction of their time developing their own inside-view models of EA-relevant stuff, in spite of the obvious reasons (like gains from specialization) that this would normally not make sense.
Happy to say more about my views here, and I’ll probably write a post explaining why I think this.
I think the Alignment Research Field Guide, in spite of nominally being about “alignment”, is the best current intro resource for “how should I go about developing my own models on EA stuff?” A lot of the core advice is important and generalizes extremely well, IMO.
Insofar as EAs should do deference at all, Eliezer is in the top tier of people it makes sense to defer to.
But I’d guess the current amount of Eliezer-deference is way too high, because the current amount of deference overall is way too high. Eliezer should get a relatively high fraction of the deference pie IMO, but the overall pie should shrink a lot.
3. I also agree with Ben on “The track records of influential intellectuals (including Yudkowsky) should be publicly discussed.”
I don’t like the execution of the OP, but I strongly disagree with the people in the comments who have said “let us never publicly talk about individuals’ epistemic track records at all”—both because I think ‘how good is EY’s reasoning’ is a genuine crux for lots of people, and because I think this is a very common topic people think about, both in more pro-Eliezer and in more anti-Eliezer camps.
Discussing cruxes is obviously good, but even if this weren’t a crux for anyone, I’m strongly in favor of EAs doing a lot more “sharing their actual thoughts out loud”, including the more awkward and potentially inflammatory ones. (I’m happy to say more about why I think this.)
I do think it’s worth talking about what the best way is to discuss individuals’ epistemic track records, without making EA feel hostile/unpleasant/scary. I think EAs are currently way too timid (on average) about sharing their thoughts, so I worry about any big norm shifts that might make that problem even worse.
But Eliezer’s views are influential enough (and cover a topic, AGI, that is complicated and difficult enough to reason about) that this just seems like an important topic to me (similar to ‘how much should we defer to Paul?’, etc.). I’d rather see crappy discussion of this in the community than zero discussion whatsoever.
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
Some specific thoughts on claims in the OP:
This is in large part Eliezer’s fault for picking such a bad post title, but I should still note that this is a very misleading summary. “Dying with dignity” often refers to giving up on taking any actions to keep yourself alive.
Eliezer’s version of “dying with dignity” is exactly the opposite: he’s advocating for doing whatever it takes to maximize the probability that humanity survives.
It’s true that he thinks we’ll probably fail (and I agree), and he thinks we should emotionally reconcile ourselves with that fact (because he thinks this emotional reconciliation will itself increase our probability of surviving!!), but he doesn’t advocate giving up.
Quoting the post:
“Q1: Does ‘dying with dignity’ in this context mean accepting the certainty of your death, and not childishly regretting that or trying to fight a hopeless battle?
“Don’t be ridiculous. How would that increase the log odds of Earth’s survival?”
I think the “no later than 2010” prediction is from when Eliezer was 20, but the bulk of the linked essay was written when he was 17. The quotation here is: “As of ’95, Drexler was giving the ballpark figure of 2015. I suspect the timetable has been accelerated a bit since then. My own guess would be no later than 2010.”
The argument for worrying about extinction via molecular nanotech to some non-small degree seems pretty straightforward and correct: molecular nanotech lets you build arbitrary structures, including dangerous ones, and some humans would want to destroy the world given the power to do so.
Eliezer was overconfident about nanotech timelines (though roughly to the same degree as Drexler, the world’s main authority on nanotech).
Eliezer may have also been overconfident about nanotech’s riskiness, but the specific thing he said when he was 17 is that he considered it important for humanity to achieve AGI “before nanotechnology, given the virtual certainty of deliberate misuse—misuse of a purely material (and thus, amoral) ultratechnology, one powerful enough to destroy the planet”.
It’s not clear to me whether this is saying that human-extinction-scale misuse from nanotech is ‘virtually certain’, versus the more moderate claim that some misuse is ‘virtually certain’ if nanotech sees wide usage (and any misuse is pretty terrifying in EV terms). The latter seems reasonable to me, given how powerful molecular nanotechnology would be.
Eliezer denies that he has a general tendency toward alarmism:
It seems fair to note that nanotech is a second example of Eliezer raising alarm bells. But this remains a pretty small number of data points, and in neither of those cases does it actually look unreasonable to worry a fair bit—those are genuinely some of the main ways we could destroy ourselves.
I think ‘Eliezer predicted nanotech way too early’ is a better data point here, as evidence for ‘maybe Eliezer tends to have overly aggressive tech forecasts’.
If Eliezer was deferring to Drexler to some extent, that makes the data a bit less relevant, but ‘I was deferring to someone else who was also wrong’ is not in fact a general-purpose excuse for getting the wrong answer.
That view seems very dumb to me — specifically the belief that SingInst’s very first unvetted idea would pan out and result in them building AGI, more so than the timelines per se.
I don’t fault 21-year-old Eliezer for trying (except insofar as he was totally wrong about the probability of Unfriendly AI at the time!), because the best way to learn that a weird new path is unviable is often to just take a stab at it. But insofar as 2001-Eliezer thought his very first idea was very likely to work, this seems like a totally fair criticism of the quality of his reasoning at the time.
Looking at the source text, I notice that the actual text is much more hedged than Ben’s summary (though it still sounds foreseeably overconfident to me, to the extent I can glean likely implicit probabilities from tone):
Note that this paper was written much earlier than its publication date. Description from yudkowsky.net: “Book chapter I wrote in 2002 for an edited volume, Artificial General Intelligence, which is now supposed to come out in late 2006. I no longer consider LOGI’s theory useful for building de novo AI. However, it still stands as a decent hypothesis about the evolutionary psychology of human general intelligence.”
I agree that Eliezer loses Bayes points (e.g., relative to Shane Legg and Dario Amodei) for not predicting the enormous success of deep learning. See also Nate’s recent post about this.
I disagree that Robin Hanson scored Bayes points off of Eliezer, on net, from the deep learning revolution, or that Hanson’s side of the Foom debate looks good (compared to Eliezer’s) with the benefit of hindsight. I side with Gwern here; I think Robin’s predictions and arguments on this topic have been terrible, as a rule.
I think Eliezer assigned too high a probability to ‘it’s easy to find relatively clean, understandable approaches to AGI’, and too low a probability to ‘it’s easy to find relatively messy, brute-forced approaches to AGI’. A consequence of the latter is that he (IMO) underestimated how compute-intensive AGI was likely to be, and overestimated how important recursive self-improvement was likely to be.
I otherwise broadly agree with his picture. E.g.:
I expect AGI to represent a large, sharp capabilities jump. (I think this is unlikely to require a bunch of recursive self-improvement.)
I think AGI is mainly bottlenecked on software, rather than hardware. (E.g., I think GPT-3 is impressive, but isn’t a baby AGI; rather than AGI just being ‘current systems but bigger’, I expect at least one more key insight lies on the shortest likely path to AGI.)
And I expect AGI to be much more efficient than current systems at utilizing small amounts of data. Though (because it’s likely to come from a relatively brute-forced, unalignable approach) I still expect it to be more compute-intensive than 2009-Eliezer was imagining.
This seems completely wrong to me. See Katja Grace’s Coherence arguments imply a force for goal-directed behavior.
I think that part of why Eliezer’s early stuff sounds weird is:
He generally had a lower opinion of the competence of elites in business, science, etc. (Which he later updated about.)
He had a lower opinion of the field of AI in particular, as it existed in the 1990s and 2000s. Maybe more like nutrition science or continental philosophy than like chemistry, on the scale of ‘field rigor and intellectual output’.
If you think of A(G)I as a weird, neglected, pre-paradigmatic field that gets very little attention outside of science fiction writing, then it’s less surprising to think it’s possible to make big, fast strides in the field. Outperforming a competitive market is very different from outperforming a small, niche market where very little high-quality effort is going into trying new things.
Similarly, if you have a lower opinion of elites, you should be more willing to endorse weird, fringe ideas, because you should be less confident that the mainstream is efficient relative to you. (And I think Eliezer still has a low opinion of elites on some very important dimensions, compared to a lot of EAs. But not to the same degrees as teenaged Eliezer.)
From Competent Elites:
And from Above-Average AI Scientists:
I think this, plus Eliezer’s general ‘fuck it, I’m gonna call it like I see it rather than be reflexively respectful to authority’ attitude, explains most of Ben’s ‘holy shit, your views were so weird!!’ thing.
“I don’t fault 21-year-old Eliezer for trying (except insofar as he was totally wrong about the probability of Unfriendly AI at the time!), because the best way to learn that a weird new path is unviable is often to just take a stab at it”
It was only weird in that involved technologies and methods that were unlikely to work, and EY could have figured that out theoretically by learning more about AI and software development.