Of course it is meaningful that Eliezer Yudkowsky has made a bunch of terrible predictions in the past that closely echo predictions he continues to make in slightly different form today.
I assume you’re mainly talking about young-Eliezer worrying about near-term risk from molecular nanotechnology, and current-Eliezer worrying about near-term risk from AGI?
I think age-17 Eliezer was correct to think widespread access to nanotech would be extremely dangerous. See my comment. If you or Ben disagree, why do you disagree?
Age-20 Eliezer was obviously wrong about the timing for nanotech, and this is obviously Bayesian evidence for ‘Eliezer may have overly-aggressive tech timelines in general’.
I don’t think this is generally true—e.g., if you took a survey of EAs worried about AI risk in 2010 or in 2014, I suspect Eliezer would have longer AI timelines than others at the time. (E.g., he expected it to take longer to solve Go than Carl Shulman did.) When I joined MIRI, the standard way we summarized MIRI’s view was roughly ‘We think AI risk is high, but not because we think AGI is imminent; rather, our worry is that alignment is likely to take a long time, and that civilization may need to lay groundwork decades in advance in order to have a realistic chance of building aligned AGI.’
But nanotech is a totally fair data point regardless.
Of course it is relevant that he has neither owned up to those earlier terrible predictions or explained how he has learned from those mistakes.
He also talks a lot about how he’s updated and revised his heuristics and world-models in other parts of the Sequences. (E.g., he writes that he underestimated elite competence when he was younger.)
What specific cognitive error do you want him to write about, that he hasn’t already written on?
This is sensible advice in any complex domain, and saying that we should “evaluate every argument in isolation on its merits” is a type of special pleading or sophistry.
I don’t think the argument I’m making (or most others are making) is ‘don’t update on people’s past mistakes’ or ‘never do deference’. Rather, a lot of the people discussing this matter within EA (Wei Dai, Gwern Branwen, Richard Ngo, Rohin Shah, Carl Shulman, Nate Soares, Ajeya Cotra, etc.) are the world’s leading experts in this area, and a lot of the world’s frontier progess on this topic is happening on Internet fora like the EA Forum and LessWrong. It makes sense for domain specialists to put much more focus into evaluating arguments on the merits; object-level conversations like these are how the intellectual advances occur that can then be reflected in aggregators like Metaculus.
Metaculus and prediction markets will be less accurate if frontier researchers replace object-level discussion with debates about who to defer to, in the same way that stock markets would be less efficient if everyone overestimated the market’s efficiency and put minimal effort into beating the market.
Insofar as we’re trying to grow the field, it also makes sense to encourage more EAs to try to think about these topics and build their own inside-view models; and this has the added benefit of reducing the risk of deference cascades.
(I also think there are other reasons it would be healthy for EA to spend a lot more time on inside-view building on topics like AI, normative ethics, and global poverty, as I briefly said here. But it’s possible to practice model-building and then decide at the end of the day, nonetheless, that you don’t put much weight on the domain-specific inside views you’ve built.)
extreme claims
When people use words like “extreme” here, I often get the sense that they aren’t crisply separating “extreme” in the sense of “weird-sounding” from “extreme” in the sense of “low prior probability”. I think Eliezer’s views are weird-sounding, not unlikely on priors.
E.g., why should we expect generally intelligent machines to be low-impact if built, or to never be built?
The idea that a post-AGI world looks mostly the same as a pre-AGI world might sound more normal and unsurprising to an early-21st-century well-off Anglophone intellectual, but I think this is just an error. It’s a clear case of the availability heuristic misfiring, not a prior anyone should endorse upon reflection.
I view the Most Important Century series as an attempt to push back against many versions of this conflation.
Epistemically, I view Paul’s model as much more “extreme” than Eliezer’s because I think it’s much more conjunctive. I obviously share the view that soft takeoff sounds more normal in some respects, but I don’t think this should inform our prior much. I’d guess we should start with a prior that assigns lots of weight to soft takeoff as well as to hard takeoff, and then mostly arrive at a conclusion based on the specific arguments for each view.
I assume you’re mainly talking about young-Eliezer worrying about near-term risk from molecular nanotechnology, and current-Eliezer worrying about near-term risk from AGI?
I think age-17 Eliezer was correct to think widespread access to nanotech would be extremely dangerous. See my comment. If you or Ben disagree, why do you disagree?
Age-20 Eliezer was obviously wrong about the timing for nanotech, and this is obviously Bayesian evidence for ‘Eliezer may have overly-aggressive tech timelines in general’.
I don’t think this is generally true—e.g., if you took a survey of EAs worried about AI risk in 2010 or in 2014, I suspect Eliezer would have longer AI timelines than others at the time. (E.g., he expected it to take longer to solve Go than Carl Shulman did.) When I joined MIRI, the standard way we summarized MIRI’s view was roughly ‘We think AI risk is high, but not because we think AGI is imminent; rather, our worry is that alignment is likely to take a long time, and that civilization may need to lay groundwork decades in advance in order to have a realistic chance of building aligned AGI.’
But nanotech is a totally fair data point regardless.
Eliezer wrote a 20,000-word essay series on his update, and the mistakes he thought he was making. Essay titles include “My Childhood Death Spiral”, “The Sheer Folly of Callow Youth”, “Fighting a Rearguard Action Against the Truth”, and “The Magnitude of His Own Folly”.
He also talks a lot about how he’s updated and revised his heuristics and world-models in other parts of the Sequences. (E.g., he writes that he underestimated elite competence when he was younger.)
What specific cognitive error do you want him to write about, that he hasn’t already written on?
I don’t think the argument I’m making (or most others are making) is ‘don’t update on people’s past mistakes’ or ‘never do deference’. Rather, a lot of the people discussing this matter within EA (Wei Dai, Gwern Branwen, Richard Ngo, Rohin Shah, Carl Shulman, Nate Soares, Ajeya Cotra, etc.) are the world’s leading experts in this area, and a lot of the world’s frontier progess on this topic is happening on Internet fora like the EA Forum and LessWrong. It makes sense for domain specialists to put much more focus into evaluating arguments on the merits; object-level conversations like these are how the intellectual advances occur that can then be reflected in aggregators like Metaculus.
Metaculus and prediction markets will be less accurate if frontier researchers replace object-level discussion with debates about who to defer to, in the same way that stock markets would be less efficient if everyone overestimated the market’s efficiency and put minimal effort into beating the market.
Insofar as we’re trying to grow the field, it also makes sense to encourage more EAs to try to think about these topics and build their own inside-view models; and this has the added benefit of reducing the risk of deference cascades.
(I also think there are other reasons it would be healthy for EA to spend a lot more time on inside-view building on topics like AI, normative ethics, and global poverty, as I briefly said here. But it’s possible to practice model-building and then decide at the end of the day, nonetheless, that you don’t put much weight on the domain-specific inside views you’ve built.)
When people use words like “extreme” here, I often get the sense that they aren’t crisply separating “extreme” in the sense of “weird-sounding” from “extreme” in the sense of “low prior probability”. I think Eliezer’s views are weird-sounding, not unlikely on priors.
E.g., why should we expect generally intelligent machines to be low-impact if built, or to never be built?
The idea that a post-AGI world looks mostly the same as a pre-AGI world might sound more normal and unsurprising to an early-21st-century well-off Anglophone intellectual, but I think this is just an error. It’s a clear case of the availability heuristic misfiring, not a prior anyone should endorse upon reflection.
I view the Most Important Century series as an attempt to push back against many versions of this conflation.
Epistemically, I view Paul’s model as much more “extreme” than Eliezer’s because I think it’s much more conjunctive. I obviously share the view that soft takeoff sounds more normal in some respects, but I don’t think this should inform our prior much. I’d guess we should start with a prior that assigns lots of weight to soft takeoff as well as to hard takeoff, and then mostly arrive at a conclusion based on the specific arguments for each view.