To complement your recommendation, I would also add that Yudowsky’s Sequences end up transmitting a somewhat packaged worldview, and I think that there are some dangers in that.
I agree that their summarization work is valuable, but some more unmediated original sources which could transmit some of the same value might be:
Probability Theory: The Logic of Science, or some other probability theory textbook
Stranger in a Strange Land, David’s Sling, The Moon is a Harsh Mistress.
Antifragile/Black Swan
The Elephant in the Brain
Superforecasting, making 100 forecasts and keeping track each week.
The Rationality Quotient, or some other Keith Stanovich book.
Some intro to nonviolent communication
Because Yudkowsky’s sequences are so long, and because I think that there is more value in reading the original sources, I’d probably lean towards recommending those instead.
Huh, I think this list of books covers less than half of the ideas in the sequences, so I don’t really think this counts as “the original sources”. Topics that get pretty extensively covered in the sequences but are absent here:
Evolutionary biology + psychology
AI Existential Risk
Metaethics & fragility of value
Something about courage/willing to do hard things/Something to protect (a recurring theme in the sequences and absent in all of the above)
Decision theory
Lots of other stuff.
Like, I don’t know, let’s look at some randomly selected sequence on LessWrong:
This sequence is particularly focused on noticing confusion and modeling scientific progress. None of the books you list above really cover that at all (The Logic of Science maybe the most, but it’s really not its core focus, and is also very technical and has giant holes in the middle of it due to its unfinished nature).
I have read all of the books/content you link above, and I don’t think it really has that much overlap with the content of the sequences, and don’t expect that someone who has read them to really have gotten close to most of the value of reading the sequences, so I don’t currently think this is a good recommendation.
This comment made me more sceptical about reading the sequences. I don’t think I can view anyone as an expert on all these topics. Is there a “best of” selection of the sequences somewhere?
I can’t speak to Yudkowsky’s knowledge of physics, economics, psychology etc, but as someone who studies philosophy I can tell you his philosophical segments are pretty weak. It’s clear that he hasn’t read a lot of philosophy and he isverydismissive of the field as a whole. He also has a tendency to reinvent the wheel (e.g his ‘Requiredism’ is what philosophers would call compatibilism).
When I read the sequences as a teenager I was very impressed by his philosophy, but as I got older and started reading more I realized how little he actually engaged with criticisms of his favorite theories, and when he did he often only engaged with weaker criticisms. If you want some good introductory texts on philosophy as well as criticism/alternatives to some of his/rationalists most central beliefs e.g physicalism, correspondence theory, scientific realism, the normativity of classical logic (all of which I have rejected as of the moment of this writing) then I highly recommend theStanford Encyclopedia of Philosophy.
In fairness, my memory of the philpapers survey is that there is more consensus amongst professional philosophers on scientific realism than on almost any other philosophical theory. (Though that’s going by the old survey, haven’t looked at the more recent one yet.) Although of course there are prominent philosophers of science who are anti-realist.
53.64% accept or lean towards classical logic (although that doesn’t tell us whether the philosophers think it has normative force).
I will say that PhilPapers has a rather small sample size and mostly collects data on english speaking philosophers, so I find it probable that these results are not representative of philosophers as a whole.
I did not say that the sequences cover all content in these books! I mean, they are quite long, so they cover a lot of adjacent topics, but I would not claim that the sequences are the canonical resource on all of these.
Eliezer isn’t (to my knowledge) an expert on, say, evolutionary biology. Reading the sequences will not make you an expert on evolutionary biology either.
They will, however, show you how to make a layman’s understanding of evolutionary biology relevant to your life.
I agree that my book list is incomplete, and it was aimed more at topics that the OP brought up.
For each of the additional topics you mentioned, it doesn’t seem like Yudkowsky’s Sequences are the best introduction. E.g., for decision theory I got more out of reading a random MIRI paper trying to formalize FDT. For AI x-risk in particular it would also surprise me if you would also recommend the sequences rather than some newer introduction.
I have read all of the books/content you link above
Is this literally true? In particular, have you read David’s Sling?
the best introduction. E.g., for decision theory I got more out of reading a random MIRI paper trying to formalize FDT.
Yeah, I think the best TDT/FDT/LDY material in-particular is probably MIRI papers. The original TDT paper is quite good, and I consider it kind of part of the sequences, since it’s written around the same time, and written in a pretty similar style.
For AI x-risk in particular it would also surprise me if you would also recommend the sequences rather than some newer introduction.
Nope, still think the sequences are by far the best (and indeed most alignment conversations I have with new people who showed up in the last 5 years tend to consist of me summarizing sequences posts, which has gotten pretty annoying after a while). There is of course useful additional stuff, but if someone wanted to start working on AI Alignment, the sequences still seem by far the best large thing to read (there are of course individual articles that do individual things best, but there isn’t really anything else textbook shaped).
What are the core pieces about AI risk in the sequences? Looking through the list, I don’t see any sequence about AI risk. Yudkowsky’s account on the Alignment Forum doesn’t have anything more than six years old, aka nothing from the sequences era.
Personally I’d point to Joe Carlsmith’s report, Richard Ngo’s writeups, Ajeya Cotra’s writeup, some of Holden Karnofsky’s writing, Concrete Problems in AI Safety and Unsolved Problems in ML Safety as the best introductions to the topic.
The primary purpose of the sequences was to communicate the generators behind AI risk and to teach the tools necessary (according to Eliezer) to make progress on it, so references to it are all over the place, and it’s the second most central theme to the essays.
Later essays in the sequences tend to have more references to AI risk than earlier ones. Here is a somewhat random selection of ones that seemed crucial when looking over the list, though this is really very unlikely to be comprehensive:
There are lots more. Indeed, towards the latter half of the sequences it’s hard not to see an essay quite straightforwardly about AI Alignment every 2-3 essays.
My guess is that he meant the sequences convey the kind of more foundational epistemology which helps people people derive better models on subjects like AI Alignment by themselves, though all of the sequences in The Machine in the Ghost and Mere Goodness have direct object-level relevance.
Excepting Ngo’s AGI safety from first principles, I don’t especially like most of those resources as introductions exactly because they offer readers very little opportunity to test or build on their beliefs. Also, I think most of them are substantially wrong. (Concrete Problems in AI Safety seems fine, but is also skipping a lot of steps. I haven’t read Unsolved Problems in ML Safety.)
I have read a good chunk of David’s Sling! Though it didn’t really click with me a ton, and I had already been spoiled on a good chunk of it because I had a bunch of conversations about it with friends, so I didn’t fully finish it.
For completeness sake, here is my reading state for all of the above:
Probability Theory: The Logic of Science, or some other probability theory textbook
Read
An introduction to general semantics
I read a bunch of general semantics stuff over the years, but I never really got into it, so a bit unclear.
Thanks for the comment and the list of recommendations. I have read most of the things on that list and the ones that I did read, I thought were great and have recommended a bunch to others, especially Probability Theory: The Logic of Science, The Elephant in the Brain, and forecasting practice. I agree that there are some dangers in recommending something that is pretty packaged but I think there is an obvious benefit in that it feels like a distilled version of reading a bunch of valuable things. Per unit time, I found reading the sequences more insightful/useful to me than the sources which it gets its ideas from (even if I read those things before the sequences, I am fairly confident).
I don’t want to oversell the sequences, I think the ideas in them have been mentioned in other places earlier. In my post, I mentioned specifically what ideas I found valuable so that people who are already familiar with them or think they are not that useful can decide not to read them. That isn’t rhetorical, I have some wise friends who are pretty well-read on philosophy and economics, and a lot of the things in the sequences I found novel, they were already familiar with.
My recommendation would look very different for someone who read the sequences and then made that their whole personality. I do know of some people who overrate them but in some of my specific circles (EA uni groups for eg), I think they are underrated/not given a chance which is why I wrote this post.
Well, if you are uncertain, note that experimentation here is very cheap, because you can download a copy of each book from an online library and quickly skim it to get a sense. So I’d recommend that.
To complement your recommendation, I would also add that Yudowsky’s Sequences end up transmitting a somewhat packaged worldview, and I think that there are some dangers in that.
I agree that their summarization work is valuable, but some more unmediated original sources which could transmit some of the same value might be:
Probability Theory: The Logic of Science, or some other probability theory textbook
An introduction to general semantics
An introduction to CBT, e.g., Feeling Good
Thinking Fast and Slow
How to Measure Anything
Surely You’re Joking Mr Feynman
Stranger in a Strange Land, David’s Sling, The Moon is a Harsh Mistress.
Antifragile/Black Swan
The Elephant in the Brain
Superforecasting, making 100 forecasts and keeping track each week.
The Rationality Quotient, or some other Keith Stanovich book.
Some intro to nonviolent communication
Because Yudkowsky’s sequences are so long, and because I think that there is more value in reading the original sources, I’d probably lean towards recommending those instead.
Huh, I think this list of books covers less than half of the ideas in the sequences, so I don’t really think this counts as “the original sources”. Topics that get pretty extensively covered in the sequences but are absent here:
Evolutionary biology + psychology
AI Existential Risk
Metaethics & fragility of value
Something about courage/willing to do hard things/Something to protect (a recurring theme in the sequences and absent in all of the above)
Decision theory
Lots of other stuff.
Like, I don’t know, let’s look at some randomly selected sequence on LessWrong:
This sequence is particularly focused on noticing confusion and modeling scientific progress. None of the books you list above really cover that at all (The Logic of Science maybe the most, but it’s really not its core focus, and is also very technical and has giant holes in the middle of it due to its unfinished nature).
I have read all of the books/content you link above, and I don’t think it really has that much overlap with the content of the sequences, and don’t expect that someone who has read them to really have gotten close to most of the value of reading the sequences, so I don’t currently think this is a good recommendation.
This comment made me more sceptical about reading the sequences. I don’t think I can view anyone as an expert on all these topics. Is there a “best of” selection of the sequences somewhere?
I can’t speak to Yudkowsky’s knowledge of physics, economics, psychology etc, but as someone who studies philosophy I can tell you his philosophical segments are pretty weak.
It’s clear that he hasn’t read a lot of philosophy and he is very dismissive of the field as a whole. He also has a tendency to reinvent the wheel (e.g his ‘Requiredism’ is what philosophers would call compatibilism).
When I read the sequences as a teenager I was very impressed by his philosophy, but as I got older and started reading more I realized how little he actually engaged with criticisms of his favorite theories, and when he did he often only engaged with weaker criticisms.
If you want some good introductory texts on philosophy as well as criticism/alternatives to some of his/rationalists most central beliefs e.g physicalism, correspondence theory, scientific realism, the normativity of classical logic (all of which I have rejected as of the moment of this writing) then I highly recommend the Stanford Encyclopedia of Philosophy.
In fairness, my memory of the philpapers survey is that there is more consensus amongst professional philosophers on scientific realism than on almost any other philosophical theory. (Though that’s going by the old survey, haven’t looked at the more recent one yet.) Although of course there are prominent philosophers of science who are anti-realist.
True, here are the results you’re talking about:
His views are moderately popular in general with:
51.37% accept or lean towards correspondence
51.93% accept or lean towards physicalism
30.56% accept or lean towards consequentialism
53.64% accept or lean towards classical logic (although that doesn’t tell us whether the philosophers think it has normative force).
I will say that PhilPapers has a rather small sample size and mostly collects data on english speaking philosophers, so I find it probable that these results are not representative of philosophers as a whole.
That’s true, I would only really trust the survey for what analytic philosophers think.
I did not say that the sequences cover all content in these books! I mean, they are quite long, so they cover a lot of adjacent topics, but I would not claim that the sequences are the canonical resource on all of these.
Eliezer isn’t (to my knowledge) an expert on, say, evolutionary biology. Reading the sequences will not make you an expert on evolutionary biology either.
They will, however, show you how to make a layman’s understanding of evolutionary biology relevant to your life.
I agree that my book list is incomplete, and it was aimed more at topics that the OP brought up.
For each of the additional topics you mentioned, it doesn’t seem like Yudkowsky’s Sequences are the best introduction. E.g., for decision theory I got more out of reading a random MIRI paper trying to formalize FDT. For AI x-risk in particular it would also surprise me if you would also recommend the sequences rather than some newer introduction.
Is this literally true? In particular, have you read David’s Sling?
Yeah, I think the best TDT/FDT/LDY material in-particular is probably MIRI papers. The original TDT paper is quite good, and I consider it kind of part of the sequences, since it’s written around the same time, and written in a pretty similar style.
Nope, still think the sequences are by far the best (and indeed most alignment conversations I have with new people who showed up in the last 5 years tend to consist of me summarizing sequences posts, which has gotten pretty annoying after a while). There is of course useful additional stuff, but if someone wanted to start working on AI Alignment, the sequences still seem by far the best large thing to read (there are of course individual articles that do individual things best, but there isn’t really anything else textbook shaped).
What are the core pieces about AI risk in the sequences? Looking through the list, I don’t see any sequence about AI risk. Yudkowsky’s account on the Alignment Forum doesn’t have anything more than six years old, aka nothing from the sequences era.
Personally I’d point to Joe Carlsmith’s report, Richard Ngo’s writeups, Ajeya Cotra’s writeup, some of Holden Karnofsky’s writing, Concrete Problems in AI Safety and Unsolved Problems in ML Safety as the best introductions to the topic.
The primary purpose of the sequences was to communicate the generators behind AI risk and to teach the tools necessary (according to Eliezer) to make progress on it, so references to it are all over the place, and it’s the second most central theme to the essays.
Later essays in the sequences tend to have more references to AI risk than earlier ones. Here is a somewhat random selection of ones that seemed crucial when looking over the list, though this is really very unlikely to be comprehensive:
Ghosts in the Machine
Optimization and the Intelligence Explosion
Belief in Intelligence
The Hidden Complexity of Wishes
That Alien Message (I think this one is particularly good)
Dreams of AI Design
Raised in Technophilia
Value is Fragile
There are lots more. Indeed, towards the latter half of the sequences it’s hard not to see an essay quite straightforwardly about AI Alignment every 2-3 essays.
My guess is that he meant the sequences convey the kind of more foundational epistemology which helps people people derive better models on subjects like AI Alignment by themselves, though all of the sequences in The Machine in the Ghost and Mere Goodness have direct object-level relevance.
Excepting Ngo’s AGI safety from first principles, I don’t especially like most of those resources as introductions exactly because they offer readers very little opportunity to test or build on their beliefs. Also, I think most of them are substantially wrong. (Concrete Problems in AI Safety seems fine, but is also skipping a lot of steps. I haven’t read Unsolved Problems in ML Safety.)
Out of curiosity, is this literally true? In particular, have you read David’s Sling?
I have read a good chunk of David’s Sling! Though it didn’t really click with me a ton, and I had already been spoiled on a good chunk of it because I had a bunch of conversations about it with friends, so I didn’t fully finish it.
For completeness sake, here is my reading state for all of the above:
Probability Theory: The Logic of Science, or some other probability theory textbook
Read
An introduction to general semantics
I read a bunch of general semantics stuff over the years, but I never really got into it, so a bit unclear.
An introduction to CBT, e.g., Feeling Good
Yep, read Feeling Good
Thinking Fast and Slow
Read
How to Measure Anything
Read
Surely You’re Joking Mr Feynman
Read
Stranger in a Strange Land, David’s Sling, The Moon is a Harsh Mistress.
All three read more than 30%. I think I finished Stranger in a Strange Land and Moon is a Harsh Mistress, but I honestly don’t remember.
Antifragile/Black Swan
Yep, read both
The Elephant in the Brain
Read like 50% of it, but got bored because a lot of it was covering Overcoming Bias stuff that I was familiar with.
Superforecasting, making 100 forecasts and keeping track each week.
Read superforecasting. Have made 100 forecasts, though haven’t been that great at keeping track.
The Rationality Quotient, or some other Keith Stanovich book.
Read 30% of it, then stopped because man, I think that book really was a huge disappointment. Would not recommend reading. See also this review by Stuart Ritchie: https://twitter.com/stuartjritchie/status/819140439827681280?lang=en
Some intro to nonviolent communication
Read Nonviolent Communication
Thanks for the comment and the list of recommendations. I have read most of the things on that list and the ones that I did read, I thought were great and have recommended a bunch to others, especially Probability Theory: The Logic of Science, The Elephant in the Brain, and forecasting practice. I agree that there are some dangers in recommending something that is pretty packaged but I think there is an obvious benefit in that it feels like a distilled version of reading a bunch of valuable things. Per unit time, I found reading the sequences more insightful/useful to me than the sources which it gets its ideas from (even if I read those things before the sequences, I am fairly confident).
I don’t want to oversell the sequences, I think the ideas in them have been mentioned in other places earlier. In my post, I mentioned specifically what ideas I found valuable so that people who are already familiar with them or think they are not that useful can decide not to read them. That isn’t rhetorical, I have some wise friends who are pretty well-read on philosophy and economics, and a lot of the things in the sequences I found novel, they were already familiar with.
My recommendation would look very different for someone who read the sequences and then made that their whole personality. I do know of some people who overrate them but in some of my specific circles (EA uni groups for eg), I think they are underrated/not given a chance which is why I wrote this post.
Would you recommend Probability Theory: The Logic of Science to people with little math background?
Well, if you are uncertain, note that experimentation here is very cheap, because you can download a copy of each book from an online library and quickly skim it to get a sense. So I’d recommend that.