This solidifies a conclusion for me: when talking about AI risk, the best/most rigorous resources aren’t the ones which are most widely shared/recommended (rigorous resources are e.g. Ajeya Cotra’s report on AI timelines, Carlsmith’s report on power-seeking AI, Superintelligence by Bostrom or (to a lesser extent) Human Compatible by Russell).
Those might still not be satisfying to skeptics, but are probably more satisfying than ” short stories by Eliezer Yudkowsky” (though one can take an alternative angle: skeptics wouldn’t bother reading a >100 page report, and I think the complaint that it’s all short stories by Yudkowsky comes from the fact that that’s what people actually read).
Additionally, there appears to be a perception that AI safety research is limited to MIRI & related organisations, which definitely doesn’t reflect the state of the field—but from the outside this multipolarity might be hard to discover (outgroup-ish homogeneity bias strikes again).
Personally I find Human Compatible the best resource of the ones you mentioned. If it were just the others I’d be less bought into taking AI risk seriously.
I agree that it occupies a spot on the layperson-understandability/rigor Pareto-frontier, but that frontier is large and the other things I mentioned are at other points.
Indeed. It just felt more grounded in reality to me than the other resources which may appeal more to us laypeople and the non laypeople prefer more speculative and abstract material.
Seconded/thirded on Human Compatible being near that frontier. I did find its ending ‘overly optimistic’ in the sense of framing it like ‘but lo, there is a solution!’ while other similar resources like Superintelligence and especially The Alignment Problem seem more nuanced in presenting uncertain proposals for paths forward not as oven-ready but preliminary and speculative.
I’m not quite sure I read the first two paragraphs correctly. Are you saying that Cotra, Carlsmith and Bostrom are the best resources but they are not widely recommended? And people mostly read short posts, like those by Eliezer, and those are accessible but might not have the right angle for skeptics?
Yes, I think that’s a fair assessment of what I was saying.
Maybe I should have said that they’re not widely recommended enough on the margin, and that there are surely many other good & rigorous-ish explanations of the problem out there.
I’m also always disappointed when I meet EAs who aren’t deep into AI safety but curious, and the only things they have read is the List of Lethalities & the Death with Dignity post :-/ (which are maybe true but definitely not good introductions to the state of the field!)
As a friendly suggestion, I think the first paragraph of your original comment would be less confusing if the parenthetical clause immediately followed “the best/most rigorous resources”. This would make it clear to the reader that Cotra, Carlsmith, et al are offered as examples of best/most rigorous resources, rather than as examples of resources that are widely shared/recommended.
There are short stories by Yudkowsky? All I ever encountered were thousands-of-pages-long sequences of blog posts (which I hence did not read, as you suggest).
If you’re unconvinced about AI danger and you tell me what specifically are your cruxes, I might be able to connect you with Yudkowskian short stories that address your concerns.
I think I would have found Ajeya’s cold takes guest post on “Why AI alignment could be hard with modern deep learning” persuasive back when I was skeptical. It is pretty short. I think the reason why I didn’t find what you call “short stories by Eliezer Yudkowsky” persuasive was because they tended to not use concepts / terms from ML. I guess even stuff like orthogonality thesis and instrumental convergence thesis was not that convincing to me on a gut level even though I didn’t disagree with the actual argument for them because I had the intuition that whether misaligned AI was a big deal depended on details of how ML actually worked, which I didn’t know. To me back then it looked like most people I knew with much more knowledge of ML were not concerned about AI x-risk so probably it wasn’t a big deal.
This solidifies a conclusion for me: when talking about AI risk, the best/most rigorous resources aren’t the ones which are most widely shared/recommended (rigorous resources are e.g. Ajeya Cotra’s report on AI timelines, Carlsmith’s report on power-seeking AI, Superintelligence by Bostrom or (to a lesser extent) Human Compatible by Russell).
Those might still not be satisfying to skeptics, but are probably more satisfying than ” short stories by Eliezer Yudkowsky” (though one can take an alternative angle: skeptics wouldn’t bother reading a >100 page report, and I think the complaint that it’s all short stories by Yudkowsky comes from the fact that that’s what people actually read).
Additionally, there appears to be a perception that AI safety research is limited to MIRI & related organisations, which definitely doesn’t reflect the state of the field—but from the outside this multipolarity might be hard to discover (outgroup-ish homogeneity bias strikes again).
Personally I find Human Compatible the best resource of the ones you mentioned. If it were just the others I’d be less bought into taking AI risk seriously.
I agree that it occupies a spot on the layperson-understandability/rigor Pareto-frontier, but that frontier is large and the other things I mentioned are at other points.
Indeed. It just felt more grounded in reality to me than the other resources which may appeal more to us laypeople and the non laypeople prefer more speculative and abstract material.
Seconded/thirded on Human Compatible being near that frontier. I did find its ending ‘overly optimistic’ in the sense of framing it like ‘but lo, there is a solution!’ while other similar resources like Superintelligence and especially The Alignment Problem seem more nuanced in presenting uncertain proposals for paths forward not as oven-ready but preliminary and speculative.
I’m not quite sure I read the first two paragraphs correctly. Are you saying that Cotra, Carlsmith and Bostrom are the best resources but they are not widely recommended? And people mostly read short posts, like those by Eliezer, and those are accessible but might not have the right angle for skeptics?
Yes, I think that’s a fair assessment of what I was saying.
Maybe I should have said that they’re not widely recommended enough on the margin, and that there are surely many other good & rigorous-ish explanations of the problem out there.
I’m also always disappointed when I meet EAs who aren’t deep into AI safety but curious, and the only things they have read is the List of Lethalities & the Death with Dignity post :-/ (which are maybe true but definitely not good introductions to the state of the field!)
As a friendly suggestion, I think the first paragraph of your original comment would be less confusing if the parenthetical clause immediately followed “the best/most rigorous resources”. This would make it clear to the reader that Cotra, Carlsmith, et al are offered as examples of best/most rigorous resources, rather than as examples of resources that are widely shared/recommended.
Thanks, will edit.
There are short stories by Yudkowsky? All I ever encountered were thousands-of-pages-long sequences of blog posts (which I hence did not read, as you suggest).
Lots of it is here
If you’re unconvinced about AI danger and you tell me what specifically are your cruxes, I might be able to connect you with Yudkowskian short stories that address your concerns.
The ones which come immediately to mind are:
That Alien Message
Sorting Pebbles Into Correct Heaps
I think I would have found Ajeya’s cold takes guest post on “Why AI alignment could be hard with modern deep learning” persuasive back when I was skeptical. It is pretty short. I think the reason why I didn’t find what you call “short stories by Eliezer Yudkowsky” persuasive was because they tended to not use concepts / terms from ML. I guess even stuff like orthogonality thesis and instrumental convergence thesis was not that convincing to me on a gut level even though I didn’t disagree with the actual argument for them because I had the intuition that whether misaligned AI was a big deal depended on details of how ML actually worked, which I didn’t know. To me back then it looked like most people I knew with much more knowledge of ML were not concerned about AI x-risk so probably it wasn’t a big deal.