Yeah, I think there’s a lot of interesting things to be said here. Some points:
-I’m not sure why we should expect the group to be well-calibrated when no individual is? That sounds a little magical to me. It’s not a market dynamic where ground-level feedback directly lowers the viability/status of incorrect hypotheses, and so leads to the collective being smarter than any individual—since we get no direct feedback about our effect on the long-term future whatsoever, and certainly none that would literally force us to stop what we’re doing (like not being in touch with what the market wants being able to cause your business to fail).
It feels more viable to me for every individual to act with epistemic virtue (or explicitly defer to someone who does so).
-For a slightly tangential point, it’s interesting to think about what the optimal social structures of deferral would be, but I’ll note that my guess is that the most influential people also tend to be among the people who work/try the hardest? That’s a big part of why people are successful, after all. So if anything, this is a cause for more worry.
-On your “other point”: Yeah, I certainly don’t endorse doing nothing. The epistemic distortion is just one consideration that happens to push towards doing less (since it lowers the EV of doing anything, if we haven’t considered it before). It needs to be weighed against the many other considerations that exist.
“I’m not sure why we should expect the group to be well-calibrated when no individual is?”—Something something marketplace of ideas? An analogy is a court, where the prosecution and the defence both have their conclusions assigned to them beforehand. They are both epistemically vicious, in opposite ways. Then the idea is that the best arguments win on their merits. I’m not sure quite how this analogy would fit though (I’ve had false starts writing a blog post on this a few times).
“I certainly don’t endorse doing nothing” Yep I get that this argument is only one consideration, but my point extends to “trying less hard” as well I think.
“Yep I get that this argument is only one consideration, but my point extends to “trying less hard” as well I think.” I’m not sure I understand what you mean, could you explain it more?
”Something something marketplace of ideas?” Yeah, this is an empirical question—theoretically, we might see better arguments rise in status and influence people more, and of course to some extent we do (e.g. AI risk rising in status over the last 10 years). But there are other factors influencing the status of ideas too, like some kind of general action-bias / power-seeking-bias. Concretely, I feel that most of the bullet points in the “uncertainty” section of the post are underrepresented in the discourse—curious if you disagree.
Also, one more point on the deferral thing—one way in which I think I’m weird is that I would genuinely raise my esteem of someone (on a gut-level) if they said “I’m hopelessly biased on this topic, I can’t think about it, don’t listen to me”. Unfortunately, you never see this. I would really like to see that more. E.g. if someone who tries extremely hard said “I can’t really clearly think about what I’m doing because I’m working so hard, so be careful about listening to me” it would be very beautiful to me. Poetically speaking, it would be… accepting that they are a human weapon, forged for a purpose, impaired by that sacrifice, baring it for the world to see. There would be a slight feeling of heartbreak and love for them in me, and I might very well value them more than before. As I argue in the “Epistemic distortion” section, there’s at least to some extent a deep tradeoff between doing and thinking—so admitting that they are trading off against thinking, crippling their mind on a deep level, could make people respect them more by showing how much they are sacrificing for the “doing” status hierarchy. It could be heroic. That’s just a fantasy I have about how things could work.
Thanks Toby!
Yeah, I think there’s a lot of interesting things to be said here. Some points:
-I’m not sure why we should expect the group to be well-calibrated when no individual is? That sounds a little magical to me. It’s not a market dynamic where ground-level feedback directly lowers the viability/status of incorrect hypotheses, and so leads to the collective being smarter than any individual—since we get no direct feedback about our effect on the long-term future whatsoever, and certainly none that would literally force us to stop what we’re doing (like not being in touch with what the market wants being able to cause your business to fail).
It feels more viable to me for every individual to act with epistemic virtue (or explicitly defer to someone who does so).
-For a slightly tangential point, it’s interesting to think about what the optimal social structures of deferral would be, but I’ll note that my guess is that the most influential people also tend to be among the people who work/try the hardest? That’s a big part of why people are successful, after all. So if anything, this is a cause for more worry.
-On your “other point”: Yeah, I certainly don’t endorse doing nothing. The epistemic distortion is just one consideration that happens to push towards doing less (since it lowers the EV of doing anything, if we haven’t considered it before). It needs to be weighed against the many other considerations that exist.
Curious what you think!
“I’m not sure why we should expect the group to be well-calibrated when no individual is?”—Something something marketplace of ideas? An analogy is a court, where the prosecution and the defence both have their conclusions assigned to them beforehand. They are both epistemically vicious, in opposite ways. Then the idea is that the best arguments win on their merits. I’m not sure quite how this analogy would fit though (I’ve had false starts writing a blog post on this a few times).
“I certainly don’t endorse doing nothing” Yep I get that this argument is only one consideration, but my point extends to “trying less hard” as well I think.
“Yep I get that this argument is only one consideration, but my point extends to “trying less hard” as well I think.” I’m not sure I understand what you mean, could you explain it more?
”Something something marketplace of ideas?”
Yeah, this is an empirical question—theoretically, we might see better arguments rise in status and influence people more, and of course to some extent we do (e.g. AI risk rising in status over the last 10 years). But there are other factors influencing the status of ideas too, like some kind of general action-bias / power-seeking-bias. Concretely, I feel that most of the bullet points in the “uncertainty” section of the post are underrepresented in the discourse—curious if you disagree.
Also, one more point on the deferral thing—one way in which I think I’m weird is that I would genuinely raise my esteem of someone (on a gut-level) if they said “I’m hopelessly biased on this topic, I can’t think about it, don’t listen to me”. Unfortunately, you never see this. I would really like to see that more.
E.g. if someone who tries extremely hard said “I can’t really clearly think about what I’m doing because I’m working so hard, so be careful about listening to me” it would be very beautiful to me. Poetically speaking, it would be… accepting that they are a human weapon, forged for a purpose, impaired by that sacrifice, baring it for the world to see. There would be a slight feeling of heartbreak and love for them in me, and I might very well value them more than before. As I argue in the “Epistemic distortion” section, there’s at least to some extent a deep tradeoff between doing and thinking—so admitting that they are trading off against thinking, crippling their mind on a deep level, could make people respect them more by showing how much they are sacrificing for the “doing” status hierarchy. It could be heroic.
That’s just a fantasy I have about how things could work.