EAs/rationalists already have strong defenses against the latter.
I think it could be stronger! While there’s a pronounced norm in EA against overconfidence, that’s not the same as having “strong defenses.” Social signalling is anti-inductive in the sense that the signals that work to trick us into being unduly impressed or intimidated by others’ perceived competence cannot be easily summarized with simple rules for what to avoid or look out for. (In other words, even with strong modesty norms, a con man or cult leader could score modesty points wherever doing so is cheap while subtly working in confidence tricks with the way they frame things and with their narrative.)
Besides, one of the reasons conmen can successfwly amass followers is that the followers lack confidence.
That doesn’t sound right to me. I think you can be confident and still follow someone you think is deserving of it. Besides, confidence is empty/shallow without the domain-appropriate competence to back it up. In the case of “who would you follow?,” the skill in question seems to be people judgment. One part of people judgment is being confident in your own judgment; another part is being the right amount of cynical.
In any case, I think we might be discussing a false dilemma. I’m personally reading your message as “it’s important to encourage/nurture people’s confidence when they have a vision and seem willing to take on ambitious or even heroic projects to make the world better.” I’m all for it! I just want to add “and make sure to check if the person seems to have high integrity.” (“Checking” in the sense of “have your eyes open” rather than “be mistrustful from the get go / don’t give them a chance.”)
I agree with you that EA aims seem bottlenecked by people with vision. I am skeptical that people capable of becoming people with vision are easily deterred by social norms (I’m reminded of the chapters in HPMOR around Hermoine wanting to become a heroine). But, admittedly, that doesn’t mean that the effect is zero.
Or maybe you don’t see it as a false dilemma and you disagree with me about where we’re at with the social pendulum swing, so maybe you consider it important to push the pendulum back as hard as possible. If so, we may find ourselves on ~25% opposed sides on the battlefield over norms. Oh well. Disagreements over “social pendulum dynamics” seem notoriously intractable to me. (My pet theory is that people’s strongest opinions on social norms, their “most sacred virtues,” are often influenced by small t-traumas from things that went particularly poorly for them or for people they care about, and there’s so much variation and situations where opposite types of advice is good for different people.)
“Social signalling is anti-inductive in the sense that the signals that work to trick us into being unduly impressed or intimidated by others’ perceived competence cannot be easily summarized with simple rules for what to avoid or look out for.”
Interesting! Good point.
- - - - -
If you follow someone because you have confidence in your ability to evaluate whether they’re worth following, then maybe you should follow them. If you follow someone to compensate for your lack of confidence, that’s less likely to produce good consequences. I “follow” some people because I’m confident in their abilities, e.g. I think Eliezer is worth trying to learn from. I think he’s also worth deferring to if you choose not to be an explorer and instead someone who wishes to spend their time Doing and less time figuring out stuff.
“I am skeptical that people capable of becoming people with vision are easily deterred by social norms”
I’d like to believe this, but I strongly disagree. I read a comment from Steven Byrnes saying that he (iirc) was was held back by being really uncertain for a long time about whether he had anything usefwl to contribute to the community. I know some other people I think have the power to guide themselves to great things, but who don’t fully trust themselves enough.
Personally, I spent years thinking I was purely “catching up” to what everyone else already knows. Every new idea I had was “ah, so this is what the smart people think already”. It took someone who believed in me to make me feel safe pursuing my own path. It’s really hard to be motivated when you’re constantly questioning yourself, even when you try to pursue independence.
I want to push the confidence pendulum way further than it has ever been. I want people to pursue independence, ambition, and self-sacrifice until the road is littered with failed or delusional projects. This is how you sample for outliers on a fat-tailed distribution and double the number of Bankman-Frieds, Yudkowskys, and Borlaugs.
But I have a bit more nuance in what I mean by “confidence”. I don’t mean unjustified probability estimates. I don’t mean lack of curiosity in other people’s viewpoints. I don’t mean the stereotypical social role, I just mean the stuff that’s usefwl for independently pursuing ambitious things with motivation and less wasted motion.
I think it could be stronger! While there’s a pronounced norm in EA against overconfidence, that’s not the same as having “strong defenses.” Social signalling is anti-inductive in the sense that the signals that work to trick us into being unduly impressed or intimidated by others’ perceived competence cannot be easily summarized with simple rules for what to avoid or look out for. (In other words, even with strong modesty norms, a con man or cult leader could score modesty points wherever doing so is cheap while subtly working in confidence tricks with the way they frame things and with their narrative.)
That doesn’t sound right to me. I think you can be confident and still follow someone you think is deserving of it. Besides, confidence is empty/shallow without the domain-appropriate competence to back it up. In the case of “who would you follow?,” the skill in question seems to be people judgment. One part of people judgment is being confident in your own judgment; another part is being the right amount of cynical.
In any case, I think we might be discussing a false dilemma. I’m personally reading your message as “it’s important to encourage/nurture people’s confidence when they have a vision and seem willing to take on ambitious or even heroic projects to make the world better.” I’m all for it! I just want to add “and make sure to check if the person seems to have high integrity.” (“Checking” in the sense of “have your eyes open” rather than “be mistrustful from the get go / don’t give them a chance.”)
I agree with you that EA aims seem bottlenecked by people with vision. I am skeptical that people capable of becoming people with vision are easily deterred by social norms (I’m reminded of the chapters in HPMOR around Hermoine wanting to become a heroine). But, admittedly, that doesn’t mean that the effect is zero.
Or maybe you don’t see it as a false dilemma and you disagree with me about where we’re at with the social pendulum swing, so maybe you consider it important to push the pendulum back as hard as possible. If so, we may find ourselves on ~25% opposed sides on the battlefield over norms. Oh well. Disagreements over “social pendulum dynamics” seem notoriously intractable to me. (My pet theory is that people’s strongest opinions on social norms, their “most sacred virtues,” are often influenced by small t-traumas from things that went particularly poorly for them or for people they care about, and there’s so much variation and situations where opposite types of advice is good for different people.)
Interesting! Good point.
- - - - -
If you follow someone because you have confidence in your ability to evaluate whether they’re worth following, then maybe you should follow them. If you follow someone to compensate for your lack of confidence, that’s less likely to produce good consequences. I “follow” some people because I’m confident in their abilities, e.g. I think Eliezer is worth trying to learn from. I think he’s also worth deferring to if you choose not to be an explorer and instead someone who wishes to spend their time Doing and less time figuring out stuff.
I’d like to believe this, but I strongly disagree. I read a comment from Steven Byrnes saying that he (iirc) was was held back by being really uncertain for a long time about whether he had anything usefwl to contribute to the community. I know some other people I think have the power to guide themselves to great things, but who don’t fully trust themselves enough.
Personally, I spent years thinking I was purely “catching up” to what everyone else already knows. Every new idea I had was “ah, so this is what the smart people think already”. It took someone who believed in me to make me feel safe pursuing my own path. It’s really hard to be motivated when you’re constantly questioning yourself, even when you try to pursue independence.
I want to push the confidence pendulum way further than it has ever been. I want people to pursue independence, ambition, and self-sacrifice until the road is littered with failed or delusional projects. This is how you sample for outliers on a fat-tailed distribution and double the number of Bankman-Frieds, Yudkowskys, and Borlaugs.
But I have a bit more nuance in what I mean by “confidence”. I don’t mean unjustified probability estimates. I don’t mean lack of curiosity in other people’s viewpoints. I don’t mean the stereotypical social role, I just mean the stuff that’s usefwl for independently pursuing ambitious things with motivation and less wasted motion.