But I doubt you can make a case that’s robustly compelling and is widely agreed upon, enough to prevent the dynamics I worry about above.
“I doubt you can make a case that’s robustly compelling...”
Systemic cascading effects and path dependency might be very coherent consequentialist frameworks & catchphrases to resolve a lot of your epistemic concerns (and this is something I want to explore further).
Naive consequentialism might incentivize you to lie to “do whatever it takes to do good”, but the impacts of lying can cascade and affect the bedrock institutional culture and systems of a movement. On aggregate, these cascading (second-order) effects will make it more difficult for people to trust each other and work together in honest ways, making the moral calculus not worth it.
Furthermore, this might have a path-dependent effect, analogous to a significant/persistent/contingent effect, where choosing this path encodes certain values in the institution and makes it harder for other community values to arise in the future.
This similarly generalizes to most “overoptimization becomes illogical” problems. Naive consequentialism & low-integrity epistemics rarely make sense in the long run anyways, so it’s just a matter of dispelling simplified, naive models of reality and coherently phrasing the importance of epistemics, diversity, and plurality through a consequentialist lens.
″...and is widely agreed upon.”
Still relatively new to the community, so I might have the wrong view on this—but I’m always remarkably surprised by how openly EAs are willing to discuss flaws in the community & are concerned about solid epistemics within the community.
E.g. recently just posted a submission to the EA criticism contest—and it’s difficult for me to imagine any other subgroup which pours $100k into a contest seriously considering and rewarding internal & external criticism about its most fundamental values and community.
There’s another problem with the norm of lying for the greater good: One, it is very easy for biased human minds to convince themselves of the lie and become systematically distorted from their path. To put it in Sarah Constantin’s words:
The problem with this reasoning should be obvious. The argument would work just as well if EA did no good at all, and only claimed to do good.
Arbitrary or unreliable claims of moral superiority function like bubbles in economic markets. If you never check the value of a stock against some kind of ground-truth reality, if everyone only looks at its current price and buys or sells based on that, we’ll see prices being inflated based on no reason at all. If you don’t insist on honesty in people’s claims of “for the greater good”, you’ll get hijacked into helping people who aren’t serving the greater good at all.
Another problem is you are much more vulnerable to Goodharting yourself, and eventually you will use it for motivated reasoning, where your pet causes can be lied about, and outsiders can’t tell if the organization is actually doing what it claims. While I think the dentological notion of honesty is too exploitable and naive for the 21st century, I definitely agree with Holden that lying should not be a norm, as well as misleading people should also not be a norm, but a regrettable exception.
I disagree with the following:
“I doubt you can make a case that’s robustly compelling...”
Systemic cascading effects and path dependency might be very coherent consequentialist frameworks & catchphrases to resolve a lot of your epistemic concerns (and this is something I want to explore further).
Naive consequentialism might incentivize you to lie to “do whatever it takes to do good”, but the impacts of lying can cascade and affect the bedrock institutional culture and systems of a movement. On aggregate, these cascading (second-order) effects will make it more difficult for people to trust each other and work together in honest ways, making the moral calculus not worth it.
Furthermore, this might have a path-dependent effect, analogous to a significant/persistent/contingent effect, where choosing this path encodes certain values in the institution and makes it harder for other community values to arise in the future.
This similarly generalizes to most “overoptimization becomes illogical” problems. Naive consequentialism & low-integrity epistemics rarely make sense in the long run anyways, so it’s just a matter of dispelling simplified, naive models of reality and coherently phrasing the importance of epistemics, diversity, and plurality through a consequentialist lens.
″...and is widely agreed upon.”
Still relatively new to the community, so I might have the wrong view on this—but I’m always remarkably surprised by how openly EAs are willing to discuss flaws in the community & are concerned about solid epistemics within the community.
E.g. recently just posted a submission to the EA criticism contest—and it’s difficult for me to imagine any other subgroup which pours $100k into a contest seriously considering and rewarding internal & external criticism about its most fundamental values and community.
There’s another problem with the norm of lying for the greater good: One, it is very easy for biased human minds to convince themselves of the lie and become systematically distorted from their path. To put it in Sarah Constantin’s words:
Another problem is you are much more vulnerable to Goodharting yourself, and eventually you will use it for motivated reasoning, where your pet causes can be lied about, and outsiders can’t tell if the organization is actually doing what it claims. While I think the dentological notion of honesty is too exploitable and naive for the 21st century, I definitely agree with Holden that lying should not be a norm, as well as misleading people should also not be a norm, but a regrettable exception.