Frankenstein (Mary Shelley): moral circle expansion to a human created AI, kinda.
Elizabeth Costello (J M Coetzee): novel about a professor who gives animal rights lectures. The chapter that’s most profoundly about animal ethics was published as “The Lives of Animals” which was printed with commentary from Peter Singer (in narrative form!).
Darkness at Noon (Arthur Koestler): Novel with reflections from an imprisoned old Bolshevik, reflecting on his past revolutionary activity. Interesting reflections on ends vs. means reasoning, and on weighing considerations of moral scale / the numbers affected vs personal emotional connection in moral tradeoff scenarios.
Zachary Brown
Karma: 197
Thanks for putting this together! Super helpful.
I really appreciated this post and it’s sequel (and await the third in the sequence)! The “second mistake” was totally new to me, and I hadn’t grasped the significance of the “first mistake”. The post did persuade me that the case for existential risk reduction is less robust than I had previously thought.
One tiny thing. I think this should read “from 20% to 10% risk”:
More rarely, we talk about absolute reductions, which subtract an absolute amount from the current level of risk. It is in this sense that a 10% reduction in risk takes us from 80% to 70% risk, from 20% to 18% risk, or from 10% to 0% risk. (Formally, relative risk reduction by f takes us from risk r to risk r – f).
Thanks for writing this! Hoping to respond more fully later.
In the meantime: I really like the example of what a “near-term AI-Governance factor collection could look like”.
So the question is ‘what governance hurdles decrease risk but don’t constitute a total barrier to entry?’
I agree. There are probably some kinds of democratic checks that honest UHNW individuals don’t mind, but have relatively big improvements for epistemics and community risk. Perhaps there are ways to add incentives for agreeing to audits or democratic checks? It seems like SBF’s reputation as a businessman benefited somewhat from his association with EA (I am not too confident in this claim). Perhaps offering some kind of “Super Effective Philanthropist” title/prize/trophy to particular UHNW donors that agree to subject their donations to democratic checks or financial audits might be an incentive? (I’m pretty skeptical, but unsure.) I’d like to do some more creative thinking here.
I wonder if submitting capital to your proposal seems a bit too much like the latter.
Probably.
I think this is a great post, efficiently summarizing some of the most important takeaways from recent events.
I think this claim is especially important:
“It’s also vital to avoid a very small number of decision-makers having too much influence (even if they don’t want that level of influence in the first place). If we have more sources of funding and more decision-makers, it is likely to improve the overall quality of funding decisions and, critically, reduce the consequences for grantees if they are rejected by just one or two major funders.”
Here’s a sketchy idea in that vein for further consideration. One additional way to avoid extremely wealthy donors having too much influence is to try to insist that UHNW donors subject their giving to democratic checks on their decision-making from other EAs. For instance, what if taking a Giving What We Can pledge entitled you to a vote of some kind on certain fund disbursements or other decisions? What if Giving What We Can pledgers could put forward “shareholder proposals” on strategic decisions (subject to getting fifty signatures, say) at EA orgs, which other pledgers could then vote on? (Not necessarily just at GWWC) Obviously there are issues:
voters may not be the epistemic peers of grantmaking experts / EA organization employees
voters may not be the epistemic peers of the UHNW donors themselves who have more reputational stake in ensuring their donations go well
UHNW donors have a lot of bargaining power when dealing with EA institutions and few incentives to open themselves up to democratic checks on their decision-making
determining who gets to vote is hard
some decisions need to be made quickly
sometimes there are infohazards
But there are advantages too, and I expect that often they outweigh the disadvantages:
wisdom of crowds
diversified incentives
democracy is a great look
This comment seems to be generating substantial disagreement. I’d be curious to hear from those who disagree: which parts of this comment do you disagree with, and why?
Hi Cesar! You might be interested to check out the transparency page for the Against Malaria Foundation: https://www.againstmalaria.com/transparency.aspx
I’d be interested in surveying on whether people believe that AI [could presently/might one day] do a better job governing the [United States/major businesses/US military/other important institutions] than [elected leaders/CEOs/generals/other leaders].
I don’t think this is true. Dunbar’s number is a limit on the number of social relationships an individual can cognitively sustain. But the sorts of networks needed to facilitate productive work are different than those needed to sustain fulfilling social relations. If there is a norm that people are willing to productively collaborate with the unknown contact of a known contact, then surely you can sustain a productive community with approx Dunbar’s number ^2 people (if each member of my Dunbar-sized community has their own equivalently-sized community with no shared members).
Thanks for contributing this critique, your invitation for argument, and your open-mindedness!
I think one important inequality in the distribution of power is that between presently living people and future generations. The latter have not only no political power, but no direct causal power at all. While we might decry a world where we have to persuade or compel billionaires -- or seek to become billionaires ourselves—to have much hope at large-scale influence, these tools are much better than anything future generations have got. Our power over future generations is asymmetric and terrifying: their mere existence may depend on our present choices. To the extent that we might care about the distribution of power intrinsically and not just because of the effects on welfare (I don’t personally find this view compelling), it seems like the highest priority redistributions of power are to those who have the least at present. One avenue of EA research I am excited about focuses on how we can build institutions and new systems of power to represent the interests of future generations in present political arrangements. You might also be interested in this analysis of opportunities for improving institutions by the Effective Institutions Project—which I think is very good EA writing on power.
Animals find themselves in a somewhat similar political situation to future generations: that is, basically powerless. Albeit for different reasons, of course.
Yes, and how many people we project will have this association in the future. I think it’s reasonably likely that this view will pick up steam among vaguely activisty people on college campuses in the next five years. That’s an important demographic for growing EA.
Great piece, I thought. I think Carrick Flynn’s loss may in no small part be due to accidentally cultivating a white crypto-bro aesthetic. If that’s right, it is a case of aesthetics mattering a fair amount. Personally, I’d like to see EA do more to avoid donning this aesthetic, which anecdotally seems to turn a lot of people off.
I’d be a little bit concerned by this. I think there’s a growing sentiment among young people (especially on university campuses) that classicism is aesthetically: regressive, retrograde, old-white-man stuff. Here’s a quote from a recent New York Times piece:
“Long revered as the foundation of “Western civilization,” [classics] was trying to shed its self-imposed reputation as an elitist subject overwhelmingly taught and studied by white men. Recently the effort had gained a new sense of urgency: Classics had been embraced by the far right, whose members held up the ancient Greeks and Romans as the originators of so-called white culture. Marchers in Charlottesville, Va., carried flags bearing a symbol of the Roman state; online reactionaries adopted classical pseudonyms; the white-supremacist website Stormfront displayed an image of the Parthenon alongside the tagline “Every month is white history month.””
Edit: this is a criticism of classicism as a useful aesthetic, not of the enlightenment. Potentially they’re severable.
I’m curious whether community size, engagement level, and competence might matter less than the general perception of EA among non-EAs.
Not just because low general positive perception of EA makes it harder to attract highly engaged, competent EAs. But also because general positive perception matters even if it never results in conversion. General positive perception increases our ability to cooperate with and influence non-EA individuals and institutions.
Suppose an aggressive community building tactic attracts one HEA, of average competence. In addition, it gives a number of people n a slightly negative view of EA—not a strongly felt opposition, just enough of a dislike that they mention it in conversations with other non-EAs sometimes. What n would we accept to make this community building tactic expected value neutral? (This piece seems to suggest that many current strategies fit this model.)
I’m currently evaluating the feasibility and expected value of building a proxy voting advisory firm that would make EA-aligned voting recommendations. Would love to meet with you or anyone with expertise.
I think the virtues of moral expansiveness and altruistic sympathy for moral patients are really important for EAs to develop, and I think being vegan increased my stock of these virtues by reversing the “moral dulling” effect you postulate. (This paper makes the case for utilitarians to develop a set of similar virtues: https://psyarxiv.com/w52zm.) I’ve also developed a visceral disgust response to meat as a result of being vegan, which is for me probably inseparable from the motivating feeling of sympathy for animals as moral patients.
When I was a nonvegan, I underestimated the extent to which eating meat was morally dulling to me, and I suspect this is common. It was hard to know how morally dulled I was until I experienced otherwise.
If a community claims to be altruistic, it’s reasonable for an outsider to seek evidence: acts of community altruism that can’t be equally well explained by selfish impulses, like financial reward or desire for praise. In practice, that seems to require that community members make visible acts of personal sacrifice for altruistic ends. To some degree, EA’s credibility as a moral movement (that moral people want to be a part of) depends on such sacrifices. GWWC pledges help; as this post points out, big spending probably doesn’t.
One shift that might help is thinking more carefully about who EA promotes as admirable, model, celebrity EAs. Communities are defined in important ways by their heroes and most prominent figures, who not only shape behaviour internally, but represent the community externally. Communities also have control over who these representatives are, to some degree: someone makes a choice over who will be the keynote speaker at EA conferences, for instance.
EA seems to allocate a lot of its prestige and attention to those it views as having exceptional intellectual or epistemic powers. When we select EA role models and representatives, we seem to optimise for demonstrated intellectual productivity. But our selections are not necessarily the people who have made the greatest personal altruistic sacrifices. Often, they’re researchers who live in relative luxury—even if they’ve taken a GWWC pledge. Perhaps we should be more conscious to elevate the EA profile of people like those in MacFarquhar’s Strangers Drowning : people who have made exceptional sacrifices to make the world better, rather than people who have been most successful at producing EA-relevant intellectual output. Maybe the keynote speaker at the next EA conference should be someone who once undertook an effective hunger strike, say. (Maybe even regardless of whether they have heard of EA, or consider themselves EA.)
There’s an obvious reason to instead continue EA’s current role model selection strategy: having a talk from a really clever researcher is helpful for internal community epistemics. We want to grant speaking platforms to those who might be able to offer the most valuable information or best thought-through view. And it’s valuable for the external reputation of our community epistemics to have such people be the face of EA. We also don’t want to promote the idea that the size of one’s sacrifice is what ultimately matters.
But there are internal and external reasons to choose a role model based on the degree of inspiring altruistic sacrifice that person has made, too. Just as Will MacAskill can make me a little more informed, or guide my thinking in a slightly better direction, an inspiring story of personal sacrifice can make me a little more dedicated, a little more willing to work hard and sacrifice to make the world better. And externally, such a role model signals community focus on altruistic commitment.My low-confidence guess is that the optimum allocation of prestige still gives most EA attention and admiration to those with greatest demonstrated intellectual or epistemic power—but not all. Those who’ve demonstrated acts of moral sacrifice should be held up as exemplars too, especially in external-facing contexts.
Proportional Chances Voting is basically equivalent to a mechanism where one vote is selected at random to be the deciding vote, as Newberry and Ord register in a footnote (they refer to it as “Random Dictator”; I’ve also seen it described as “lottery voting”). Newberry and Ord do say that Proportional Chances is supposed to be different because of the negotiation period, but I don’t see how Random Dictator is incompatible with negotiation.
Anyway, some of the literature on this mechanism may be of interest here, given footnotes 8-9. This paper proposes such a mechanism, defends its plausibility: Saunders, Ben. “Democracy, Political Equality, and Majority Rule.” Ethics 121, no. 1 (2010): 148–77. I haven’t read any good papers which offer interesting critiques of Saunders, but the paper seems to be influential, so maybe someone else knows of one?
As for calling new votes (footnote 9 of this post), votes could be scheduled by a separate body than that doing the voting, or could be scheduled by some regularised rule. For instance, in Kira’s Dinner, the thought experiment in the Newberry and Ord paper, votes on what Kira should eat are scheduled according to the regular rhythm of Kira getting hungry. The voters take the votes as given—I think there are usually similar ways to establish systems like this in real-world multi-person organizations.
I’m in the early stages of corporate campaign work similar to what’s discussed in this post. I’m trying to mobilise investor pressure to advocate for safety practices at AI labs and chipmakers. I’d love to meet with others working on similar projects (or anyone interested in funding this work!). I’d be eager for feedback.
You can see a write-up of the project here.