I am a lawyer and policy researcher interested in improving the governance of artificial intelligence. I currently work as Director of Research at the Institute for Law & AI. I previously worked in various legal and policy roles at OpenAI.
I am also a Research Affiliate with the Centre for the Governance of AI and a VP at the O’Keefe Family Foundation.
My research focuses on the law, policy, and governance of advanced artificial intelligence.
You can share anonymous feedback with me here.
Ah sorry, I read your post too quickly :-)
There used to be a website to try to coordinate this; not sure what ever happened to it.
I also want to point out that having better outside income-maximizing options makes you more financially secure than other people in your income bracket, all else equal, which pro tanto would give you more reason to donate than them.
I’m not going to defend my whole view here, but I want to give a though experiment as to why I don’t think that “shadow donations”—the delta between what you could earn if you were income-maximizing, and what you’re actually earning in your direct work job—are a great measure for the purposes of practical philosophy (though I agree they’re both a relevant consideration and a genuine sacrifice).
Imagine two twins, Anna and Belinda. Both have just graduated with identical grades, skills, degrees, etc. Anna goes directly from college work on AI safety at Safety Org, making $75,000 / year. Belinda goes to work for OpenMind doing safety-neutral work, making $1M per year total compensation. Belinda learns more marketable skills; she could make at least $1M / year indefinitely. Anna, on the other hand, has studiously plugged away at AI safety work, but since her work is niche, she can’t easily transfer these skills to do something that pays better.
Then imagine that, after three years, Belinda joins Anna at Safety Org. Belinda was not fired; she could have stayed at OpenMind and made $1M per year indefinitely. At this point, Anna has gotten a few raises and is making $100,000, and donating 3% of her salary. Belinda gets the same job on the same pay scale, and does equally good work, but donates nothing. Belinda reasons that, because she could still be $1M per year, she has “really” donated $900,000 of labor to Safety Org, and so has sacrificed roughly 90% of her income.
Anna, on the other hand, thinks that it is an immense privilege to be able to have a comfortable job where she can use her skills to do good, while still earning more than 99% of all people in the world. She knows that, if she had made different choices in life, she probably could have a higher earning potential. But that has never been her goal in life. Anna knows that the average person in her income bracket donate around 3% regardless of their outside job options, so it seems reasonable for her to at least match that.
Is Belinda more altruistic than Anna? Which attitude should EAs aspire to?
To give some more color on my general view:
I don’t really think there’s a first-order fact of the matter as to who of these two (or anyone) is “more altruistic,” or what one’s “obligations” are. At bottom, there are just worlds with more or less value in them.
My view mostly comes from a practical view of how the EA community and project can be most impactful, credible, and healthy. I think the best attitude is closer to Anna’s than Belinda’s.
Donating also has other virtues over salary reductions, since it is concrete, measurable, and helps create a more diversified funding ecosystem.
To be clear, I think it’s great that people like Belinda exist, and they should be welcomed and celebrated in the community. But I don’t think the particular mindset of “well I have really sacrificed a lot because if I was purely selfish I could have made a lot more money” is one that we ought to recognize as particularly good or healthy.
It’s not clear to me whether you’re talking about people who (a) do a voluntary salary sacrifice while working at an EA org, or (b) people who could have earned much more in industry but moved to a nonprofit so now earn much less than their hypothetical maximum earning potential.
In case (a), yes, their salary sacrifice should count towards their real donations.
But I think a practical moral philosophy wherein donation expectations are based on your actual material resources (and constraints), not your theoretical maximum earning potential, seems more justifiable. So I don’t think people who do (b) (which includes myself) should get to say that doing (b) liberates them from the same obligation to donate that would attend to a person in the same material circumstances with worse outside options.
Is your claim that somehow FTX investing in Anthropic has caused Anthropic to be FTX-like in the relevant ways? That seems implausible.
Thanks for this very thoughtful reply!
I have a lot to say about this, much of which boils down to a two points:
I don’t think Jeremy is a good example of unnecessary polarization.
I think “avoid unnecessary polarization” is a bad heuristic for policy research (which, related to my first point, is what Jeremy was responding to in Dislightenment), at least if it means anything other than practicing the traditional academic virtues of acknowledging limitations, noting contrary opinion, being polite, being willing to update, inviting disagreement, etc.
The rest of your comment I agree with.
I realize that point (1) may seem like nitpicking, and that I am also emotionally invested in it for various reasons. But this is all in the spirit of something like avoiding reasoning from fictional evidence: if we want to have a good discussion of avoiding unnecessary polarization, we should reason from clear examples of it. If Jeremy is not a good example of it, we should not use him as a stand-in.
I was just using Jeremy as a stand-in for the polarisation of Open Source vs AI Safety more generally.
Right, this is in large part where our disagreement is: whether Jeremy is good evidence for or an example of unnecessary polarization. I just simply don’t think that Jeremy is a good example of where there has been unnecessary (more on this below) polarization because I think that he, explicitly and somewhat understanably, just finds the idea of approval regulation for frontier AI abhorrent. So to use Jeremy as evidence or example of unnecessary polarization, we have to ask what he was reacting to, and whether something unnecessary was done to polarize him against us.
Dislightenment “started out as a red team review” of FAIR, and FAIR is the most commonly referenced policy proposal in the piece, so I think that Jeremy’s reaction in Dislightenment is best understood as, primarily, a reaction to FAIR. (More generally, I don’t know what else he would have been reacting to, because in my mind FAIR was fairly catalytic in this whole debate, though it’s possible I’m overestimating its importance. And in any case I wasn’t on Twitter at the time so may lack important context that he’s importing into the conversation.) In which case, in order to support your general claim about unnecessary polarization, we would need to ask whether FAIR did unnecessary things polarize him.
Which brings us to the question of what exactly unnecessary polarization means. My sense is that avoiding unnecessary polarization would, in practice, mean that policy researchers write and speak extremely defensively to avoid making any unnecessary enemies. This would entail falsifying not just their own personal beliefs about optimal policy, but also, crucially, falsifying their prediction about what optimal policy is from the set of preferences that the public already holds. It would lead to writing positive proposals shot through with diligent and pervasive reputation management, leading to a lot of unnecessary and confusing hedges and disjunctive asides. I think pieces like that can be good, but it would be very bad if every piece was like that.
Instead, I think it is reasonable and preferable for discourse to unfold like this: Policy researchers write politely about the things that they think are true, explain their reasoning, acknowledge limitations and uncertainties, and invite further discussion. People like Jeremy then enter the conversation, bringing a useful different perspective, which is exactly what happened here. And then we can update policy proposals over time, to give more or less weight to different considerations in light of new arguments, political evidence (what do people think is riskier: too much centralization or too much decentralization?) and technical evidence. And then maybe eventually there is enough consensus to overcome the vetocratic inertia of our political system and make new policy. Or maybe a consensus is reached that this is not necessary. Or maybe no consensus is ever reached, in which case the default is nothing happens.
Contrast this with what I think the “reduce unnecessary polarization” approach would tend to recommend, which is something closer to starting the conversation with an attempt at a compromise position. It is sometimes useful to do this. But I think that, in terms of actual truth discovery, laying out the full case for one’s own perspective is productive and necessary. Without full-throated policy proposals, policy will tend too much either towards an unprincipled centrism (wherein all perspectives are seen as equally valid and therefore worthy of compromise) or towards the perspectives of those who defect from the “start at compromise” policy. When the stakes are really high, this seems bad.
To be clear, I don’t think you’re advocating for this “compromise-only” position. But in the case of Jeremy and Dislightenment specifically, I think this is what it would have taken to avoid polarization (and I doubt even that would have worked): writing FAIR with a much mushier, “who’s to say?” perspective.
In retrospect, I think it’s perfectly reasonable to think that we should have talked about centralization concerns more in FAIR. In fact, I endorse that proposition. And of course it was in some sense unnecessary to write it with the exact discussion of centralization that we did. But I nevertheless do not think that we can be said to have caused Jeremy to unnecessarily polarize against us, because I think him polarizing against us on the basis of FAIR is in fact not reasonable.
On ‘elite panic’ and ‘counter-enlightenment’, he’s not directly comparing FAIR to it I think. He’s saying that previous attempts to avoid democratisation of power in the Enlightenment tradition have had these flaws.
I disagree with this as a textual matter. Here are some excerpts from Dislightenment (emphases added):
Proposals for stringent AI model licensing and surveillance will . . . potentially roll[] back the societal gains of the Enlightenment.
bombing data centers and global surveillance of all computers is the only way[!!!] to ensure the kind of safety compliance that FAR proposes.
FAR briefly considers this idea, saying ‘for frontier AI development, sector-specific regulations can be valuable, but will likely leave a subset of the high severity and scale risks unaddressed’ But it . . . promote[s] an approach which, as we’ve seen, could undo centuries of cultural, societal, and political development.
He fairly consistently paints FAIR (or licensing more generally, which is a core part of FAIR) as the main policy he is responding to.
I think, from Jeremy’s PoV, that centralization of power is the actual ballgame and what Frontier AI Regulation should be about. So one mention on page 31 probably isn’t good enough for him.
It is definitely fair for him to think that we should have talked about decentralization more! But I don’t think it’s reasonable for him to polarize against us on that basis. That seems like the crux of the issue.
Jeremy’s reaction is most sympathetic if you model the FAIR authors specifically or the TAI governance community more broadly as a group of people totally unsympathetic to distribution of power concerns. The problem is that that is not true. My first main publication in this space was on the risk of excessively centralized power from AGI; another lead FAIR coauthor was on that paper too. Other coauthors have also written about this issue: e.g., 1; 2; 3 at 46–48; 4; 5; 6. It’s a very central worry in the field, dating back to the first research agenda. So I really don’t think polarization against us on the grounds that we have failed to give centralization concerns a fair shake is reasonable.
I think the actual explanation is that Jeremy and the group of which he is representative have a very strong prior in favor of open-sourcing things, and find it morally outrageous to propose restrictions thereon. While I think a prior in favor of OS is reasonable (and indeed correct), I do not think it reasonable for them to polarize against people who think there should be exceptions to the right to OS things. I think that it generally stems from an improper attachment to a specific method of distributing power without really thinking through the limits of that justification, or acknowledging that there even could be such limits.
You can see this dynamic at work very explicitly with Jeremy. In the seminar you mention, we tried to push Jeremy on whether, if a certain AI system turns out to be more like an atom bomb and less like voting, he would still think it’s good to open-source it. His response was that AI is not like an atomic bomb.
Again, a perfectly fine proposition to hold on its own. But it completely fails to either: (a) consider what the right policy would be if he is wrong, (b) acknowledge that there is substantial uncertainty or disagreement about whether any given AI system will be more bomb-like or voting-like.
That’s a fine reaction to me, just as it’s fine for you and Marcus to disagree on the relative costs/benefits and write the FAIR paper the way you did.
I agree! But I guess I’m not sure where the room for Jeremy’s unnecessary polarization comes in here. Do reasonable people get polarized against reasonable takes? No.
I know you’re not necessarily saying that FAIR was an example of unnecessary polarizing discourse. But my claim is either (a) FAIR was in fact unnecessarily polarizing, or (b) Jeremy’s reaction is not good evidence of unnecessary polarization, because it was a reaction to FAIR.
There’s probably a difference between ~July23-Jeremy and ~Nov23Jeremy
I think all of the opinions of his we’re discussing are from July 23? Am I missing something?
On the actual points though, I actually went back and skim-listened to the the webinar on the paper in July 2023, which Jeremy (and you!) participated in, and man I am so much more receptive and sympathetic to his position now than I was back then, and I don’t really find Marcus and you to be that convincing in rebuttal,
A perfectly reasonable opinion! But one thing that is not evident from the recording is that Jeremy showed up something like 10-20 minutes into the webinar, and so in fact missed a large portion of our presentation. Again, I think this is more consistent with some story other than unnecessary polarization. I don’t think any reasonable panelist would think it appropriate to participate in a panel where they missed the presentation of the other panelists, though maybe he had some good excuse.
Ah, interesting, not exactly the case that I thought you were making.
I more or less agree with the claim that “Elon changing the twitter censorship policies was a big driver of a chunk of Silicon Valley getting behind Trump,” but probably assign it lower explanatory power than you do (especially compared to nearby explanatory factors like, Elon crushing internal resistance and employee power at Twitter). But I disagree with the claim that anyone who bought Twitter could have done that, because I think that Elon’s preexisting sources of power and influence significantly improved his ability to drive and shape the emergence of the Tech Right.
I also don’t think that the Tech Right would have as much power in the Trump admin if not for Elon promoting Trump and joining the administration. So a different Twitter CEO who also created the Tech Right would have created a much less powerful force.
I will say that not appreciating arguments from open-source advocates, who are very concerned about the concentration of power from powerful AI, has lead to a completely unnecessary polarisation against the AI Safety community from it.
I think if you read the FAIR paper to which Jeremy is responding (of which I am a lead author), it’s very hard to defend the proposition that we did not acknowledge and appreciate his arguments. There is an acknowledgment of each of the major points he raises on page 31 of FAIR. If you then compare the tone of the FAIR paper to his tone in that article, I think he was also significantly escalatory, comparing us to an “elite panic” and “counter-enlightenment” forces.
To be clear, notwithstanding these criticisms, I think both Jeremy’s article and the line of open-source discourse descending from it have been overall good in getting people to think about tradeoffs here more clearly. I frequently cite to it for that reason. But I think that a failure to appreciate these arguments is not the cause of the animosity in at least his individual case: I think his moral outrage at licensing proposals for AI development is. And that’s perfectly fine as far as I’m concerned. People being mad at you is the price of trying to influence policy.
I think a large number of commentators in this space seem to jump from “some person is mad at us” to “we have done something wrong” far too easily. It is of course very useful to notice when people are mad at you and query whether you should have done anything differently, and there are cases where this has been true. But in this case, if you believe, as I did and still do, that there is a good case for some forms of AI licensing notwithstanding concerns about centralization of power, then you will just in fact have pro-OS people mad at you, no matter how nicely your white papers are written.
(Elon’s takeover of twitter was probably the second—it’s crazy that you can get that much power for $44 billion.)
I think this is pretty significantly understating the true cost. Or put differently, I don’t think it’s good to model this as an easily replicable type of transaction.
I don’t think that if, say, some more boring multibillionaire did the same thing, they could achieve anywhere close to the same effect. It seems like the Twitter deal mainly worked for him, as a political figure, because it leveraged existing idiosyncratic strengths that he had, like his existing reputation and social media following. But to get to the point where he had those traits, he needed to be crazy successful in other ways. So the true cost is not $44 billion, but more like: be the world’s richest person, who is also charismatic in a bunch of different ways, have an extremely dedicated online base of support from consumers and investors, have a reputation for being a great tech visionary, and then spend $44B.
A warm welcome to the forum!
I don’t claim to speak authoritatively, or to answer all of your questions, but perhaps this will help continue your exploration.
There’s an “old” (by EA standards) saying in EA, that EA is a Question, Not an Ideology. Most of what connects the people on this forum is not necessarily that they all work in the same cause area, or share the same underlying philosophy, or have the same priorities. Rather, what connects us is rigorous inquiry into the question of how we can do the most good for others with our spare resources. Because many of these questions are philosophical, people who start from that same question can and do disagree.
Accordingly, people in EA fall on both sides of many of the questions you ask. There are definitely people in EA that don’t think that we should prioritize future lives over present lives. There are definitely people who are skeptical about AI safety. There are definitely people who are concerned about the “moral licensing” effects of earning-to-give.
So I guess my general answer to your closing question is: you are not missing anything; on the contrary, you have identified a number of questions that people in EA have been debating for the past ~20 years and will likely continue doing so. If you share the general goal of effectively doing good for the world (as, from your bio, it looks like you do), I hope you will continue to think about these questions in an open-minded and curious way. Hopefully discussions and interactions with the EA community will provide you some value as you do so. But ultimately, what is more important than your agreement or disagreement with the EA community about any particular issue is your own commitment to thinking carefully about how you can do good.
I upvoted and didn’t disagree-vote, because I generally agree that using AI to nudge online discourse in more productive directions seems good. But if I had to guess where disagree votes come from, it might be a combination of:
It seems like we probably want politeness-satisficing rather than politeness-maximizing. (This could be consistent with some versions of the mechanism you describe, or a very slightly tweaked version).
There’s a fine line between politiness-moderating and moderating the substance of ideas that make people uncomfortable. Historically, it has been hard to police this line, and given the empirically observable political preferences of LLMs, it’s reasonable for people who don’t share those preferences to worry that this will disadvantage them (though I expect this bias issue to get better over time, possibly very soon)
There is a time and place for spirited moral discourse that is not “polite,” because the targets of the discourse are engaging in highly morally objectionable action, and it would be bad to always discourage people from engaging in such discourse.*
*This is a complicated topic that I don’t claim to have either (a) fully coherent views on, or (b) have always lived up to the views I do endorse.
Both Sam and Dario saying that they now believe they know how to build AGI seems like an underrated development to me. To my knowledge, they only started saying this recently. I suspect they are overconfident, but still seems like a more significant indicator than many people seem to be tracking.
I also have very wide error bars on my $1B estimate; I have no idea how much equity early employees would normally retain in a startup like Anthropic. That number is also probably dominated by the particular compensation arrangements and donation plans of ~5–10 key people and so very sensitive to assumptions about them individually.
Indeed, though EAs were less well-represented at the senior managerial and executive levels of OpenAI, especially after the Anthropic departures.
One factor this post might be failing to account for: the wealth of Anthropic founders and early-stage employees, many of whom are EAs, EA-adjacent, or at minimum very interested in supporting existential AI safety. I don’t know how much equity they have collectively, how liquid it is, how much they plan to donate, etc. But if I had to guess, there’s probably at least $1B earmarked for EA projects there, at least in NPV terms?
(In general, this topic seems under-discussed.)
I’m obviously not a bankruptcy lawyer, but I was surprised to read this because I assumed that the statute of limitations depended on the jurisdiction of the bankruptcy estate, not the clawback claimee. Am I wrong? Or are there parallel proceedings in other jurisdictions?
Related recent discussion here: https://forum.effectivealtruism.org/posts/eMWsKbLFMy7ABdCLw/alfredo-parra-s-quick-takes?commentId=vo3jDMAhFFd2XQgYa