GCR capacity-building grantmaking and projects at Open Phil.
Eli Roseđ¸
Karma: 2,587
Cool! Re how to build it, Iâd just talk to CEA here or maybe EAG goers, donât think I have any insight to add.
It is popular to hate on Swapcard, and yet Swapcard seems like the best available solution despite its flaws. Claude Code or other AI coding assistants are very good nowadays, and conceivably, someone could just Claude Code a better Swapcard that maintained feature parity while not having flaws.
Overall Iâm guessing this would be too hard right now, but we do live in an age of mysteries and wonders. It gets easier every month. One reason for optimism is it seems like the Swapcard team is probably not focused on the somewhat odd use case of EAGs in general (e.g. from what I understand, most conferences in the world have much less emphasis on 1-1 meetings).
And if you made a Swapcard replacement good enough to replace Swapcard for CEA event purposes, just think of the glory. (And the impact.)
Anecdotally, the authors of this post have now persuaded nearly 10% of the students at their university to take a GWWC pledge (trial or full), while the pledge rate at most universities is well under 1%. After getting to know these authors, I believe this incredible success is due to their kindness, charisma, passion and integrityânot because the audience at their university is fundamentally different than at other universities.
Wow, that boggles my mind, especially as someone who attended a similar school for undergrad. Anywhere we can read about this? Presumably this happened after the initial post which reported 80 pledges.
(Middlebury has about 2800 undergrads so this would be 280 students taking the trial or full GWWC pledges.)
Unsurprising that Buggs approve of this topic.
I like the core point and think itâs very important â though I donât really vibe with statements about calibration being actively dangerous.
I think EA culture can make it seem like being calibrated is the most important thing ever. But I think on the topic of âwill my ambitious projects succeed?â it seems very difficult to be calibrated and fairly cursed overall, and it may overall be unhelpful to try super hard at this vs. just executing.
For example, Iâm guessing that Norman Borlaug didnât feed a billion people primarily by being extremely well-calibrated. I think he did it via being a good scientist, dedicating himself fully to something impactful-in-principle even when the way forward was unclear, and being willing to do things outside his normal wheelhouse â like bureaucracy or engaging with Indian government officials. Iâd guess he was well-calibrated about micro-aspects of his wheat germination work, such as which experiments were likely to work out, or perhaps which politicians would listen to him (but on the other hand, he could simply have been uncalibrated and very persistent). I wouldnât expect heâd be well-calibrated about the overall shape of his career early on, and it doesnât seem very important for him to have been calibrated about that.
One often hears about successful political candidates that they always had unwarranted-seeming confidence in themselves and always thought theyâd win office. Iâve noticed that the most successful researchers tend to seem a bit âcrazyâ and have unwarranted confidence in their own work. Successful startup founders too are not exactly known for realistic ex-ante estimates of their own success. (Of course this all applies to many unsuccessful political candidates, researchers and founders as well.)
I think something psychologically important is going on here; my guess is that âpart of youâ really needs to believe in outsized success in order to have a chance of achieving it. This old Nate Soares post is relevant.
Yeah, totally a contextual call about how to make this point in any given conversation, it can be easy to get bogged down with irrelevant context.
I do think itâs true that utilitarian thought tends to push one towards centralization and central planning, despite the bad track record here. Itâs worth engaging with thoughtful critiques of EA vibes on this front.
Salaries are the most basic way our economy does allocation, and one possible âEA government utopiaâ scenario is one where the government corrects market inefficiencies such that salaries perfectly track âvalue added to the world.â This is deeply sci-fi of course, but hey why not dream. In such a utopia world, if we really did reach the point where marginal safety researchers are not adding more value than marginal post office workers, salaries would presumably change as well.
I like the main point youâre making.
However, I think âthe governmentâs version of 80,000 Hoursâ is a very command-economy vision. Command economies have a terrible track record, and if there were such a thing as an âEA world governmentâ (which I would have many questions about regardless) I would strongly think it shouldnât try to plan and direct everyoneâs individual careers, and should instead leverage market forces like ~all successful large economies.
+1 on wanting a more model-based version of this.
And +1 to you vibe coding it!
Upon seeing this, I had the same thought about vibe coding a more model-based version ⌠so, race you to whoever gets around to it?
Very cool!
I mostly donated to democracy preservation work and did some political giving. And a little to the shrimp.
Wow awesome thanks for letting me know!
Thanks for writing this!!
This risk seems equal or greater to me than AI takeover risk. Historically the EA & AIS communities focused more on misalignment, but Iâm not sure if that choice has held up.
Come 2027, Iâd love for it to be the case that an order of magnitude more people are usefully working on this risk. I think it will be rough going for the first 50 people in this area; I expect thereâs a bunch more clarificatory and scoping work to do; this is virgin territory. We need some pioneers.
People with plans in this area should feel free to apply for career transition funding from my team at Coefficient (fka Open Phil) if they think that would be helpful to them.
Iâm quite excited about EAs making videos about EA principles and their applications, and I think this is an impactful thing for people to explore. It seems quite possible to do in a way that doesnât compromise on idea fidelity; I think sincerity counts for quite a lot. In many cases I think videos and other content can be lighthearted /â fun /â unserious and still transmit the ideas well.
thank machine doggo
Oh woops didnât look at parent comment, haah
I think the vast majority of people making decisions about public policy or who to vote for either arenât ethically impartial, or theyâre âspotlightingâ, as you put it. I expect the kind of bracketing Iâd endorse upon reflection to look pretty different from such decision-making.
But suppose I want to know who of two candidates to vote for, and Iâd like to incorporate impartial ethics into that decision. What do I do then?
That said, maybe youâre thinking of this point I mentioned to you on a call
Hmm, I donât recall this; another Eli perhaps? : )
(vibesy post)
People often want to be part of something bigger than themselves. At least for a lot of people this is pre-theoretic. Personally, Iâve felt this since I was little: to spend my whole life satisfying the particular desires of the particular person I happened to be born into the body of, seemed pointless and uninteresting.
I knew I wanted âsomething biggerâ even when I was young (e.g. 13 years old). Around this age my dream was to be a novelist. This isnât a kind of desire people would generally call âaltruistic,â nor would my younger self have called it âaltruistic.â But it was certainly grounded in a desire for my life to mean something to other people. Stuff like the Discworld series and Watchmen really meant something to me, and I wanted to write stuff that meant something to others in the same way.
My current dreams and worldview, after ~10 years of escalating involvement with EA, seem to me to spring from the same seed. I feel continuous with my much younger self. I want my life to mean something to others: that is the obvious yardstick. I want to be doing the most I can on that front.
The empirics were the surprising part. It turns out that the âbasic shape of the worldâ is much more mutable than my younger self thought, and in light of this my earlier dreams seem extremely unambitious. Astonishingly, I can probably:
save many lives over my career at minimum, by donating to GiveWell, and likely more by doing more off the beaten path things
save <large number> of e.g. chickens from lives full of torture
be part of a pretty small set of people seriously trying to do something about truly wild risks from new AI and bioengineering technologies
It probably matters more to others that they are not tortured, or dying of malaria, or suffering some kind of AI catastrophe, than that there is another good book for them to read, especially given there are already a lot of good novelists. The seed of the impulse is the same â wanting to be part of something bigger, wanting to live for my effect on others and not just myself. My sense of what is truly out there in the world and of what I can do about it are whatâs changed.
Like if youâre contemplating running a fellowship program for AI interested people, and you have animals in your moral circle, youâre going to have to build this botec that includes the probability an X% of the people you bring into the fellowship are not going to care about animals and likely, if they get a policy role, to pass policies that are really bad for them...
...I sort of suspect that only a handful of people are trying to do this, and I get why! I made a reasonably straightforward botec for calculating the benefits to birds of bird-safe glass, that accounted for backfire to birds, and it took a lot of research effort. If you asked me how bird-safe glass policy is going to affect AI risk after all that, I might throw my computer at you. But I think the precise probabilities approach would imply that I should.
Just purely on the descriptive level and not the normative one â
I agree but even more strongly: in AI safety Iâve basically never seen a BOTEC this detailed. I think Eric Neymanâs BOTEC of the cost-effectiveness of donating to congressional candidate Alex Bores is a good public example of the type of analysis common in EA-driven AI safety work: it bottoms out in pretty general goods like âgovernment action on AI safetyâ and does not try to model second-order effects to the degree described here. It doesnât model even considerations like âwhat if AI safety legislation is passed, but that legislation backfires by increasing polarization on the issue?â let alone anything about animals.
Instead, this kind of strategic discussion tends to be qualitative, and is hashed out in huge blocks of prose and comment threads e.g. on LessWrong, or verbally.
I sort of wonder if some people in the AI communityâany maybe you, from what youâve said here? -- are using precise probabilities to get to the conclusion that you want to work primarily on AI stuff, and then spotlighting to that cause area when youâre analyzing at the level of interventions.
I see why you describe it this way, and this directionally this seems right. But, what we do doesnât really sound like âspotlightingâ as you describe it in the post: focusing on specific moral patient groups and explicitly setting aside others.
Essentially I think the epistemic framework we use is just more anarchic and freeform than that! In AIS discourse, it feels like âbut this intervention could slow down the US relative to Chinaâ or âbut this intervention could backfire by increasing polarizationâ or âbut this intervention could be bad for animalsâ exist at the same epistemic level, and all are considered valid points to raise.
(I do think that there is a significant body of orthodox AI safety thought which takes particular stances on each of these issues and other issues, which in a lot of contexts likely makes various points feel like not âvalidâ to raise. I think this is unfortunate.)
Maybe itâs similar to the difference between philosophy and experimental science, where in philosophy a lot of discourse is fundamentally unstructured and qualitative, and in the experimental sciences there is much more structure because any contribution needs to be an empirical experiment, and there are specific norms and formats for those, which have certain implications for how second-order effects are or arenât considered. AI safety discourse also feels similar at times to wonk-ish policy discourse.
(Within certain well-scoped sub-areas of AI safety things are less epistemically anarchic; e.g. research into AI interpretability usually needs empirical results if itâs to be taken seriously.)
I think someone using precise probabilities all the way down is building a lot more explicit models every time they consider a specific intervention. Like if youâre contemplating running a fellowship program for AI interested people, and you have animals in your moral circle, youâre going to have to build this botec that includes the probability an X% of the people you bring into the fellowship are not going to care about animals and likely, if they get a policy role, to pass policies that are really bad for them. And all sorts of things like that. So your output would be a bunch of hypotheses about exactly how these fellows are going to benefit AI policy, and some precise probabilities about how those policy benefits are going to help people, and possibly animals to what degree, etc.
Hmm, I wouldnât agree that someone using precise probabilities âall the way downâ is necessarily building these kind of explicit models. I wonder if the term âprecise probabilitiesâ is being understood differently in our two areas.
In the Bayesian epistemic style that EA x AI safety has, itâs felt that anyone can attach precise probabilities to their beliefs with ~no additional thought, and that these probabilities are subjective things which may not be backed by any kind of explicit or even externally legible model. Thereâs a huge focus on probabilities as betting odds, and betting odds donât require such things (diverging notably from how probabilities are used in science).
I mean, I think typically people have something to say to justify their beliefs, but this can be & often is something as high-level as âit seems good if AGI companies are required to be more transparent about their safety practices,â with little in the way of explicit models about downstream effects thereof.[1]
Apologies for not responding to some of the other threads in your post, ran out of time; looking forward to discussing in person sometime.
- ^
While itâs common for AI safety people to agree with my statement about transparency here, some may flatly disagree (i.e. disagree about sign), and others (more commonly) may disagree massively about the magnitude of the effect. There are many verbal arguments but relatively few explicit models to adjudicate these disputes.
- ^
I just remembered Matthew Barnettâs 2022 post My Current Thoughts on the risks from SETI which is a serious investigation into how to mitigate this exact scenario.
I think it sounds like an exciting idea. In my role funding EA CB work over the years Iâve seen a few of these clubs, so thereâs not literally nothing, but itâs true that itâs much less common than at universities, and Iâm not aware of EA groups at these specific high schools.
The answer to many questions of the form âwhy isnât there an EA group for XYZâ tends to be âno organizer /â no one else working to make it happenâ and Iâm guessing thatâs the main answer here too.