GCR capacity-building grantmaking and projects at Open Phil.
Eli Roseđ¸
Karma: 2,549
Unsurprising that Buggs approve of this topic.
I like the core point and think itâs very important â though I donât really vibe with statements about calibration being actively dangerous.
I think EA culture can make it seem like being calibrated is the most important thing ever. But I think on the topic of âwill my ambitious projects succeed?â it seems very difficult to be calibrated and fairly cursed overall, and it may overall be unhelpful to try super hard at this vs. just executing.
For example, Iâm guessing that Norman Borlaug didnât feed a billion people primarily by being extremely well-calibrated. I think he did it via being a good scientist, dedicating himself fully to something impactful-in-principle even when the way forward was unclear, and being willing to do things outside his normal wheelhouse â like bureaucracy or engaging with Indian government officials. Iâd guess he was well-calibrated about micro-aspects of his wheat germination work, such as which experiments were likely to work out, or perhaps which politicians would listen to him (but on the other hand, he could simply have been uncalibrated and very persistent). I wouldnât expect heâd be well-calibrated about the overall shape of his career early on, and it doesnât seem very important for him to have been calibrated about that.
One often hears about successful political candidates that they always had unwarranted-seeming confidence in themselves and always thought theyâd win office. Iâve noticed that the most successful researchers tend to seem a bit âcrazyâ and have unwarranted confidence in their own work. Successful startup founders too are not exactly known for realistic ex-ante estimates of their own success. (Of course this all applies to many unsuccessful political candidates, researchers and founders as well.)
I think something psychologically important is going on here; my guess is that âpart of youâ really needs to believe in outsized success in order to have a chance of achieving it. This old Nate Soares post is relevant.
Yeah, totally a contextual call about how to make this point in any given conversation, it can be easy to get bogged down with irrelevant context.
I do think itâs true that utilitarian thought tends to push one towards centralization and central planning, despite the bad track record here. Itâs worth engaging with thoughtful critiques of EA vibes on this front.
Salaries are the most basic way our economy does allocation, and one possible âEA government utopiaâ scenario is one where the government corrects market inefficiencies such that salaries perfectly track âvalue added to the world.â This is deeply sci-fi of course, but hey why not dream. In such a utopia world, if we really did reach the point where marginal safety researchers are not adding more value than marginal post office workers, salaries would.
I like the main point youâre making.
However, I think âthe governmentâs version of 80,000 Hoursâ is a very command-economy vision. Command economies have a terrible track record, and if there were such a thing as an âEA world governmentâ (which I would have many questions about regardless) I would strongly think it shouldnât try to plan and direct everyoneâs individual careers, and should instead leverage market forces like ~all successful large economies.
+1 on wanting a more model-based version of this.
And +1 to you vibe coding it!
Upon seeing this, I had the same thought about vibe coding a more model-based version ⌠so, race you to whoever gets around to it?
Very cool!
I mostly donated to democracy preservation work and did some political giving. And a little to the shrimp.
Wow awesome thanks for letting me know!
Thanks for writing this!!
This risk seems equal or greater to me than AI takeover risk. Historically the EA & AIS communities focused more on misalignment, but Iâm not sure if that choice has held up.
Come 2027, Iâd love for it to be the case that an order of magnitude more people are usefully working on this risk. I think it will be rough going for the first 50 people in this area; I expect thereâs a bunch more clarificatory and scoping work to do; this is virgin territory. We need some pioneers.
People with plans in this area should feel free to apply for career transition funding from my team at Coefficient (fka Open Phil) if they think that would be helpful to them.
Iâm quite excited about EAs making videos about EA principles and their applications, and I think this is an impactful thing for people to explore. It seems quite possible to do in a way that doesnât compromise on idea fidelity; I think sincerity counts for quite a lot. In many cases I think videos and other content can be lighthearted /â fun /â unserious and still transmit the ideas well.
thank machine doggo
Oh woops didnât look at parent comment, haah
I think the vast majority of people making decisions about public policy or who to vote for either arenât ethically impartial, or theyâre âspotlightingâ, as you put it. I expect the kind of bracketing Iâd endorse upon reflection to look pretty different from such decision-making.
But suppose I want to know who of two candidates to vote for, and Iâd like to incorporate impartial ethics into that decision. What do I do then?
That said, maybe youâre thinking of this point I mentioned to you on a call
Hmm, I donât recall this; another Eli perhaps? : )
(vibesy post)
People often want to be part of something bigger than themselves. At least for a lot of people this is pre-theoretic. Personally, Iâve felt this since I was little: to spend my whole life satisfying the particular desires of the particular person I happened to be born into the body of, seemed pointless and uninteresting.
I knew I wanted âsomething biggerâ even when I was young (e.g. 13 years old). Around this age my dream was to be a novelist. This isnât a kind of desire people would generally call âaltruistic,â nor would my younger self have called it âaltruistic.â But it was certainly grounded in a desire for my life to mean something to other people. Stuff like the Discworld series and Watchmen really meant something to me, and I wanted to write stuff that meant something to others in the same way.
My current dreams and worldview, after ~10 years of escalating involvement with EA, seem to me to spring from the same seed. I feel continuous with my much younger self. I want my life to mean something to others: that is the obvious yardstick. I want to be doing the most I can on that front.
The empirics were the surprising part. It turns out that the âbasic shape of the worldâ is much more mutable than my younger self thought, and in light of this my earlier dreams seem extremely unambitious. Astonishingly, I can probably:
save many lives over my career at minimum, by donating to GiveWell, and likely more by doing more off the beaten path things
save <large number> of e.g. chickens from lives full of torture
be part of a pretty small set of people seriously trying to do something about truly wild risks from new AI and bioengineering technologies
It probably matters more to others that they are not tortured, or dying of malaria, or suffering some kind of AI catastrophe, than that there is another good book for them to read, especially given there are already a lot of good novelists. The seed of the impulse is the same â wanting to be part of something bigger, wanting to live for my effect on others and not just myself. My sense of what is truly out there in the world and of what I can do about it are whatâs changed.
Like if youâre contemplating running a fellowship program for AI interested people, and you have animals in your moral circle, youâre going to have to build this botec that includes the probability an X% of the people you bring into the fellowship are not going to care about animals and likely, if they get a policy role, to pass policies that are really bad for them...
...I sort of suspect that only a handful of people are trying to do this, and I get why! I made a reasonably straightforward botec for calculating the benefits to birds of bird-safe glass, that accounted for backfire to birds, and it took a lot of research effort. If you asked me how bird-safe glass policy is going to affect AI risk after all that, I might throw my computer at you. But I think the precise probabilities approach would imply that I should.
Just purely on the descriptive level and not the normative one â
I agree but even more strongly: in AI safety Iâve basically never seen a BOTEC this detailed. I think Eric Neymanâs BOTEC of the cost-effectiveness of donating to congressional candidate Alex Bores is a good public example of the type of analysis common in EA-driven AI safety work: it bottoms out in pretty general goods like âgovernment action on AI safetyâ and does not try to model second-order effects to the degree described here. It doesnât model even considerations like âwhat if AI safety legislation is passed, but that legislation backfires by increasing polarization on the issue?â let alone anything about animals.
Instead, this kind of strategic discussion tends to be qualitative, and is hashed out in huge blocks of prose and comment threads e.g. on LessWrong, or verbally.
I sort of wonder if some people in the AI communityâany maybe you, from what youâve said here? -- are using precise probabilities to get to the conclusion that you want to work primarily on AI stuff, and then spotlighting to that cause area when youâre analyzing at the level of interventions.
I see why you describe it this way, and this directionally this seems right. But, what we do doesnât really sound like âspotlightingâ as you describe it in the post: focusing on specific moral patient groups and explicitly setting aside others.
Essentially I think the epistemic framework we use is just more anarchic and freeform than that! In AIS discourse, it feels like âbut this intervention could slow down the US relative to Chinaâ or âbut this intervention could backfire by increasing polarizationâ or âbut this intervention could be bad for animalsâ exist at the same epistemic level, and all are considered valid points to raise.
(I do think that there is a significant body of orthodox AI safety thought which takes particular stances on each of these issues and other issues, which in a lot of contexts likely makes various points feel like not âvalidâ to raise. I think this is unfortunate.)
Maybe itâs similar to the difference between philosophy and experimental science, where in philosophy a lot of discourse is fundamentally unstructured and qualitative, and in the experimental sciences there is much more structure because any contribution needs to be an empirical experiment, and there are specific norms and formats for those, which have certain implications for how second-order effects are or arenât considered. AI safety discourse also feels similar at times to wonk-ish policy discourse.
(Within certain well-scoped sub-areas of AI safety things are less epistemically anarchic; e.g. research into AI interpretability usually needs empirical results if itâs to be taken seriously.)
I think someone using precise probabilities all the way down is building a lot more explicit models every time they consider a specific intervention. Like if youâre contemplating running a fellowship program for AI interested people, and you have animals in your moral circle, youâre going to have to build this botec that includes the probability an X% of the people you bring into the fellowship are not going to care about animals and likely, if they get a policy role, to pass policies that are really bad for them. And all sorts of things like that. So your output would be a bunch of hypotheses about exactly how these fellows are going to benefit AI policy, and some precise probabilities about how those policy benefits are going to help people, and possibly animals to what degree, etc.
Hmm, I wouldnât agree that someone using precise probabilities âall the way downâ is necessarily building these kind of explicit models. I wonder if the term âprecise probabilitiesâ is being understood differently in our two areas.
In the Bayesian epistemic style that EA x AI safety has, itâs felt that anyone can attach precise probabilities to their beliefs with ~no additional thought, and that these probabilities are subjective things which may not be backed by any kind of explicit or even externally legible model. Thereâs a huge focus on probabilities as betting odds, and betting odds donât require such things (diverging notably from how probabilities are used in science).
I mean, I think typically people have something to say to justify their beliefs, but this can be & often is something as high-level as âit seems good if AGI companies are required to be more transparent about their safety practices,â with little in the way of explicit models about downstream effects thereof.[1]
Apologies for not responding to some of the other threads in your post, ran out of time; looking forward to discussing in person sometime.
- ^
While itâs common for AI safety people to agree with my statement about transparency here, some may flatly disagree (i.e. disagree about sign), and others (more commonly) may disagree massively about the magnitude of the effect. There are many verbal arguments but relatively few explicit models to adjudicate these disputes.
- ^
I just remembered Matthew Barnettâs 2022 post My Current Thoughts on the risks from SETI which is a serious investigation into how to mitigate this exact scenario.
That does seem right, thanks. I intended to include dictator-ish human takeover there (which seems to me to be at least as likely as misaligned AI takeover) as well, but didnât say that clearly.
Edited to ârelatively amoral forcesâ which still isnât great but maybe a little clearer.
Enjoyed this post.
Maybe Iâll speak from an AI safety perspective. The usual argument among EAs working on AI safety is:
the future is large and plausibly contains much goodness
today, we can plausibly do things to steer (in expectation) towards achieving this goodness and away from catastrophically losing it
the invention of powerful AI is a super important leverage point for such steering
This is also the main argument motivating me â though I retain meaningful meta-uncertainty and am also interested in more commonsense motivations for AI safety work.
A lot of the potential goodness in 1. seems to come from digital minds that humans create, since it seems that at some point these will be much quicker to replicate than humans or animals. But lots of the interventions in 2. seem to also be helpful for getting things to go better for current farmed and wild animals, e.g. because they are aimed avoiding a takeover of society by forces which donât care at all about morals. Personally I hope we use technology to lift wild animals out of their current predicament, although I have little idea what it would look like with any concreteness.
This relies on what you call the âassigning precise probabilitiesâ approach, and indeed I rarely encounter AI safety x EA people who arenât happy assigning precise probabilities, even in the face of deep uncertainty. I really like how your post points out that this is a difference from the discourse around wild animal welfare but that itâs not clear what the high-level reason for this is. I donât see a clear high-level reason either from my vantage point. Some thoughts:
It might be interesting to move out of high-level reason zone entirely and just look at the interventions, e.g. directly compare the robustness of installing bird-safe glass in a building vs. something like developing new technical techniques to help us avoid losing control of AIs.
What would the justification standards in wild animal welfare say about uncertainty-laden decisions that involve neither AI nor animals: e.g. as a government, deciding which policies to enact, or as a US citizen, deciding who to vote for President?
Coda: to your âwhy should justification standards be the sameâ question, Iâd just want to say Iâm very interested in maintaining the ideal that EAs compare and debate these things; thanks for writing this!
No one is dying of not reading Proust, but many people are leading hollower and shallower lives because the arts are so inaccessible.
Tangential to your main point, and preaching to the choir, but⌠why are âthe artsâ âinaccessible?â The Internet is a huge revolution in the democratization of art relative to most of human history, TV dramas are now much more complex and interesting than they have been in the past, A24 is pumping out tons of weird/âinteresting movies, way more people are making interesting music and distributing it than before.
I think (and this is a drive-by comment, I havenât read the article), the author is conflating âserious literatureâ â often an acquired taste that people need to get from a class or similar â with all of âthe arts.â I studied literature in college, read poetry and e.g. Tolstoy in my free time now, yada yada â and I think this is extremely paternalistic.
I think thereâs value in someone teaching you to enjoy Proust, and indeed wish more people had access to that sort of thing. But I donât think it comes anywhere close to deserving the kind of uniquely elevated position over other forms of artistic production that literature professors etc sometimes (not always) want to give it, and which I feel is on display in this quote.
In any case, the obvious thing to do is ask whether the beneficiaries would prefer more soup or more Proust.
Wow, that boggles my mind, especially as someone who attended a similar school for undergrad. Anywhere we can read about this? Presumably this happened after the initial post which reported 80 pledges.
(Middlebury has about 2800 undergrads so this would be 280 students taking the trial or full GWWC pledges.)