[Is longtermism bottlenecked by “great people”?]
Someone very influential in EA recently claimed in conversation with me that there are many tasks X such that (i) we currently don’t have anyone in the EA community who can do X, (ii) the bottleneck for this isn’t credentials or experience or knowledge but person-internal talent, and (iii) it would be very valuable (specifically from a longtermist point of view) if we could do X. And that therefore what we most need in EA are more “great people”.
I find this extremely dubious. (In fact, it seems so crazy to me that it seems more likely than not that I significantly misunderstood the person who I think made these claims.) The first claim is of course vacuously true if, for X, we choose some ~impossible task such as “experience a utility-monster amount of pleasure” or “come up with a blueprint for how to build safe AGI that is convincing to benign actors able to execute it”. But of course more great people don’t help with solving impossible tasks.
Given the size and talent distribution of the EA community my guess is that for most apparent X, the issue either is that (a) X is ~impossible, or (b) there are people in EA who could do X, but the relevant actors cannot identify them, or (c) acquiring the ability to do X is costly (e.g. perhaps you need time to acquire domain-specific expertise), even for maximally talented “great people”, and the relevant actors either are unable to help pay that cost (e.g. by training people themselves, or giving them the resources to allow them to get training elsewhere) or make a mistake by not doing so.
My best guess for the genesis of the “we need more great people” perspective: Suppose I talk a lot to people at an organization that thinks there’s a decent chance we’ll develop transformative AI soon but it will go badly, and that as a consequence tries to grow as fast as possible to pursue various ambitious activities which they think reduces that risk. If these activities are scalable projects with short feedback loops on some intermediate metrics (e.g. running some super-large-scale machine learning experiments), then I expect I would hear a lot of claims like “we really need someone who can do X”. I think it’s just a general property of a certain kind of fast-growing organization that’s doing practical things in the world that everything constantly seems like it’s on fire. But I would also expect that, if I poked a bit at these claims, it would usually turn out that X is something like “contribute to this software project at the pace and quality level of our best engineers, w/o requiring any management time” or “convince some investors to give us much more money, but w/o anyone spending any time transferring relevant knowledge”. If you see that things break because X isn’t done, even though something like X seems doable in principle (perhaps you see others do it), it’s tempting to think that what you need is more “great people” who can do X. After all, people generally are the sort of stuff that does things, and maybe you’ve actually seen some people do X. But it still doesn’t follow that in your situation “great people” are the bottleneck …
Curious if anyone has examples of tasks X for which the original claims seem in fact true. That’s probably the easiest way to convince me that I’m wrong.
I’m not quite sure how high your bar is for “experience”, but many of the tasks that I’m most enthusiastic about in EA are ones which could plausibly be done by someone in their early 20s who eg just graduated university. Various tasks of this type:
Work at MIRI on various programming tasks which require being really smart and good at math and programming and able to work with type theory and Haskell. Eg we recently hired Seraphina Nix to do this right out of college. There are other people who are recent college graduates who we offered this job to who didn’t accept. These people are unusually good programmers for their age, but they’re not unique. I’m more enthusiastic about hiring older and more experienced people, but that’s not a hard requirement. We could probably hire several more of these people before we became bottlenecked on management capacity.
Generalist AI safety research that Evan Hubinger does—he led the writing of “Risks from Learned Optimization” during a summer internship at MIRI; before that internship he hadn’t had much contact with the AI safety community in person (though he’d read stuff online).
Richard Ngo is another young AI safety researcher doing lots of great self-directed stuff; I don’t think he consumed an enormous amount of outside resources while becoming good at thinking about this stuff.
I think that there are inexperienced people who could do really helpful work with me on EA movement building; to be good at this you need to have read a lot about EA and be friendly and know how to talk to lots of people.
My guess is that EA does not have a lot of unidentified people who are as good at these things as the people I’ve identified.
I think that the “EA doesn’t have enough great people” problem feels more important to me than the “EA has trouble using the people we have” problem.
Thanks, very interesting!
I agree the examples you gave could be done by a recent graduate. (Though my guess is the community building stuff would benefit from some kinds of additional experience that has trained relevant project management and people skills.)
I suspect our impressions differ in two ways:
1. My guess is I consider the activities you mentioned less valuable than you do. Probably the difference is largest for programming at MIRI and smallest for Hubinger-style AI safety research. (This would probably be a bigger discussion.)
2. Independent of this, my guess would be that EA does have a decent number of unidentified people who would be about as good as people you’ve identified. E.g., I can think of ~5 people off the top of my head of whom I think they might be great at one of the things you listed, and if I had your view on their value I’d probably think they should stop doing what they’re doing now and switch to trying one of these things. And I suspect if I thought hard about it, I could come up with 5-10 more people—and then there is the large number of people neither of us has any information about.
Two other thoughts I had in response:
It might be quite relevant if “great people” refers only to talent or also to beliefs and values/preferences. E.g. my guess is that there are several people who could be great at functional programming who either don’t want to work for MIRI, or don’t believe that this would be valuable. (This includes e.g. myself.) If to count as “great person” you need to have the right beliefs and preferences, I think your claim that “EA needs more great people” becomes stronger. But I think the practical implications would differ from the “greatness is only about talent” version, which is the one I had in mind in the OP.
One way to make the question more precise: At the margin, is it more valuable (a) to try to add high-potential people to the pool of EAs or (b) change the environment (e.g. coordination, incentives, …) to increase the expected value of activities by people in the current pool. With this operationalization, I might actually agree that the highest-value activities of type (a) are better than the ones of type (b), at least if the goal is finding programmers for MIRI and maybe for community building. (I’d still think that this would be because, while there are sufficiently talented people in EA, they don’t want to do this, and it’s hard to change beliefs/preferences and easier to get new smart people excited about EA. - Not because the community literally doesn’t have anyone with a sufficient level of innate talent. Of course, this probably wasn’t the claim the person I originally talked to was making.)
My guess is I consider the activities you mentioned less valuable than you do. Probably the difference is largest for programming at MIRI and smallest for Hubinger-style AI safety research. (This would probably be a bigger discussion.)
I don’t think that peculiarities of what kinds of EA work we’re most enthusiastic about lead to much of the disagreement. When I imagine myself taking on various different people’s views about what work would be most helpful, most of the time I end up thinking that valuable contributions could be made to that work by sufficiently talented undergrads.
Independent of this, my guess would be that EA does have a decent number of unidentified people who would be about as good as people you’ve identified. E.g., I can think of ~5 people off the top of my head of whom I think they might be great at one of the things you listed, and if I had your view on their value I’d probably think they should stop doing what they’re doing now and switch to trying one of these things. And I suspect if I thought hard about it, I could come up with 5-10 more people—and then there is the large number of people neither of us has any information about.
I am pretty skeptical of this. Eg I suspect that people like Evan (sorry Evan if you’re reading this for using you as a running example) are extremely unlikely to remain unidentified, because one of the things that they do is think about things in their own time and put the results online. Could you name a profile of such a person, and which of the types of work I named you think they’d maybe be as good at as the people I named?
It might be quite relevant if “great people” refers only to talent or also to beliefs and values/preferences
I am not intending to include beliefs and preferences in my definition of “great person”, except for preferences/beliefs like being not very altruistic, which I do count.
E.g. my guess is that there are several people who could be great at functional programming who either don’t want to work for MIRI, or don’t believe that this would be valuable. (This includes e.g. myself.)
I think my definition of great might be a higher bar than yours, based on the proportion of people who I think meet it? (To be clear I have no idea how good you’d be at programming for MIRI because I barely know you, and so I’m just talking about priors rather than specific guesses about you.)
For what it’s worth, I think that you’re not credulous enough of the possibility that the person you talked to actually disagreed with you—I think you might doing that thing whose name I forget where you steelman someone into saying the thing you think instead of the thing they think.
I agree we have important disagreements other than what kinds of EA work we’re most enthusiastic about. While not of major relevance for the original issue, I’d still note that I’m surprised by what you say about various other people’s view on EA, and I suspect it might not be true for me: while I agree there are some highly-valuable tasks that could be done by recent undergrads, I’d guess that if I made a list of the most valuable possible contributions then a majority of the entries would require someone to have a lot of AI-weighted generic influence/power (e.g. the kind of influence over AI a senior government member responsible for tech policy has, or a senior manager in a lab that could plausibly develop AGI), and that because of the way relevant existing institutions are structured this would usually require a significant amount of seniority. (It’s possible for some smart undergrads to embark on a path culminating in such a position, but my guess this is not the kind of thing you had in mind.)
I am pretty skeptical of this. Eg I suspect that people like Evan (sorry Evan if you’re reading this for using you as a running example) are extremely unlikely to remain unidentified, because one of the things that they do is think about things in their own time and put the results online. [...]
I don’t think these two claims are plausibly consistent, at least if “people like Evan” is also meant to exclude beliefs and preferences: For instance, if someone with Evan-level abilities doesn’t believe that thinking in their own time and putting results online is a worthwhile thing to do, then the identification mechanism you appeal to will fail. More broadly, someone’s actions will generally depend on all kinds of beliefs and preferences (e.g. on what they are able to do, on what people around them expect, on other incentives, …) that are much more dependent on the environment than relatively “innate” traits like fluid intelligence. The boundary between beliefs/preferences and abilities is fuzzy, but as I suggested at the end of my previous comment, I think for the purpose of this discussion it’s most useful to distinguish changes in value we can achieve (a) by changing the “environment” of existing people vs. (b) by adding more people to the pool.
Could you name a profile of such a person, and which of the types of work I named you think they’d maybe be as good at as the people I named?
What do you mean by “profile”? Saying what properties they have, but without identifying them? Or naming names or at least usernames? If the latter, I’d want to ask the people if they’re OK with me naming them publicly. But in principle happy to do either of these things, as I agree it’s a good way to check if my claim is plausible.
I think my definition of great might be a higher bar than yours, based on the proportion of people who I think meet it?
Maybe. When I said “they might be great”, I meant something roughly like: if it was my main goal to find people great at task X, I’d want to invest at least 1-10 hours per person finding out more about how good they’d be at X (this might mean talking to them, giving them some sort of trial tasks etc.) I’d guess that for between 5 and 50% of these people I’d eventually end up concluding they should work full-time doing X or similar.
Also note that originally I meant to exclude practice/experience from the relevant notion of “greatness” (i.e. it just includes talent/potential). So for some of these people my view might be something like “if they did 2 years of deliberate practice, they then would have a 5% to 50% chance of meeting the bar for X”. But I know think that probably the “marginal value from changing the environment vs. marginal value from adding more people” operationalization is more useful, which would require “greatness” to include practice/experience to be consistent with it.
If we disagree about the bar, I suspect that me having bad models about some of the examples you gave explains more of the disagreement than me generally dismissing high bars. “Functional programming” just doesn’t sound like the kind of task to me with high returns to super-high ability levels, and similar for community building; but it’t plausible that there are bundles of tasks involving these things where it matters a lot if you have someone whose ability is 6 instead of 5 standard deviations above the mean (not always well-defined, but you get the idea). E.g. if your “task” is “make a painting that will be held in similar regards as the Mona Lisa” or “prove P != NP” or “be as prolific as Ramanujan at finding weird infinite series for pi”, then, sure, I agree we need an extremely high bar.
Thanks for pointing this out. FWIW, I think there likely is both substantial disagreement between me and that person and that I misunderstood their view in some ways.