On factors of potential relevance to moral weight, including some that could intuitively upweight many invertebrates, see also my Preliminary thoughts on moral weight.
lukeprog
Karma: 992
Presumably, though I know very little about that and don’t know how much value would be added there by someone focused on worst case scenarios (over their replacement).
This all sounds right to me, though I think some people have different views, and I’m hardly an expert. Speaking for myself at least, the things you point to are roughly why I wanted the “maybe” in front of “relevant roles in government.” Though one added benefit of doing security in government is that, at least if you get a strong security clearance, you might learn classified helpful things about e.g. repelling state-originating APTs.
Yeah, something closer to the former.
Very minor: “GJP” should be “GJI.” Good Judgment Project ended with the end of the IARPA ACE tournaments. The company that pays superforecasters from that project to continue making forecasts for clients is Good Judgment Inc.
I think we meant a bit of (b) and (c) but especially (a).
IIRC the main concern in the earlier conversations was about how many high-impact roles of this type there might really be in the next couple decades. Probably the number is smaller than (e.g.) the number of similarly high-impact “AI policy” roles, but (as our post says) we think the number of high-impact roles of this type will be substantial. And given how few GCR-focused people there are in general, and how few of them are likely a personal fit for this kind of career path anyway, it might well be that even if many of the people who are a good fit for this path pursue it, that would still not be enough to meet expected need in the next couple decades.
The key roles we have in mind are a bit closer to what is sometimes called “security officer,” i.e. someone who can think through (novel, GCR-focused) threat models, plausibly involving targeted state-based attacks, develop partly-custom system and software solutions that are a match to those threat models, think through and gather user feedback about tradeoffs between convenience and security of those solutions, develop and perhaps deliver appropriate training for those users, etc. Some of this might include things like “protect some unusual configuration of AWS services,” but I imagine that might also be something that the security officer is able to outsource. We’ve tried working with a few security consultants, and it hasn’t met our needs so far.
Projects like “develop novel cryptographic methods” might also be useful in some cases — see my bullet points on research (rather than implementation) applications of security expertise in the context of AI — but they aren’t the modal use-case we’re thinking of.
But also, we haven’t studied this potential career path to the level of depth that (e.g.) 80,000 Hours typically does when developing a career profile, so we have more uncertainty about many of the details here even than is typically represented in an 80,000 Hours career profile.
You could dive into the specific examples in the spreadsheet linked here (the MIRI AI predictions dataset).
Information security careers for GCR reduction
Exciting — I look forward to the rest! Ya’ll might want to consider writing a target article (summarizing your findings) for Animal Sentience; I suspect Harnad would be interested.
Cases where the scientific knowledge was actually lost and then rediscovered much later provide especially strong evidence w.r.t. the discovery counterfactuals. E.g. Hero’s eolipile or al-Kindi’s development of relative frequency analysis for decoding messages. Probably there are far more cases of this than we realize, because the evidence that someone somewhere once had the knowledge and then lost it has itself been lost; e.g. we could easily have just never rediscovered the Antikythera mechanism.
I looked into this a bit. Unfortunately the quality of evidence in sleep medicine was underwhelming, e.g. on behavioral treatments.
BTW my “reflections on the 2018 RA recruiting round” post is now up, here.
I agree that if it’s true that “many EAs feel that either they’re working at a top EA org or they’re not contributing much,” then that is much worse than anything about application time cost and urgently needs to be fixed. I’ve never felt that way about EA org work vs. alternatives, so I may have just missed that this is a message many people are getting.
E.g. Scott’s post also says:
Should also acknowledge the possibility that “talent-constrained” means the world needs more clean meat researchers, malaria vaccine scientists, and AI programmers, and not just generic high-qualification people applying to EA organizations. This wasn’t how I understood the term but it would make sense.
…and my reply is “Yes, talent-constrained also means those other things, and it’s a big problem if that was unclear to a noticeable fraction of the community.”
FWIW I suspect there’s also something a bit more subtle going on than overly narrow misunderstandings of “talent-constrained,” e.g. something like Max Daniel’s hypothesis.
Thanks, that all makes sense to me. Will think more about this. Also still curious to hear replies from others here.
Thanks for +1ing the above comment. I’d be keen to hear your reply to this comment, too.
Thanks for sharing. As someone who spends a lot of time trying to fill EA meta/longtermist talent gaps — e.g. by managing Open Phil RA recruiting, helping to match the strongest applicants we don’t hire to other openings, and by working on field-building in AI strategy/policy (e.g. CSET) — hearing stories like yours is unnerving.
What changes to the landscape, or hiring processes, or whatever, do you think would’ve made the most difference in your case?
I’m also curious to hear your reaction to my comment elsewhere about available paths:
There are lots of exciting things for EAs to do besides “apply to one or more of the 20 most competitive jobs at explicitly EA-motivated employers,” including “keep doing what you’re doing and engage with EA as an exciting hobby” and “apply to key positions in top-priority cause areas that are on the 80,000 Hours Job Board but aren’t at one of a handful of explicitly EA-motivated orgs” and “do earn to give for a while while gaining skills and then maybe transition to more direct work later or maybe not,” as well as other paths that are specific to particular priority causes, e.g. for AI strategy & policy I’d be excited to see EAs (a) train up in ML, for later work in either AI safety or AI strategy/policy, (b) follow these paths into a US AI policy career (esp. for US citizens, and esp. now that CSET exists), and (c) train up as a cybersecurity expert (I hope to say more later about why this path should be especially exciting for AI-interested EAs; also the worst that happens is that you’ll be in extremely high demand and highly paid).
Sorry to hear about your long, very difficult experience. I think part of what happened is that it did in fact get a lot harder to get a job at leading EA-motivated employers in the past couple years, but that wasn’t clear to many EAs (including me, to some extent) until very recently, possibly as recently as this very post. So while it’s good news that the EA community has grown such that these particular high-impact jobs can attract talent sufficient for them to be so competitive, it’s unfortunate that this change wasn’t clearer sooner, and posts like this one help with that, albeit not soon enough to help mitigate your own 1.5 years of suffering.
Also, the thing about some people not having runway is true and important, and is a major reason Open Phil pays people to take our remote work tests, and does quite a few things for people who do an in-person RA trial with us (e.g. salary, health benefits, moving costs, severance pay for those not made a subsequent offer). We don’t want to miss out on great people just because they don’t have enough runway/etc. to interact with our process.
FWIW, I found some of your comments about “elite culture” surprising. For context: I grew up in rural Minnesota, then dropped out of counseling psychology undergrad at the University of Minnesota, then worked at a 6-person computer repair shop in Glendale, CA. Only in the past few years have I begun to somewhat regularly interact with many people from e.g. top schools and top tech companies. There are aspects of interacting with such “elites” that I’ve had to learn on the fly and to some degree am still not great at, but in my experience the culture in those circles is still pretty different from the culture at major EA-motivated employers, even though many of the staff at EA-motivated employers are now people who e.g. graduated from schools like Oxford or Harvard. For example, it’s not my experience that people at major EA organizations are as effusively positive as many people in non-EA “elite” circles are. In fact, I would’ve described the culture at the EA organizations I interact with the most in sorta opposite terms, in that it’s hard to get them excited about things. E.g. if you tell one of my Open Phil RA colleagues about a new study in Nature on some topic they care about, a pretty common reaction is to shrug and say “Yeah but who knows if it’s true; most of the time we dig into a top-journal study, it completely falls apart.” Or if you tell people at most EA orgs about a cool-sounding global health or poverty-reduction intervention, they’ll probably say “Could be interesting, but very low chance it’ll end up looking as cost-effective as AMF or even GiveDirectly upon further investigation, so: meh.” Also, EA-motivated employers are generally not as “credentialist,” in my experience, as most “elite” employers (perhaps except for tech companies).
Finally, re: “you never know for sure if it’s not just perfect meritocracy correctly filtering [certain people out].” I can’t speak to your case in particular, but at least w.r.t. Open Phil’s RA recruiting efforts (which I’ve been managing since early 2018), I think I am sure it’s not a perfect meritocracy. We think our application process probably has a high false negative rate (i.e. rejecting people who are actually strong fits, or would be with 3mo of training), and it’s just very difficult to reduce the false negative rate without also greatly increasing the false positive rate. Just to make this more concrete: in our 2018 RA hiring round, if somebody scored really well on our stage-3 work test, we typically thought “Okay, decent chance this person is a good fit,” but when somebody scored medium/low on it, we often threw up our hands and said “No clue if this person is a good fit or not, there are lots of reasons they could’ve scored poorly without actually being a poor fit, I guess we just don’t get to know either way without us and them paying infeasibly huge time costs.” (So why not just improve that aspect of our work tests? We’re trying, e.g. by contracting several “work test testers,” but it’s harder than one might think, at least for such ill-defined “generalist” roles.)
On the difference between the role we’ve tried to hire for at Open Phil specifically and a typical Security Analyst or Security Officer role, a few things come to mind, though we also think we don’t yet have a great sense of the range of security roles throughout the field. One possible difference is that many security roles focus on security systems for a single organization, whereas we’ve primarily looked for someone who could help both Open Phil and some of our grantees, each of whom have potentially quite different needs. Another possible difference is that our GCR focus in AI and biosecurity leads us to some non-standard threat models, and it has been difficult thus far for us to find experienced security experts who readily adapt standard security thinking to a somewhat different set of threat models.
Re: industry roles that would be particularly good or bad preparation. My guess is that for the GCR-mitigating roles we discuss above (i.e. not just potential future roles at Open Phil), the better-preparation for roles will tend to (a) expose one to many different types of challenges, and different aspects of those challenges, rather than being very narrowly scoped, (b) involve threat modeling of, and defense from, very capable and well-resourced attackers, and (c) require some development of novel solutions (not necessary new crypto research; could also just be new configurations of interacting hardware/software systems and user behavior policies and training), among other things.