FWIW I was substantially positively surprised by the amount and quality of the work you put out in 2019, though I didn’t vet any of it in depth. (And prior to 2019 I think I wasn’t aware of Rethink.)
lukeprog
Karma: 1,185
FWIW, it’s not clear to me that AI alignment folks with different agendas have put less effort into (or have made less progress on) understanding the motivations for other agendas than is typical in other somewhat-analogous fields. Like, MIRI leadership and Paul have put >25 (and maybe >100, over the years?) hours into arguing about merits of their differing agendas (in person, on the web, in GDocs comments), and my impression is that central participants to those conversations (e.g. Paul, Eliezer, Nate) can pass the others’ ideological Turing tests reasonably well on a fair number of sub-questions and down 1-3 levels of “depth” (depending on the sub-question), and that might be more effort and better ITT performance than is typical for “research agenda motivation disagreements” in small niche fields that are comparable on some other dimensions.
Interesting stuff, thanks!
Thanks!
Notes on ‘Atomic Obsession’ (2009)
Awesome, thanks for letting us know!
Thanks! I knew there was one major study I was missing from the 70s, and that I had emailed people about before, but I couldn’t track it down when I was writing this post, and I’m pretty sure this is the one I was thinking of. Of course, this study suffers from several of the problems I list in the post.
The linked “full guide” seem to require sign-in?
Nice work!
Great post!
Even just a few decades ago, a longtermist altruist would not have thought of risk from AI or synthetic biology, and wouldn’t have known that they could have taken action on them.
Minor point, but I think this is unclear. On AI see e.g. here. On synbio I’m less familiar but I’m guessing someone more than a few decades ago was able to think thoughts like “Once we understand cell biology realy well, seems like we might be able to engineer pathogens much more destructive than those served up by nature.”
On the difference between the role we’ve tried to hire for at Open Phil specifically and a typical Security Analyst or Security Officer role, a few things come to mind, though we also think we don’t yet have a great sense of the range of security roles throughout the field. One possible difference is that many security roles focus on security systems for a single organization, whereas we’ve primarily looked for someone who could help both Open Phil and some of our grantees, each of whom have potentially quite different needs. Another possible difference is that our GCR focus in AI and biosecurity leads us to some non-standard threat models, and it has been difficult thus far for us to find experienced security experts who readily adapt standard security thinking to a somewhat different set of threat models.
Re: industry roles that would be particularly good or bad preparation. My guess is that for the GCR-mitigating roles we discuss above (i.e. not just potential future roles at Open Phil), the better-preparation for roles will tend to (a) expose one to many different types of challenges, and different aspects of those challenges, rather than being very narrowly scoped, (b) involve threat modeling of, and defense from, very capable and well-resourced attackers, and (c) require some development of novel solutions (not necessary new crypto research; could also just be new configurations of interacting hardware/software systems and user behavior policies and training), among other things.
On factors of potential relevance to moral weight, including some that could intuitively upweight many invertebrates, see also my Preliminary thoughts on moral weight.
Presumably, though I know very little about that and don’t know how much value would be added there by someone focused on worst case scenarios (over their replacement).
This all sounds right to me, though I think some people have different views, and I’m hardly an expert. Speaking for myself at least, the things you point to are roughly why I wanted the “maybe” in front of “relevant roles in government.” Though one added benefit of doing security in government is that, at least if you get a strong security clearance, you might learn classified helpful things about e.g. repelling state-originating APTs.
Yeah, something closer to the former.
Very minor: “GJP” should be “GJI.” Good Judgment Project ended with the end of the IARPA ACE tournaments. The company that pays superforecasters from that project to continue making forecasts for clients is Good Judgment Inc.
I think we meant a bit of (b) and (c) but especially (a).
IIRC the main concern in the earlier conversations was about how many high-impact roles of this type there might really be in the next couple decades. Probably the number is smaller than (e.g.) the number of similarly high-impact “AI policy” roles, but (as our post says) we think the number of high-impact roles of this type will be substantial. And given how few GCR-focused people there are in general, and how few of them are likely a personal fit for this kind of career path anyway, it might well be that even if many of the people who are a good fit for this path pursue it, that would still not be enough to meet expected need in the next couple decades.
The key roles we have in mind are a bit closer to what is sometimes called “security officer,” i.e. someone who can think through (novel, GCR-focused) threat models, plausibly involving targeted state-based attacks, develop partly-custom system and software solutions that are a match to those threat models, think through and gather user feedback about tradeoffs between convenience and security of those solutions, develop and perhaps deliver appropriate training for those users, etc. Some of this might include things like “protect some unusual configuration of AWS services,” but I imagine that might also be something that the security officer is able to outsource. We’ve tried working with a few security consultants, and it hasn’t met our needs so far.
Projects like “develop novel cryptographic methods” might also be useful in some cases — see my bullet points on research (rather than implementation) applications of security expertise in the context of AI — but they aren’t the modal use-case we’re thinking of.
But also, we haven’t studied this potential career path to the level of depth that (e.g.) 80,000 Hours typically does when developing a career profile, so we have more uncertainty about many of the details here even than is typically represented in an 80,000 Hours career profile.
Is it easy to say more about (1) which personality/mindset traits might predict infosec fit, and (2) infosec experts’ objections to typical GCR concerns of EAs?