Aryeh Englander is a mathematician and AI researcher at the Johns Hopkins University Applied Physics Laboratory. His work is focused on AI safety and AI risk analysis.
Aryeh Englander
Karma: 705
On presenting the case for AI risk
Crazy ideas sometimes do work
[Link] Luke Muehlhauser: Effective Altruism As I See It
I’ve been meaning to ask: Are there plans to turn your Cold Takes posts on AI safety and The Most Important Century into a published book? I think the posts would make for a very compelling book, and a book could reach a much broader audience and would likely get much more attention. (This has pros and cons of course, as you’ve discussed in your posts.)
[Disclaimers: My wife Deena works with Kat as a business coach—see my wife’s comment elsewhere on this post. I briefly met Kat and Emerson while visiting in Puerto Rico and had positive interactions with them. My personality is such that I have a very strong inclination to try to see the good in others, which I am aware can bias my views.]
A few random thoughts related to this post:
1. I appreciate the concerns over potential for personal retaliation, and the other factors mentioned by @Habryka and others for why it might be good to not delay this kind of post. I think those concerns and factors are serious and should definitely not be ignored. That said, I want to point out that there’s a different type of retaliation in the other direction that posting this kind of thing without waiting for a response can cause: Reputational damage. As others have pointed out, many people seem to update more strongly on negative reports that come first and less on subsequent follow up rebuttals. If it turned out that the accusations are demonstrably false in critically important ways, then even if that comes to light later the reputational damage to Kat, Emerson, and Drew may now be irrevocable.Reputation is important almost everywhere, but in my anecdotal experience reputation seems to be even more important in EA than in many other spheres. Many people in EA seem to have a very strong in-group bias towards favoring other “EAs” and it has long seemed to me that (for example) getting a grant from an EA organization often feels to be even more about having strong EA personal connections than for other places. (This is not to say that personal connections aren’t important for securing other types of grants or deals or the like, and it’s definitely not to say that getting an EA grant is only or even mostly about having strong EA connections. But from my own personal experience and from talking to quite a few others both in and out of EA, this is definitely how it feels to me. Note that I have received multiple EA grants in the past, and I have helped other people apply to and receive substantial EA grants.) I really don’t like this sort of dynamic and I’ve low-key complained about it for a long time—it feels unprofessional and raises all sorts of in-group bias flags. And I think a lot of EA orgs feel like they’ve gotten somewhat better about this over time. But I think it is still a factor.
Additionally, it sometimes feels to me that EA Forum dynamics tend to lead to very strongly upvoting posts and comments that are critical of people or organizations, especially if they’re more “centrally connected” in EA, while ignoring or even downvoting posts and comments in the other direction. I am not sure why the dynamic feels like this, and maybe I’m wrong about it really being a thing at all. Regardless, I strongly suspect that any subsequent rebuttal by Nonlinear would receive significantly fewer views and upvotes, even if the rebuttal were actually very strong.
Because of all this, I think that the potential for reputational harm towards Kat, Emerson, and Drew may be even greater than if this were in the business world or some other community. Even if they somehow provide unambiguous evidence that refutes almost everything in this post, I would not be terribly surprised if their potential to get EA funding going forward or to collaborate with EA orgs was permanently ended. In other words, I wouldn’t be terribly surprised if this post spelled the end of their “EA careers” even if the central claims all turned out to be false. My best guess is that this is not the most likely scenario, and that if they provide sufficiently good evidence then they’ll be most likely “restored” in the EA community for the most part, but I think there’s a significant chance (say 1%-10%) that this is basically the end of their EA careers regardless of the actual truth of the matter.
Does any of this outweigh the factors mentioned by @Habryka? I don’t know. But I just wanted to point out a possible factor in the other direction that we may want to consider, particularly if we want to set norms for how to deal with other such situations going forward.
2. I don’t have any experience with libel law or anything of the sort, but my impression is that suing for slander over this kind of piece is very much within the range of normal responses in the business world, even if in the EA world it is basically unheard of. So if your frame of reference is the world outside of EA then suing seems at least like a reasonable response, while if your frame of reference is the EA community then maybe it doesn’t. I’ll let others weigh in on whether my impressions on this are correct, but I didn’t notice others bring this up so I figured I’d mention it.
3. My general perspective on these kinds of things is that… well, people are complicated. We humans often seem to have this tendency to want our heroes to be perfect and our villains to be horrible. If we like someone we want to think they could never do anything really bad, and unless presented with extremely strong evidence to the contrary we’ll look for excuses for their behavior so that it matches our pictures of them as “good people”. And if we decide that they did do something bad, then we label them as “bad people” and retroactively reject everything about them. And if that’s hard to do we suffer from cognitive dissonance. (Cf. halo effect.)
But the reality, at least in my opinion, is that things are more complicated. It’s not just that there are shades of grey, it’s that people can simultaneously be really good people in some ways and really bad people in other ways. Unfortunately, it’s not at all a contradiction for someone to be a genuinely kind, caring, supportive, and absolutely wonderful person towards most of the people in their life, while simultaneously being a sexual predator or committing terrible crimes.
I’m not saying that any of the people mentioned in this post necessarily did anything wrong at all. My point here is mostly just to point out something that may be obvious to almost all of us, but which feels potentially relevant and probably bears repeating in any case. Personally I suspect that everybody involved was acting in what they perceived to be good faith and are / were genuinely trying to do the right thing, just that they’re looking at the situation through lenses based on very different perspectives and experiences and so coming to very different conclusions. (But see my disclaimer at the beginning of this comment about my personality bias coloring my own perspective.)
[Question] Brief summary of key disagreements in AI Risk
[Book] On Assessing the Risk of Nuclear War
Idea: Pay experts to provide detailed critiques / analyses of EA ideas
New article from Oren Etzioni
Day One Project Technology Policy Accelerator
[Link] GCRI’s Seth Baum reviews The Precipice
Half-baked ideas thread (EA / AI Safety)
I think I mostly lean towards general agreement with this take, but with several caveats as noted by others.
On the one hand, there are clearly important distinctions to be made between actual AI risk scenarios and Terminator scenarios. On the other hand, in my experience people pattern-matching to the Terminator usually doesn’t make anything seem less plausible to them, at least as far as I could tell. Most people don’t seem to have any trouble separating the time travel and humanoid robot parts from the core concern of misaligned AI, especially if you immediately point out the differences. In fact, in my experience, at least, the whole Terminator thing seems to just make AI risks feel more viscerally real and scary rather than being some sort of curious abstract thought experiment—which is how I think it often comes off to people.
Amusingly, I actually only watched Terminator 2 for the first time a few months ago, and I was surprised to realize that Skynet didn’t seem so far off from actual concerns about misaligned AI. Before that basically my whole knowledge of Skynet came from reading AI safety people complaining about how it’s nothing like the “real” concerns. In retrospect I was kind of embarrassed by the fact that I myself had repeated many of those complaints, even though I didn’t really know what Skynet was really about!
[Question] Pros and cons of working on near-term technical AI safety and assurance
Meta-comment: I noticed while reading this post and some of the comments that I had a strong urge to upvote any comment that was critical of EA and had some substantive content. Introspecting, I think this was partly due to trying to signal-boost critical comments because I don’t think we get enough of those, partly because I agreed with some of those critiques, … but I think mostly because it feels like part of the EA/rationalist tribal identity that self-critiquing should be virtuous. I also found myself being proud of the community that a critical post like this gets upvoted so much—look how epistemically virtuous we are, we even upvote criticisms!
On the one hand that’s perhaps a bit worrying—are we critiquing and/or upvoting critiques because of the content or because of tribal identity? On the other hand, I suppose if I’m going to have some tribal identity then being part of a tribe where it’s virtuous to give substantive critiques of the tribe is not a bad starting place.
But back on the first hand, I wonder if this would be so upvoted if it came from someone outside of EA, didn’t include things about how the author really agrees with EA overall, and perhaps was written in a more polemical style. Are we only virtuously upvoting critiques from fellow tribe members, but if it came as an attack from outside then our tribal defense instincts would kick in and we would fight against the perceived threat?
[EDIT: To be clear, I am not saying anything about this particular post. I happened to agree with a lot of the content in the OP, and I have voiced these and related concerns several times myself.]
As I mentioned on one of those Facebook threads: At least don’t bill the event as a global conference for EA people and then tell people no you can’t come. Call it maybe the EA Professionals Networking Event or something, which (a) makes it clear this is for networking and not the kind of academic conference people might be used to, and (b) implies this might be exclusive. But if you bill it as a global conference, then make it be like a global conference. And at the very least make it very clear that it’s exclusive! Personally I didn’t notice any mention of exclusivity at all in any EA Global posts or advertising until I heard about people actually getting rejected and feeling bad about that.