I was also surprised by this, and I wonder how many people interpreted “It is acceptable for an EA org to break minor laws” as “It is acceptable for an EA org to break laws willy-nilly as long as it feels like the laws are ‘minor’”, rather than interpreting it as “It is acceptable for an EA org to break at least one minor law ever”.
How easy is it to break literally zero laws? There are an awful lot of laws on the books in the US, many of which aren’t enforced.
If someone uses the phrase “saving the world” on any level approaching consistent, run.
I use this phrase a lot, so if you think this phrase is a red flag, well, include me on the list of people who have that flag.
If someone pitches you on something that makes you uncomfortable, but for which you can’t figure out your exact objection—or if their argument seems wrong but you don’t see the precise hole in their logic—it is not abandoning your rationality to listen to your instinct.
Agreed (here, and with most of your other points). Instincts like those can be wrong, but they can also be right. “Rationality” requires taking all of the data into consideration, including illegible hunches and intuitions.
If someone says “the reputational risks to EA of you publishing this outweigh the benefits of exposing x’s bad behavior. if there’s even a 1% chance that AI risk is real, then this could be a tremendously evil thing to do”, nod sagely then publish that they said that.
Agreed!
Yeah, though if I learned “Alice is just not the sort of person to loudly advocate for herself” it wouldn’t update me much about Nonlinear at this point, because (a) I already have a fair amount of probability on that based on the publicly shared information, and (b) my main concerns are about stuff like “is Nonlinear super cutthroat and manipulative?” and “does Nonlinear try to scare people into not criticizing Nonlinear?”.
Those concerns are less directly connected to the vegan-food thing, and might be tricky to empirically distinguish from the hypothesis “Alice and/or Chloe aren’t generally the sort of people to be loud about their preferences” if we focus on the food situation.
(Though I cared about the vegan-food thing in its own right too, and am glad Kat shared more details.)
I’d appreciate it if Nonlinear spent their limited resources on the claims that I think are most shocking and most important, such as the claim that Woods said “your career in EA would be over with a few DMs” to a former employee after the former employee was rumored to have complained about the company.
I agree that this is a way more important incident, but I downvoted this comment because:
I don’t want to discourage Nonlinear from nitpicking smaller claims. A lot of what worries people here is a gestalt impression that Nonlinear is callous and manipulative; if that impression is wrong, it will probably be because of systematic distortions in many claims, and it will probably be hard to un-convince people of the impression without weighing in on lots of the claims, both major and minor.
I expect some correlation between “this concern is easier to properly and fully address” and “this concern is more minor”, so I think it’s normal and to be expected that Nonlinear would start with relatively-minor stuff.
I do think it’s good to state your cruxes, but people’s cruxes will vary some; I’d rather that Nonlinear overshare and try to cover everything, and I don’t want to locally punish them for addressing a serious concern even if it’s not the top concern. “I’d appreciate if Nonlinear spent their limited resources...” makes it sound like you didn’t want Nonlinear to address the veganism thing at all, which I think would have been a mistake.
I’m generally wary of the dynamic “someone makes a criticism of Nonlinear, Nonlinear addresses it in a way that’s at least partly exculpatory, but then a third party steps in to say ‘you shouldn’t have addressed that claim, it’s not the one I care about the most’”. This (a) makes it more likely that Nonlinear will feel pushed into not-correcting-the-record on all sorts of false claims, and (b) makes it more likely that EAs will fail to properly dwell on each data point and update on it (because they can always respond to a refutation of X by saying ‘but what about Y?!’ when the list of criticisms is this danged long).
I also think it’s pretty normal and fine to need a week to properly address big concerns. Maybe you’ve forgotten a bunch of the details and need to fact-check things. Maybe you’re emotionally processing stuff and need another 24h to draft a thing that you trust to be free of motivated reasoning.
I think it’s fine to take some time, and I also think it’s fine to break off some especially-easy-to-address points and respond to those faster.
This also updates me about Kat’s take (as summarized by Ben Pace in the OP):
Kat doesn’t trust Alice to tell the truth, and that Alice has a history of “catastrophic misunderstandings”.
When I read the post, I didn’t see any particular reason for Kat to think this, and I worried it might be just be an attempt to dismiss a critic, given the aggressive way Nonlinear otherwise seems to have responded to criticisms.
With this new info, it now seems plausible to me that Kat was correct (even though I don’t think this justifies threatening Alice or Ben in the way Kat and Emerson did). And if Kat’s not correct, I still update that Kat was probably accurately stating her epistemic state, and that a lot of reasonable people might have reached the same epistemic state.
I think that there’s a big difference between telling everyone “I didn’t get the food I wanted, but they did get/offer to cook me vegan food, and I told them it was ok!” and “they refused to get me vegan food and I barely ate for 2 days”.
Agreed.
It also seems totally reasonable that no one at Nonlinear understood there was a problem. Alice’s language throughout emphasizes how she’ll be fine, it’s no big deal [...] I do not think that these exchanges depict the people at Nonlinear as being cruel, insane, or unusual as people.
100% agreed with this. The chat log paints a wildly different picture than what was included in Ben’s original post.
Given my experience with talking with people about strongly emotional events, I am inclined towards the interpretation where Alice remembers the 15th with acute distress and remembers it as ‘not getting her needs met despite trying quite hard to do so’, and the Nonlinear team remembers that they went out of their way that week to get Alice food—which is based on the logs from the 16th clearly true! But I don’t think I’d call Alice a liar based on reading this
Agreed. I did update toward “there’s likely a nontrivial amount of distortion in Alice’s retelling of other things”, and toward “normal human error and miscommunication played a larger role in some of the Bad Stuff that happened than I previously expected”. (Ben’s post was still a giant negative update for me about Nonlinear, but Kat’s comment is a smaller update in the opposite direction.)
Without making any comment about the accuracy or inaccuracy of this post, I would just point out that nobody in EA should be shocked that an organization (e.g. Nonlinear) that is being libeled (in its view) would threaten a libel suit to deter the false accusations (as they see them), to nudge the author(e.g. Ben Pace) towards making sure that their negative claims are factually correct and contextually fair.
Per https://www.dmlp.org/legal-guide/proving-fault-actual-malice-and-negligence (h/t kave):
[...] To support a claim for defamation, in most states a private figure need only show negligence by the publisher, a much lower standard than “actual malice.”
[...] A plaintiff can establish negligence on the part of the defendant by showing that the defendant did not act with a reasonable level of care in publishing the statement at issue. This basically turns on whether the defendant did everything reasonably necessary to determine whether the statement was true, including the steps the defendant took in researching, editing, and fact checking his work. Some factors that the court might consider include:
the amount of research undertaken prior to publication;
the trustworthiness of sources;
attempts to verify questionable statements or solicit opposing views; and
whether the defendant followed other good journalistic practices.
Ben says he spent “100-200 hours” researching this post, which is way beyond the level of thoroughness we should require for criticizing an organization on LessWrong or the EA Forum. I think it’s pretty clear that the post reflects this work: whether or not the post is wrong (e.g., maybe Ben was tricked by Alice and Chloe), it’s obvious that Ben did a supererogatory amount of fact-checking, investigating, and collecting takes from multiple parties.
I think there should be a strong norm against threatening people with libel merely for saying a falsehood; the standard should at minimum be that you have good reason to think the person is deliberately lying or indulging in tabloid-newspaper levels of bullshit.
I think the standard should be way higher than that, given the chilling effect of litigiousness; but in this case I think it’s sufficient to say that Ben clearly isn’t lying or flippantly disregarding the truth (whether or not he got some of the facts wrong), so threatening him with a lawsuit is clearly inappropriate. The standard for libel lawsuits needs to be way higher than “I factually disagree with something you said (that makes me look bad)”.
This seems particularly clear in the case of non-anonymous posts like Ben’s. Ben posted a thing that risks damaging Nonlinear’s reputation. In the process, he’s put his own reputation at risk: Nonlinear can publicly respond with information that shows Ben was wrong (and perhaps negligent, unfair, etc.), causing us to put a lot less stock in Ben’s word next time around.
Alice and Chloe are anonymous, but by having a named individual vouch for them to some degree, we create a situation where ordinary reputational costs can do a good job of incentivizing honesty on everyone’s part.
Agreed on all counts.
Dustin Moskovitz comments on Twitter:
The deployment problem is part of societal response to me, not separate.
[...] Eg race dynamics, regulation (including ability to cooperate with competitors), societal pressure on leaders, investment in watchdogs (human and machine), safety testing norms, whether things get open sourced, infohazards.
“The deployment problem is hard and weird” comes from a mix of claims about AI (AGI is extremely dangerous, you don’t need a planet-sized computer to run it, software and hardware can and will improve and proliferate by default, etc.) and about society (“if you give a decent number of people the ability to wield dangerous AGI tech, at least one or them will choose to use it”).
The social claims matter — two people who disagree about how readily Larry Page and/or Mark Zuckerberg would put the world at risk might as a result disagree about whether a Good AGI Project has median 8 months vs. 12 months to do a pivotal act.
When I say “AGI ruin rests on strong claims about the alignment problem and deployment problem, not about society”, I mean that the claims you need to make about society in order to think the alignment and deployment problems are that hard and weird, are weak claims (e.g. “if fifty random large AI companies had the ability to use dangerous AGI, at least one would use it”), and that the other claims about society required for high p(doom) are weak too (e.g. “humanity isn’t a super-agent that consistently scales up its rationality and effort in proportion to a problem’s importance, difficulty, and weirdness”).
Arguably the difficulty of the alignment problem itself also depends in part on claims about society. E.g., the difficulty of alignment depends on the difficulty of the task we’re aligning, which depends on “what sort of task is needed to end the acute x-risk period?”, which depends again on things like “will random humans destroy the world if you hand them world-destroying AGI?”.
The thing I was trying to communicate (probably poorly) isn’t “Alignment, Deployment, and Society partitions the space of topics”, but rather:
High p(doom) rests on strong claims about AI/compute/etc. and quite weak claims about humanity/society.
The most relevant claims (~all the strong ones, and an important subset of the weak ones) are mostly claims about the difficulty, novelty, and weirdness of the alignment and deployment problems.
To chime in, I think it would be helpful to distinguish between:
1. AI risks on a ‘business as usual’ model, where society continues as it was before, ie not doing much
and
2. AI risks given different levels of society response.
I like this! Richard Ngo and Eliezer discuss this a bit in Ngo’s view on alignment difficulty:
[Ngo] (Sep. 25 [2021] Google Doc) Perhaps the best way to pin down disagreements in our expectations about the effects of the strategic landscape is to identify some measures that could help to reduce AGI risk, and ask how seriously key decision-makers would need to take AGI risk for each measure to be plausible, and how powerful and competent they would need to be for that measure to make a significant difference. Actually, let’s lump these metrics together into a measure of “amount of competent power applied”. Some benchmarks, roughly in order (and focusing on the effort applied by the US):
|
[Yudkowsky][12:02] (Sep. 25 [2021] comment)
This level of effort starts to buy significant amounts of time. This level will not be reached, nor approached, before the world ends. |
See the post for more discussion, including an update from Eliezer: “I’ve updated somewhat off of Carl Shulman’s argument that there’s only one chip supply chain which goes through eg a single manufacturer of lithography machines (ASML), which could maybe make a lock on AI chips possible with only WW1 levels of cooperation instead of WW2.”
Eliezer’s Pausing AI Developments Isn’t Enough. We Need to Shut it All Down is also trying to do something similar, as is his (written-in-2017) post Six Dimensions of Operational Adequacy in AGI Projects.
I interpret “Pausing AI Developments Isn’t Enough” as saying “if governments did X, then we’d still probably be in enormous amounts of danger, but there would now be a non-tiny probability of things going well”. (Maybe even a double-digit probability of things going well for humanity.)
Eliezer doesn’t think governments are likely to do X, but he thinks we should make a desperate effort to somehow pull off getting governments to do X anyway on EV grounds: there aren’t any markedly-more-hopeful alternatives, and we’re all dead if we fail.
(Though there may be some other similarly-hopeless-but-worth-trying-anyway options, like moonshot attempts to solve the alignment problem, or a Manhattan Project to build nanotechnology, or what-have-you. My Eliezer-model wants highly competent and sane people pursuing all of these unlikely-to-work ideas in parallel, because then it’s more likely that at least one succeeds.)
Six Dimensions of Operational Adequacy in AGI Projects divides amounts of effort into “token”, “improving”, “adequate”, “excellent”, and “unrealistic”, but it doesn’t say how high the risk level is under different buckets. I think this is mostly because Eliezer’s model gives a macroscopic probability to success if an AGI project is “adequate” on all six dimensions at once, and a tiny probability to success if it falls short of adequacy on any dimension.
My Eliezer-model thinks that “token” and “improving” both mean you’re dead, and he doesn’t necessarily think he can give meaningful calibrated confidences that distinguish degrees of deadness when the situation looks that bad.
(A) we’re almost certainly screwed, whatever we do
(B) we might be screwed, but not if we get our act together, which we’re not doing now
(C) we might be screwed, but not if we get our act together, which I’m confident will happen anyway
(D) there’s nothing to worry about in the first place.
Obviously, these aren’t the only options. (A), (C), and (D) imply that few or no additional resources are useful, whereas (B) implies extra resources are worthwhile. My impression is Yudowsky’s line is (A).
Seems like a wrong framing to me. My model (and Eliezer’s) is that A and B are both right: We’re almost certainly screwed, whatever we do; but not if humanity gets its act together in a massive way (which we’re currently not doing, but should try to do because otherwise we’re dead).
“No additional resources are useful” makes it sound like Eliezer is advocating for humanity to give up, which he obviously isn’t doing. Rather, my view and Eliezer’s is that we should try to save the world (because the alternative is ruin), even though some things will have to go miraculously right in order for our efforts to succeed.
Note that if it were costless to make the title way longer, I’d change this post’s title from “AGI ruin mostly rests on strong claims about alignment and deployment, not about society” to the clearer:
The AGI ruin argument mostly rests on claims that the alignment and deployment problems are difficult and/or weird and novel, not on strong claims about society
Your example with humanity fails because humans have always and continue to be a social species that is dependent on each other.
I would much more say that it fails because humans have human values.
Maybe a hunter-gatherer would have worried that building airplanes would somehow cause a catastrophe? I don’t exactly see why; the obvious hunter-gatherer rejoinder could be ‘we built fire and spears and our lives only improved; why would building wings to fly make anything bad happen?’.
Regardless, it doesn’t seem like you can get much mileage via an analogy that sticks entirely to humans. Humans are indeed safe, because “safety” is indexed to human values; when we try to reason about non-human optimizers, we tend to anthropomorphize them and implicitly assume that they’ll be safe for many of the same reasons. Cf. The Tragedy of Group Selectionism and Anthropomorphic Optimism.
You argue that most paths to some ambitious goal like whole-brain emulation end terribly for humans, because how else could the AI do whole-brain emulation without subjugating, eliminating or atomising everyone?
‘Wow, I can’t imagine a way to do something so ambitious without causing lots of carnage in the process’ is definitely not the argument! On the contrary, I think it’s pretty trivial to get good outcomes from humans via a wide variety of different ways we could build WBE ourselves.
The instrumental convergence argument isn’t ‘I can’t imagine a way to do this without killing everyone’; it’s that sufficiently powerful optimization behaves like maximizing optimization for practical purposes, and maximizing-ish optimization is dangerous if your terminal values aren’t included in the objective being maximized.
If it helps, we could maybe break the disagreement about instrumental convergence into three parts, like:
Would a sufficiently powerful paperclip maximizer kill all humans, given the opportunity?
Would sufficiently powerful inhuman optimization of most goals kill all humans, or are paperclips an exception?
Is ‘build fast-running human whole-brain emulation’ an ambitious enough task to fall under the ‘sufficiently powerful’ criterion above? Or if so, is there some other reason random policies might be safe if directed at this task, even if they wouldn’t be safe for other similarly-hard tasks?
For a STEM-capable AGI (or any intelligence for that matter) to do new science, it would have to interact with the physical environment to conduct experiments.
Or read arXiv papers and draw inferences that humans failed to draw, etc.
Doesn’t this significantly throttles the speed of AGI gaining advantage over humanity, giving us more time for alignment?
I expect there’s a ton of useful stuff you can learn (that humanity is currently ignorant about) just from looking at existing data on the Internet. But I agree that AGI will destroy the world a little slower in expectation because it may get bottlenecked on running experiments, and it’s at least conceivable that at least one project will decide not let it run tons of physical experiments.
(Though I think the most promising ways to save the world involve AGIs running large numbers of physical experiments, so in addition to merely delaying AGI doom by some number of months, ‘major labs don’t let AGIs run physical experiments’ plausibly rules out the small number of scenarios where humanity has a chance of surviving.)
The full text of the TIME piece is now available on the EA Forum here, with two clarifying notes by Eliezer added at the end.
Yep, I think this is a big problem.
More generally, I think a lot of EAs give lip service to the value of people trying weird new ambitious things, “adopt a hits-based approach”, “if you’re never failing then you’re playing it too safe”, etc.; but then we harshly punish visible failures, especially ones that are the least bit weird. In cases like those, I think the main solution is to be more forgiving of failures, rather than to give up on ambitious projects.
From my perspective, none of this is particularly relevant to what bothers me about Ben’s post and Nonlinear’s response. My biggest concern about Nonlinear is their attempt to pressure people into silence (via lawsuits, bizarre veiled threats, etc.), and “I really wish EAs would experiment more with coercing and threatening each other” is not an example of the kind of experimentalism I’m talking about when I say that EAs should be willing to try and fail at more things (!).
“Keep EA weird” does not entail “have low ethical standards”. Weirdness is not an excuse for genuinely unethical conduct.
I think the failures that seem like the biggest deal to me (Nonlinear threatening people and trying to shut down criticism and frighten people) genuinely are matters of character and lack of integrity, not matters of bad engineering. I agree that not all of the failures in Ben’s OP are necessarily related to any character/integrity issues, and I generally like the lens you’re recommending for most cases; I just don’t think it’s the right lens here.