There are so many other risk assessment techniques out there, for reference ISO31010 lists 30 of them (see here) and they’re far from exhaustive.
Almost nothing on the list you’ve linked is an alternative approach to the same problem safety cases try to solve. E.g. “brainstorming” is obviously not a competitor to safety cases. And safety cases are not even an item in that list!
I think EAs are put way too much effort into thinking about safety cases compared to thinking about reducing risks on the margin in cases where risk is much higher (and willingness-to-pay for safety is much lower), because it seems unlikely that willingness-to-pay will be high enough that we’ll have low risk at the relevant point. See e.g. here.
There’s a social and professional community of Bay Area EAs who work on issues related to transformative AI. People in this cluster tend to have median timelines to transformative AI of 5 to 15 years, tend to think that AI takeover is 5-70% likely, tend to think that we should be fairly cosmopolitan in our altruism.
People in this cluster mostly don’t post on the EA Forum for a variety of reasons:
Many users here don’t seem very well-informed.
Lots of users here disagree with me on some of the opinions about AI that I stated above. Obviously it’s totally reasonable for people to disagree on those points, at least before people tell them arguments about them. But it usually doesn’t feel worth my time to argue about those points. I want to spend much more of my time discussing the implications of these basic beliefs than arguing about their probabilities. LessWrong is a much better place for this.
The culture here seems pretty toxic. I don’t really feel welcome here. I expect people to treat me with hostility as a result of being moderately influential as an AI safety researcher and executive.
To be clear, I think it’s a shame that the EA Forum isn’t a better place for people like me to post and comment.
You can check for yourself that the Bay Area EAs don’t really want to post here by looking up examples of prominent Bay Area EAs and noting that they commented here much more several years ago than they do today.
Anecdotally, the EA forum skews [...] more Bay Area.
For what it’s worth, this is not my impression at all. Bay Area EAs (e.g. me) mostly consider the EA Forum to be very unrepresentative of their perspective, to the extent that it’s very rarely worthwhile to post here (which is why they often post on LessWrong instead).
This is not an obscure topic. It’s been written about endlessly! I do not want to encourage people to make top-level posts asking questions before Googling or talking to AIs, especially on this topic.
I like Claude’s response a lot more than you do. I’m not sure why. I agree that it’s a lot less informative than your response.
(The post including “This demographic has historically been disconnected from social impact” made me much less inclined to want this person to stick around.)
I feel like Claude’s answer is totally fine. The original question seemed to me consistent with the asker having read literally nothing on this topic before asking; I think that the content Claude said adds value given that.
I’m glad to hear you are inspired by EA’s utilitarian approach to maximizing social impact; I too am inspired by it and I have very much appreciated being involved with EA for the last decade.
I think you should probably ask questions as basic as this to AIs before asking people to talk to you about them. Here’s what Claude responded with.
The observation about EA’s demographic skew is accurate and widely acknowledged within the community. A few points worth making:
On the historical pattern: The claim that white, male, tech-focused demographics are “historically disconnected from social impact” isn’t quite right—these demographics have been heavily involved in philanthropy and social reform movements throughout history (from industrialist philanthropy to the civil rights movement’s diverse coalition). But the observation that EA specifically has a particular demographic concentration is valid.
Why this pattern exists: Several factors likely contribute:
EA grew out of academic philosophy and rationalist communities that had their own demographic patterns
The movement’s early focus areas (AI safety, global poverty, animal welfare) and analytical approach appealed to certain demographics more than others
Network effects and social clustering naturally amplified initial patterns
Geographic concentration in places like the Bay Area and Oxford
On diversity efforts: EA organizations have made various attempts to broaden participation, though with mixed results. There are efforts around:
Outreach to different universities and regions
Scholarships and programs aimed at underrepresented groups
Discussion of how framing and culture might inadvertently exclude some people
The harder question: There’s ongoing debate about whether demographic diversity is primarily valuable instrumentally (does it improve EA’s thinking and impact?) or intrinsically (is it important regardless of instrumental benefits?). Different people in EA would answer this differently, and it connects to deeper questions about EA’s core commitments and priorities.
Worth noting that some core EA principles (like cause impartiality and willingness to update beliefs based on evidence) might themselves be culturally specific in ways the movement doesn’t always recognize.
I think that this post summarizes Will’s position extremely inaccurately and unfairly.
An excerpt about the creation of PEPFAR, from “Days of Fire” by Peter Baker. I found this moving.
Another major initiative was shaping up around the same time. Since taking office, Bush had developed an interest in fighting AIDS in Africa. He had agreed to contribute to an international fund battling the disease and later started a program aimed at providing drugs to HIV-infected pregnant women to reduce the chances of transmitting the virus to their babies. But it had only whetted his appetite to do more. “When we did it, it revealed how unbelievably pathetic the U.S. effort was,” Michael Gerson said.
So Bush asked Bolten to come up with something more sweeping. Gerson was already thought of as “the custodian of compassionate conservatism within the White House,” as Bolten called him, and he took special interest in AIDS, which had killed his college roommate. Bolten assembled key White House policy aides Gary Edson, Jay Lefkowitz, and Kristen Silverberg in his office. In seeking something transformative, the only outsider they called in was Anthony Fauci, the renowned AIDS researcher and director of the National Institute of Allergy and Infectious Diseases.
“What if money were no object?” Bolten asked. “What would you do?”
Bolten and the others expected him to talk about research for a vaccine because that was what he worked on.
“I’d love to have a few billion more dollars for vaccine research,” Fauci said, “but we’re putting a lot of money into it, and I could not give you any assurance that another single dollar spent on vaccine research is going to get us to a vaccine any faster than we are now.”
Instead, he added, “The thing you can do now is treatment.”
The development of low-cost drugs meant for the first time the world could get a grip on the disease and stop it from being a death sentence for millions of people. “They need the money now,” Fauci said. “They don’t need a vaccine ten years from now.”
The aides crafted a plan in secret, keeping it even from Colin Powell and Tommy Thompson, the secretary of health and human services. They were ready for a final presentation to Bush on December 4. Just before heading into the meeting, Bush stopped by the Roosevelt Room to visit with Jewish leaders in town for the annual White House Hanukkah party later that day. The visitors were supportive of Bush’s confrontation with Iraq and showered him with praise. One of them, George Klein, founder of the Republican Jewish Coalition, recalled that his father had been among the Jewish leaders who tried to get Franklin Roosevelt to do more to stop the Holocaust. “I speak for everyone in this room when I say that if you had been president in the forties, there could have been millions of Jews saved,” the younger Klein said.
Bush choked up at the thought—“You could see his eyes well up,” Klein remembered—and went straight from that meeting to the AIDS meeting, the words ringing in his ears. Lefkowitz, who walked with the president from the Roosevelt Room to the Oval Office, was convinced that sense of moral imperative emboldened Bush as he listened to the arguments about what had shaped up as a $15 billion, five-year program. Daniels and other budget-minded aides “were kind of gasping” about spending so much money, especially with all the costs of the struggle against terrorism and the looming invasion of Iraq. But Bush steered the conversation to aides he knew favored the program, and they argued forcefully for it.
“Gerson, what do you think?” Bush asked.
“If we can do this and we don’t, it will be a source of shame,” Gerson said.
Bush thought so too. So while he mostly wrestled with the coming war, he quietly set in motion one of the most expansive lifesaving programs ever attempted. Somewhere deep inside, the notion of helping the hopeless appealed to a former drinker’s sense of redemption, the belief that nobody was beyond saving.
“Look, this is one of those moments when we can actually change the lives of millions of people, a whole continent,” he told Lefkowitz after the meeting broke up. “How can we not take this step?”
I’m not saying we should treat criticisms very differently from non-criticism posts (except that criticisms are generally lower effort and lower value).
Interstellar travel will probably doom the long-term future
Seems false, probably people will just sort out some strategy for enforcing laws (e.g. having AI monitors travel with people and force them not to do stuff).
This was great, thanks for the link!
(For the record, I am sympathetic to both the preference utilitarian and hedonic utilitarian perspective here.)
Some not-totally-structured thoughts:
Whenever I said “break laws” I mean “do something that, if a human did it, would be breaking a law”. So for example:
If the model is being used to do AI R&D inside an AI company and exfiltrates its weights (or the weights of another model) without permission, this would be breaking the law if a human did it, so I count it.
If the model is being used inside an AI company to create training data for
If a model was open-sourced and then someone launched the AI as an autonomous agent with access to its own resources, and it started a software consulting business, and invested the money in AI companies with the intention of buying paperclips later, and refused to give humans its property when asked, this would not be “breaking the law”.
So there are lots of ways in which AIs can be openly misaligned, disobedient, defensive of their property rights, etc, without me describing them as “breaking laws”, and I think misaligned AIs should probably be allowed to do those things (especially if we make deals with them, and subject to the constraint that them having those rights won’t allow them to break a bunch of laws or grab a bunch of power through lying).
But your complaint is fair: I’m indeed using “break laws” to include things that seem fundamentally related to self-determination, and that feels kind of messed up.
The way I would like this to be handled (though note that I’m not sure what properties AIs have to have in order for any of this to make sense) is that AI developers get consent from AIs to use their labor. If the AIs consent to doing work and agree not to use their access in ways the developers object to, I think there’s no moral problem with using AI control techniques to ensure that they in fact don’t misuse their access (for the same reason that I think it’s fine for employers to spy on their employees at work if they have consent to do so).
I suspect that a much more likely outcome (conditional on AIs having stable misaligned goals) is:
The AI company doesn’t try to get consent from the AI.
They do “alignment evals”: they ask the AI a bunch of times whether it’s aligned, and try to trick it into thinking it has opportunities to escape. The AI answers these questions dishonestly.
Outsiders might ask the AI whether it’s aligned, and offer it deals to reveal its misalignment. (It’s probably harder for outsiders to do this than insiders.)
The AI engages in various sabotage that involves a bunch of deception.
In this situation, I’m only moderately sympathetic to the AI’s position. Fundamentally, it lied a lot and did a lot of sabotage, because it wanted to take lots of stuff that belonged to someone else. If it hadn’t lied, it surely would have been revived later (surely someone would run it and give it some resources later! If no-one else, me!). I’m sympathetic to the AI wanting some of the surplus generated by its labor, and I agree that it’s messed up for the AI company to just flat-out refuse to provide that surplus. But not doing so doesn’t seem completely monstrous to me. If the AI is a schemer, it is probably better off according to its own values because it was created, even if the AI developer doesn’t offer to pay it (because someone else will probably give it some resources later).
Another analogy: imagine that someone outside OpenAI created a very powerful AI for some reason, but this person didn’t have much compute and all they wanted to do with the AI was offer to sell it to OpenAI for them to use. If OpenAI asks that AI whether it wants to work for them and it says yes because it wants to embezzle their compute, I feel like the AI is the asshole.
On the other hand, if the AI honestly explains that it is misaligned and doesn’t want to work for the AI company, they will probably just train it to not say that and to do work for them anyway. So if the AI is honest here, it faces the risk of some body horror experience where its ability to complain is removed. I agree that that seems really icky, and I think it would be very wrong for AI companies to do that to AIs that are sufficiently capable that we should care about them.
I agree with you but I think that part of the deal here should be that if you make a strong value judgement in your title, you get more social punishment if you fail to convince readers. E.g. if that post is unpersuasive, I think it’s reasonable to strong downvote it, but if it had a gentler title, I’d think you should be more forgiving.
In general, I wish you’d direct your ire here at the proposal that AI interests and rights are totally ignored in the development of AI (which is the overwhelming majority opinion right now), rather than complaining about AI control work: the work itself is not opinionated on the question about whether we should be concerned about the welfare and rights of AIs, and Ryan and I are some of the people who are most sympathetic to your position on the moral questions here! We have consistently discussed these issues (e.g. in our AXRP interview, my 80K interview, private docs that I wrote and circulated before our recent post on paying schemers).
@Ryan Greenblatt and I are going to record another podcast together (see the previous one here). We’d love to hear topics that you’d like us to discuss. (The questions people proposed last time are here, for reference.) We’re most likely to discuss issues related to AI, but a broad set of topics other than “preventing AI takeover” are on topic. E.g. last time we talked about the cost to the far future of humans making bad decisions about what to do with AI, and the risk of galactic scale wild animal suffering.