AI safety researcher
Thomas Kwađč
Karma: 3,936
Being really good at your job is a good way to achieve impact in general, because your âimpact above replacementâ is what counts. If a replacement level employee who is barely worth hiring has productivity 100, and the average productivity is 150, the average employee will get 50 impact above replacement. If you do your job 1.67x better than average (250 productivity), you earn 150 impact above replacement, which is triple the average.
I strongly disagree with a couple of claims:
MIRIâs business model relies on the opposite narrative. MIRI pays Eliezer Yudkowsky $600,000 a year. It pays Nate Soares $235,000 a year. If they suddenly said that the risk of human extinction from AGI or superintelligence is extremely low, in all likelihood that money would dry up and Yudkowsky and Soares would be out of a job.
[...] The kind of work MIRI is doing and the kind of experience Yudkowsky and Soares have isnât really transferable to anything else.
$235K is not very much money [edit: in the context of the AI industry]. I made close to Nateâs salary as basically an unproductive intern at MIRI. $600K is also not much money. A Preparedness researcher at OpenAI has a starting salary of $310K â $460K plus probably another $500K in equity. As for nonprofit salaries, METRâs salary range goes up to $450K just for a âseniorâ level RE/âRS, and I think itâs reasonable for nonprofits to pay someone with 20 years of experience, who might be more like a principal RS, $600K or more.
In contrast, if Mechanize succeeds, Matthew Barnett will probably be a billionaire.
If Yudkowsky said extinction risks were low and wanted to focus on some finer aspect of alignment, e.g. ensuring that AIs respect human rights a million years from now, donors who shared their worldview would probably keep donating. Indeed, this might increase donations to MIRI because it would be closer to mainstream beliefs.
MIRIâs work seems very transferable to other risks from AI, which governments and companies both have an interest in preventing. Yudkowsky and Soares have a somewhat weird skillset and I disagree with some of their research style but itâs plausible to me they could still work productively in a mathy theoretical role in either capabilities or safety.
However, things I agree with
If the Mechanize co-founders wanted to focus on safety rather than capabilities, they could.
the Mechanize co-founders decided to start the company after forming their views on AI safety.
The Yudkowsky/âSoares/âMIRI argument about AI alignment is specifically that an AGIâs goals and motivations are highly likely to be completely alien from human goals and motivations in a way thatâs highly existentially dangerous.
Is there a formula for the pledge somewhere? I couldnât find one.
See the gpt-5 report. âWorking lower boundâ is maybe too strong; maybe itâs more accurate to describe it as an initial guess at a warning threshold for rogue replication and 10x uplift (if we can even measure time horizons that long). I donât know what the exact reasoning behind 40 hours was, but one fact is that humans canât really start viable companies using plans that only take a ~week of work. IMO if AIs could do the equivalent with only a 40 human hour time horizon and continuously evade detection, theyâd need to use their own advantages and have made up many current disadvantages relative to humans (like being bad at adversarial and multi-agent settings).
What scale is the METR benchmark on? I see a line that âScores are normalized such that 100% represents a 50% success rate on tasks requiring 8 human-expert hours.â, but is the 0% point on the scale 0 hours?
METR does not think that 8 human hours is sufficient autonomy for takeover; in fact 40 hours is our working lower bound.
What if we decide that the Amazon rainforest has a negative WAW sign? Would you be in favor of completely replacing it with a parking lot, if doing so could be done without undue suffering of the animals that already exist there?
Definitely not completely replacing because biodiversity has diminishing returns to land. If we pave the whole Amazon weâll probably extinct entire families (not to mention we probably cause ecological crises elsewhere and disrupt ecosystem services etc), whereas on the margin weâll only extinct species endemic to the deforested regions.
If the research on WAW comes out super negative I could imagine it being OK to replace half the Amazon with higher-welfare ecosystems now, and work on replacing the rest when some crazy AI tech allows all changes to be fully reversible. But the moral parliament would probably still not be happy about this. Eg killing is probably bad, and there is no feasible way to destroy half the Amazon in the near term without killing most of the animals in it.
Itâs plausible to me that biodiversity is valuable, but with AGI on the horizon it seems a lot cheaper in expectation to do more out-there interventions, like influencing AI companies to care about biodiversity (alongside wild animal welfare), recording the DNA of undiscovered rainforest species about to go extinct, and buying the cheapest land possible (middle of Siberia or Australian desert, not productive farmland). Then when the technology is available in a few decades and weâre better at constructing stable ecosystems de novo, we can terraform the deserts into highly biodiverse nature preserves. Another advantage of this is that weâll know more about animal welfareâas it stands now the sign of habitat preservation is pretty unclear.
Nukit ships to many countries.
Thanks for the reply.
Everyone has different emotional reactions so I would be wary about generalizing here. Of the vegetarians I know, certainly not all are disgusted by meat. Disgust is often more correlated with whether they use a purity frame of morality or experience disgust in general than how much they empathize with animals [1]. Empathy is not an end; itâs not required for virtuous action, and many people have utilitarian, justice-centered, or other frames that can prescribe actions with empathy taking a lesser role. As for me, I feel that after experiencing heightened empathy for those 40 days in 2021 and occasionally since, I understand its psychological effects on me well enough to know Iâm not making a grave moral error.
I would only feel averse to eating human meat if the human were murdered just for people to eat, and wouldnât feel much disgust unless it still looked like human body parts, so maybe Iâm an exception. But Iâm not sure how this is relevant.
Agree with the social signal purpose being different. I guess which one is better would depend on the social group. Around my friends who are either omnivores or vegan, I feel ok just signaling itâs bad to eat the worst treated animals. But if everyone else avoided chicken and seemed to think eating everything else were fine, I would give up something else for signaling purposes, and maybe at some point itâs better just to go vegan.
[1] Or just whether they grew up vegetarian, like how people are often disgusted by any strange food
Didnât realize my only post of the year was from April 1st. Longforms are just so scary to write other than on April Foolâs Day!
Are you interested in betting on these beliefs? I couldnât find a bet with Vasco but it seems more likely we could find one, because it seems like youâre more confident
Youâre shooting the messenger. Iâm not advocating for downvoting posts that smell of âthe outgroupâ, just saying that this happens in most communities that are centered around an ideological or even methodological framework. Itâs a way you can be downvoted while still being correct, especially from the LEAST thoughtful 25% of EA forum voters
Please read the quote from Claude more carefully. MacAskill is not an âanti-utilitarianâ who thinks consequentialism is âfundamentally misguidedâ, heâs the moral uncertainty guy. The moral parliament usually recommends actions similar to consequentialism with side constraints in practice.
I probably wonât engage more with this conversation.
Claude thinks possible outgroups include the following, which is similar to what I had in mind
Based on the EA Forumâs general orientation, here are five individuals/âgroups whose characteristic opinions would likely face downvotes:
Effective accelerationists (e/âacc) - Advocates for rapid AI development with minimal safety precautions, viewing existential risk concerns as overblown or counterproductive
TESCREAL critics (like Emile Torres, as you mentioned) - Scholars who frame longtermism/âEA as ideologically dangerous, often linking it to eugenics, colonialism, or techno-utopianism
Anti-utilitarian philosophersâStrong deontologists or virtue ethicists who reject consequentialist frameworks as fundamentally misguided, particularly on issues like population ethics or AI risk trade-offs
Degrowth/âanti-progress advocatesâThose who argue economic/âtechnological growth is net-negative and should be reduced, contrary to EAâs generally pro-progress orientation
Left-accelerationists and systemic change advocatesâCritics who view EA as a âneoliberalâ distraction from necessary revolutionary change, or who see philanthropic approaches as fundamentally illegitimate compared to state redistribution
My main concern is that the arrival of AGI completely changes the situation in some unexpected way.
e.g. in the recent 80k podcast on fertility, Rob Wiblin opines that the fertility crash would be a global priority if not for AI likely replacing human labor soon and obviating the need for countries to have large human populations. There could be other effects.
My guess is that due to advanced AI, both artificial wombs and immortality will be technically feasible in the next 40 years, as well as other crazy healthcare tech. This is not an uncommon view
Before anything like a Delphi forecast it seems better to informally interview a couple of experts, and then write your own quick report on what the technical barriers are to artificial wombs. This way you can incorporate this into the structure of any forecasting exercise, e.g. by asking experts to forecast when each of hurdles X, Y, and Z will be solved, whereupon you can do things like identifying where the level of agreement is highest and lowest, as well as consistency checks against the overall forecast.
Most infant mortality still happens in the developing world, due to much more basic factors like tropical diseases. So if the goal is reducing infant mortality globally, you wonât be addressing most of the problem, and for maternal mortality, the tech will need to be so mature that itâs affordable for the average person in low-income countries, as well as culturally accepted.
Yeah, while I think truth-seeking is a real thing I agree itâs often hard to judge in practice and vulnerable to being a weasel word.
Basically I have two concerns with deferring to experts. First is that when the world lacks people with true subject matter expertise, whoever has the most prestigeâmaybe not CEOs but certainly mainstream researchers on slightly related questionsâwill be seen as experts and we will need to worry about deferring to them.
Second, because EA topics are selected for being too weird/âunpopular to attract mainstream attention/âfunding, I think a common pattern is that of the best interventions, some are already funded, some are recommended by mainstream experts and remain underfunded, and some are too weird for the mainstream. Itâs not really possible to find the âtoo weirdâ kind without forming an inside view. We can start out deferring to experts, but by the time weâve spent enough resources investigating the question that youâre at all confident in what to do, the deferral to experts is partially replaced with understanding the research yourself as well as the load-bearing assumptions and biases of the experts. The mainstream experts will always get some weight, but it diminishes as your views start to incorporate their models rather than their views (example that comes to mind is economists on whether AGI will create explosive growth, and how recently good economic models have been developed by EA sources, now including some economists that vary assumptions and justify differences from the mainstream economistsâ assumptions).
Wish I could give more concrete examples but Iâm a bit swamped at work right now.
Not âeveryone agreesâ what âutilitarianismâ means either and it remains a useful word. In context you can tell I mean someone whose attitude, methods and incentives allow them to avoid the biases I listed and others.
I think the âmost topicsâ thing is ambiguous. There are some topics on which mainstream experts tend to be correct and some on which theyâre wrong, and although expertise is valuable on topics experts think about, they might be wrong on most topics central to EA. [1] Do we really wish we deferred to the CEO of PETA on what animal welfare interventions are best? EAs built that field in the last 15 years far beyond what âexpertsâ knew before.
In the real world, assuming we have more than five minutes to think about a question, we shouldnât âdeferâ to experts or immediately âembrace contrarian viewsâ, rather use their expertise and reject it when appropriate. Since this wasnât an option in the poll, my guess is many respondents just wrote how much they like being contrarian, and EAs have to often be contrarian on topics they think about so it came out in favor of contrarianism.
[1] Experts can be wrong because they donât think in probabilities, they have a lack of imagination, there are obvious political incentives to say one thing over another, and probably other reasons, and lots of the central EA questions donât have actual well-developed scientific fields around them, so many of the âexpertsâ arenât people who have thought about similar questions in a truth-seeking way for many years
I think this is a significant reason why people downvote some, but not all, things they disagree with. Especially a member of the outgroup who makes arguments EAs have refuted before and need to reexplain, not saying itâs actually you
Can you explain what you mean by âcontextualizing moreâ? (What a curiously recursive question...)
I mean it in this sense; making people think youâre not part of the outgroup and donât have objectionable beliefs related to the ones you actually hold, in whatever way is sensible and honest.
Maybe LW is better at using disagreement button as I find itâs pretty common for unpopular opinions to get lots of upvotes and disagree votes. One could use the API to see if the correlations are different there.
On a global scale I agree. My point is more that due to the salary standards in the industry, Eliezer isnât necessarily out of line in drawing $600k, and itâs probably not much more than he could earn elsewhere; therefore the financial incentive is fairly weak compared to that of Mechanize or other AI capabilities companies.