AI safety researcher
Thomas Kwađč
Karma: 3,932
See the gpt-5 report. âWorking lower boundâ is maybe too strong; maybe itâs more accurate to describe it as an initial guess at a warning threshold for rogue replication and 10x uplift (if we can even measure time horizons that long). I donât know what the exact reasoning behind 40 hours was, but one fact is that humans canât really start viable companies using plans that only take a ~week of work. IMO if AIs could do the equivalent with only a 40 human hour time horizon and continuously evade detection, theyâd need to use their own advantages and have made up many current disadvantages relative to humans (like being bad at adversarial and multi-agent settings).
A slidÂing scale for donaÂtion percentage
What scale is the METR benchmark on? I see a line that âScores are normalized such that 100% represents a 50% success rate on tasks requiring 8 human-expert hours.â, but is the 0% point on the scale 0 hours?
METR does not think that 8 human hours is sufficient autonomy for takeover; in fact 40 hours is our working lower bound.
What if we decide that the Amazon rainforest has a negative WAW sign? Would you be in favor of completely replacing it with a parking lot, if doing so could be done without undue suffering of the animals that already exist there?
Definitely not completely replacing because biodiversity has diminishing returns to land. If we pave the whole Amazon weâll probably extinct entire families (not to mention we probably cause ecological crises elsewhere and disrupt ecosystem services etc), whereas on the margin weâll only extinct species endemic to the deforested regions.
If the research on WAW comes out super negative I could imagine it being OK to replace half the Amazon with higher-welfare ecosystems now, and work on replacing the rest when some crazy AI tech allows all changes to be fully reversible. But the moral parliament would probably still not be happy about this. Eg killing is probably bad, and there is no feasible way to destroy half the Amazon in the near term without killing most of the animals in it.
Itâs plausible to me that biodiversity is valuable, but with AGI on the horizon it seems a lot cheaper in expectation to do more out-there interventions, like influencing AI companies to care about biodiversity (alongside wild animal welfare), recording the DNA of undiscovered rainforest species about to go extinct, and buying the cheapest land possible (middle of Siberia or Australian desert, not productive farmland). Then when the technology is available in a few decades and weâre better at constructing stable ecosystems de novo, we can terraform the deserts into highly biodiverse nature preserves. Another advantage of this is that weâll know more about animal welfareâas it stands now the sign of habitat preservation is pretty unclear.
Nukit ships to many countries.
Thanks for the reply.
Everyone has different emotional reactions so I would be wary about generalizing here. Of the vegetarians I know, certainly not all are disgusted by meat. Disgust is often more correlated with whether they use a purity frame of morality or experience disgust in general than how much they empathize with animals [1]. Empathy is not an end; itâs not required for virtuous action, and many people have utilitarian, justice-centered, or other frames that can prescribe actions with empathy taking a lesser role. As for me, I feel that after experiencing heightened empathy for those 40 days in 2021 and occasionally since, I understand its psychological effects on me well enough to know Iâm not making a grave moral error.
I would only feel averse to eating human meat if the human were murdered just for people to eat, and wouldnât feel much disgust unless it still looked like human body parts, so maybe Iâm an exception. But Iâm not sure how this is relevant.
Agree with the social signal purpose being different. I guess which one is better would depend on the social group. Around my friends who are either omnivores or vegan, I feel ok just signaling itâs bad to eat the worst treated animals. But if everyone else avoided chicken and seemed to think eating everything else were fine, I would give up something else for signaling purposes, and maybe at some point itâs better just to go vegan.
[1] Or just whether they grew up vegetarian, like how people are often disgusted by any strange food
Didnât realize my only post of the year was from April 1st. Longforms are just so scary to write other than on April Foolâs Day!
Are you interested in betting on these beliefs? I couldnât find a bet with Vasco but it seems more likely we could find one, because it seems like youâre more confident
Youâre shooting the messenger. Iâm not advocating for downvoting posts that smell of âthe outgroupâ, just saying that this happens in most communities that are centered around an ideological or even methodological framework. Itâs a way you can be downvoted while still being correct, especially from the LEAST thoughtful 25% of EA forum voters
Please read the quote from Claude more carefully. MacAskill is not an âanti-utilitarianâ who thinks consequentialism is âfundamentally misguidedâ, heâs the moral uncertainty guy. The moral parliament usually recommends actions similar to consequentialism with side constraints in practice.
I probably wonât engage more with this conversation.
Claude thinks possible outgroups include the following, which is similar to what I had in mind
Based on the EA Forumâs general orientation, here are five individuals/âgroups whose characteristic opinions would likely face downvotes:
Effective accelerationists (e/âacc) - Advocates for rapid AI development with minimal safety precautions, viewing existential risk concerns as overblown or counterproductive
TESCREAL critics (like Emile Torres, as you mentioned) - Scholars who frame longtermism/âEA as ideologically dangerous, often linking it to eugenics, colonialism, or techno-utopianism
Anti-utilitarian philosophersâStrong deontologists or virtue ethicists who reject consequentialist frameworks as fundamentally misguided, particularly on issues like population ethics or AI risk trade-offs
Degrowth/âanti-progress advocatesâThose who argue economic/âtechnological growth is net-negative and should be reduced, contrary to EAâs generally pro-progress orientation
Left-accelerationists and systemic change advocatesâCritics who view EA as a âneoliberalâ distraction from necessary revolutionary change, or who see philanthropic approaches as fundamentally illegitimate compared to state redistribution
My main concern is that the arrival of AGI completely changes the situation in some unexpected way.
e.g. in the recent 80k podcast on fertility, Rob Wiblin opines that the fertility crash would be a global priority if not for AI likely replacing human labor soon and obviating the need for countries to have large human populations. There could be other effects.
My guess is that due to advanced AI, both artificial wombs and immortality will be technically feasible in the next 40 years, as well as other crazy healthcare tech. This is not an uncommon view
Before anything like a Delphi forecast it seems better to informally interview a couple of experts, and then write your own quick report on what the technical barriers are to artificial wombs. This way you can incorporate this into the structure of any forecasting exercise, e.g. by asking experts to forecast when each of hurdles X, Y, and Z will be solved, whereupon you can do things like identifying where the level of agreement is highest and lowest, as well as consistency checks against the overall forecast.
Most infant mortality still happens in the developing world, due to much more basic factors like tropical diseases. So if the goal is reducing infant mortality globally, you wonât be addressing most of the problem, and for maternal mortality, the tech will need to be so mature that itâs affordable for the average person in low-income countries, as well as culturally accepted.
Yeah, while I think truth-seeking is a real thing I agree itâs often hard to judge in practice and vulnerable to being a weasel word.
Basically I have two concerns with deferring to experts. First is that when the world lacks people with true subject matter expertise, whoever has the most prestigeâmaybe not CEOs but certainly mainstream researchers on slightly related questionsâwill be seen as experts and we will need to worry about deferring to them.
Second, because EA topics are selected for being too weird/âunpopular to attract mainstream attention/âfunding, I think a common pattern is that of the best interventions, some are already funded, some are recommended by mainstream experts and remain underfunded, and some are too weird for the mainstream. Itâs not really possible to find the âtoo weirdâ kind without forming an inside view. We can start out deferring to experts, but by the time weâve spent enough resources investigating the question that youâre at all confident in what to do, the deferral to experts is partially replaced with understanding the research yourself as well as the load-bearing assumptions and biases of the experts. The mainstream experts will always get some weight, but it diminishes as your views start to incorporate their models rather than their views (example that comes to mind is economists on whether AGI will create explosive growth, and how recently good economic models have been developed by EA sources, now including some economists that vary assumptions and justify differences from the mainstream economistsâ assumptions).
Wish I could give more concrete examples but Iâm a bit swamped at work right now.
Not âeveryone agreesâ what âutilitarianismâ means either and it remains a useful word. In context you can tell I mean someone whose attitude, methods and incentives allow them to avoid the biases I listed and others.
I think the âmost topicsâ thing is ambiguous. There are some topics on which mainstream experts tend to be correct and some on which theyâre wrong, and although expertise is valuable on topics experts think about, they might be wrong on most topics central to EA. [1] Do we really wish we deferred to the CEO of PETA on what animal welfare interventions are best? EAs built that field in the last 15 years far beyond what âexpertsâ knew before.
In the real world, assuming we have more than five minutes to think about a question, we shouldnât âdeferâ to experts or immediately âembrace contrarian viewsâ, rather use their expertise and reject it when appropriate. Since this wasnât an option in the poll, my guess is many respondents just wrote how much they like being contrarian, and EAs have to often be contrarian on topics they think about so it came out in favor of contrarianism.
[1] Experts can be wrong because they donât think in probabilities, they have a lack of imagination, there are obvious political incentives to say one thing over another, and probably other reasons, and lots of the central EA questions donât have actual well-developed scientific fields around them, so many of the âexpertsâ arenât people who have thought about similar questions in a truth-seeking way for many years
I think this is a significant reason why people downvote some, but not all, things they disagree with. Especially a member of the outgroup who makes arguments EAs have refuted before and need to reexplain, not saying itâs actually you
Can you explain what you mean by âcontextualizing moreâ? (What a curiously recursive question...)
I mean it in this sense; making people think youâre not part of the outgroup and donât have objectionable beliefs related to the ones you actually hold, in whatever way is sensible and honest.
Maybe LW is better at using disagreement button as I find itâs pretty common for unpopular opinions to get lots of upvotes and disagree votes. One could use the API to see if the correlations are different there.
IMO the real answer is that veganism is not an essential part of EA philosophy, just happens to be correlated with it due to the large number of people in animal advocacy. Most EA vegans and non-vegans think that their diet is a small portion of their impact compared to their career, and itâs not even close! Every time you spend an extra $5 finding a restaurant with a vegan option you could help 5,000 shrimp instead. Vegans have other reasons like non-consequentialist ethics, virtue signaling or self-signaling, or just a desire not to eat the actual flesh/âbody fluids of tortured animals.
If you have a similar emotional reaction to other products it seems completely valid to boycott them, although as you mention there can be significant practical burdens, both in adjusting oneâs lifestyle to avoid such products and in judging whether the claims of marginal impact are valid. Being vegan is not obligatory in my culture and neither should boycotts beâunless the marginal impact of the boycott is larger than any other life choice which is essentially never true.
I really enjoyed reading this post; thanks for writing it. I think itâs important to take space colonization seriously and shift into ânear modeâ given that, as you say, the first entity to start a Dyson Swarm has a high chance to get DSA if it isnât already decided by AGI, and itâs probably only 10-35 years away.
Is there a formula for the pledge somewhere? I couldnât find one.