David_Althaus

• David_Althaus 19 Feb 2026 18:46 UTC

  2 points

  0 ∶ 0

  in reply to: Ebenezer Dukakis’s comment on: Long-term risks from ide­olog­i­cal fanaticism

  Interesting points!
  
  (I think the US example is perhaps a bit more complicated. It’s not just very wealthy, it’s also highly unequal and offers much weaker safety nets than most other liberal democracies. So the bitter politics may have more to do with material insecurity than with post-scarcity boredom.)
  
  That said, I do agree that as scarcity recedes, zero-sum status games could become more prevalent.
  
  Another reason why fanaticism could matter more in the long run is that future disagreements may be much more about terminal value differences than instrumental policy questions like how to create jobs or make things more affordable (no one will need jobs and there could be huge abundance). That’s where fanaticism becomes especially relevant because it entails potentially drastic value disagreements that are locked in, with potentially no room for change, trade, or compromise.

• David_Althaus 19 Feb 2026 18:24 UTC

  2 points

  0 ∶ 0

  in reply to: David Mathers🔸’s comment on: Long-term risks from ide­olog­i­cal fanaticism

  I’m mostly warning about complacency about liberals being safe from error

  I can certainly agree with that. :)

  I don’t really understand why liberalism is getting the prefix “classical” here though.

  Mostly to reduce the chance of misinterpretation. In the US, “liberal” is often used interchangeably with something like “leftist”, “Democrat”, or “progressive”, and I wanted to make clear that I don’t want any of these connotations. I also wanted to emphasize the core principles of liberalism, and avoid getting bogged down in specific policy debates.

• David_Althaus 17 Feb 2026 19:38 UTC

  8 points

  1 ∶ 0

  in reply to: wren’s comment on: Long-term risks from ide­olog­i­cal fanaticism

  Thanks for the comment!
  
  On the No True Scotsman concern
  
  Fair point. But I think there’s a genuine structural asymmetry here. When liberal democracies commit atrocities, they do so by violating their own safeguards—secrecy, executive overreach, circumventing checks and balances. The CIA’s Cold War operations required hiding what they were doing from Congress and the public, precisely because the actions were incompatible with the system’s principles. And liberal democracies contain built-in self-correcting mechanisms: free press, independent courts, elections, public accountability. The US eventually declassified the documents and the atrocities became part of the historical record that we can openly discuss and condemn. This self-correction is a core feature of classical liberalism.
  
  The “not real communism” defense has the opposite problem. Concentrating all power in a vanguard party, suppressing class enemies, and eliminating institutional checks aren’t deviations from Marxism-Leninism, they’re core features. Once you’ve done all that, totalitarian horror isn’t a failure of implementation but a foreseeable consequence of the design. (Also, the CCP is still putting Mao on all their banknotes.)
  
  ”Time-tested” doesn’t mean “never fails”, it means better outcomes on average, less catastrophic failures, and mechanisms to recognize and correct its own failures. The right question isn’t “does liberalism guarantee safety?” (nothing does), it’s “which system produces the best outcomes and has the strongest safeguards?” The historical record is pretty clear.
  
  On the Nazis exploiting Weimar democracy
  
  The argument seems to be essentially: “Nazis rose to power in a liberal society, therefore liberalism enabled Nazism.” But this arguably confuses background conditions with causation. The Nazis also exploited elections, but most people still seem quite partial to them.
  
  The actual causal story involves the Treaty of Versailles, hyperinflation, the Great Depression, Weimar’s specific constitutional weaknesses, the mutual radicalization spiral between communists and Nazis, and the political establishment’s catastrophic miscalculation in thinking they could “control” Hitler. Free speech was a minor ingredient, if that.
  
  The fact that safeguards sometimes fail doesn’t mean the safeguards are the problem. Some people die in car crashes even while wearing seatbelts.
  
  The wide classical liberalism bucket
  
  I agree that not all values in the admittedly wide “classical liberal bucket” are equally anti-fanatical. For what it’s worth, I’m quite concerned about extreme wealth inequality, partly because it enables potential oligarchs to subvert the very system of liberal democracy. But the core claim is about the procedural principles—separation of powers, rule of law, universal rights, institutional checks—and those seem pretty robustly anti-fanatical to me.

• David_Althaus 17 Feb 2026 18:45 UTC

  12 points

  1 ∶ 0

  in reply to: fandi-chen’s comment on: Long-term risks from ide­olog­i­cal fanaticism

  Thanks, I think this narrows the disagreement productively! :)
  
  On the reframed Frankfurt School argument: I strongly agree with the claim that modern societies can retain technical rationality while losing wisdom and ethical reflection (cf. the section “differential intellectual regress”).
  
  Where I still disagree is with locating this tension inside Enlightenment reason. The decoupling of technological competence from moral reasoning isn’t something Enlightenment values produce. It’s what happens when Enlightenment values are abandoned while the technology remains. Nazi Germany didn’t gradually narrow Enlightenment reason into instrumental reason; it rejected Enlightenment values from the start and kept the trains running. It seems that the Frankfurt School framing suggests we need to be suspicious of reason itself, while the fanaticism framing suggests we need more reason, more epistemic humility, more willingness to revise beliefs—i.e., more Enlightenment values, not fewer.
  
  On whether the decoupling of technological capacity from wisdom is “accidental or structurally enabled by modern forms of organization”: I think the empirical record makes this fairly clear. Barbarism long predates modernity, antiquity and the Middle Ages were full of it. Hunter-gatherers engaged in lots of tribal warfare. In contrast, most modern liberal democracies conduct far fewer wars, have far less poverty, and produce far better outcomes across virtually every metric of human flourishing than any pre-modern society (see footnote 9 on the huge drop in violence rates). So modern institutions are clearly neither necessary nor sufficient for producing barbarism.
  
  What often leads to barbarism is the abandonment of core principles of liberal democracy like separation of powers, universal rights, and the rule of law—especially when ideologically fanatical or malevolent actors are in charge. Modernity gives you better tools, but the tools aren’t inherently the problem. That said, I agree that modernity results in great technological capacity which increases the stakes and increases the harm if bad things happen.
  
  On the “productive version” of critical theory as methodological modesty (that reason operates within institutions and incentives that can distort it) I certainly agree with that! But I’d note that many Enlightenment thinkers themselves already understood this perfectly well. Adam Smith, for instance, warned that regulatory proposals from businessmen “ought always to be listened to with great precaution… It comes from an order of men, whose interest is never exactly the same with that of the public, who have generally an interest to deceive and even oppress the public.” So you don’t necessarily need the Frankfurt School apparatus to get to “institutional incentives distort reasoning.”
  
  On Cold War violence, it sounds like we agree! :)

• David_Althaus 17 Feb 2026 18:13 UTC

  8 points

  3 ∶ 0

  in reply to: David Mathers🔸’s comment on: Long-term risks from ide­olog­i­cal fanaticism

  Communism probably also provides intellectual resources that would enable you to condemn most of the many very bad things communists have done, but that doesn’t mean that those outcomes aren’t relevant to assessing how good an idea communism is in practice.

  But much less clearly so than classical liberalism. Communism lacks any clear rejection of violence, provides no robust mechanism for resolving disagreements or conflicts, and doesn’t advocate for universal individual rights. Lenin, one of its most influential theorists, explicitly advocated for a “desperate, bloody war of extermination”.
  
  So it’s no surprise that communism almost always led to disaster and essentially never to a prosperous, flourishing society, in contrast to classical liberalism, which much more rarely led to disaster and much more often to flourishing.

• David_Althaus 16 Feb 2026 20:10 UTC

  8 points

  1 ∶ 0

  in reply to: Wei Dai’s comment on: How good would a CCP-dom­i­nated AI fu­ture be?

  Thanks for flagging this, but you’re looking at an unfiltered sample (N=2,980) which includes almost all participants regardless of data quality. All statistics in the main text use a filtered sample (N=1,084), which excludes participants who failed two attention checks, reported not answering honestly, gave invalid birth years, or strongly violated additivity (see the relevant section in the main post, including footnotes 94 and 95 for more details). The unfiltered numbers should be ignored as they clearly contain a lot of inattentive participants. (We will update the supplementary materials to label them clearly, sorry about the confusion).

  Here is the table that only includes participants who passed our inclusion criteria (the 7th table in the selected stats doc.

  

  As you can see, the numbers are much lower: 25% of Chinese respondents believe that at least 10% of people deserve unbearable pain forever.
  
  (Note: this table shows N=1,036 rather than the N=1,084 in the main text; the small discrepancy likely reflects a stricter additivity filter. I’m confirming with my co-author Clare Harris who analyzed the survey and wrote the supplementary materials.)

• David_Althaus 16 Feb 2026 19:19 UTC

  49 points

  6 ∶ 1

  in reply to: fandi-chen’s comment on: Long-term risks from ide­olog­i­cal fanaticism

  I appreciate your thoughtful comment, but I think several of the claims here don’t hold up under scrutiny.
  
  Nazism was not a product of Enlightenment rationality
  The Frankfurt School’s thesis in Dialectic of Enlightenment, that the Holocaust was somehow a product of Enlightenment reason, is one of the most influential yet poorly supported claims in 20th-century social theory. The problem is simply that Nazism was explicitly anti-Enlightenment and anti-rational.
  
  The Nazis rejected virtually every core Enlightenment principle. They replaced reason with Blut und Boden mysticism, the Führerprinzip (the principle that the leader’s intuition supersedes all evidence and deliberation), and a racial pseudoscience that bore no meaningful resemblance to the scientific method. They burned books and celebrated instinct, will, and blood over careful reasoning.
  
  The Frankfurt School argument relies on equivocating between two very different meanings of “rationality.” Yes, the Nazis used trains, bureaucracy, and industrial logistics efficiently. But efficiently using technology to implement your goals is far broader than the Enlightenment commitment to reason and evidence as guides to truth—and was already done in antiquity. We discuss a closely related issue in the essay under differential intellectual regress: fanatical regimes can maintain or even advance technological capabilities while systematically degrading wisdom, moral reflection, and reason. Scientific and technological progress requires some narrow form of rationality, and can certainly enable you to do much more harm. But that’s a point about the power of technology, not about Enlightenment values or classical liberalism.
  
  The Jakarta Method and Cold War interventions
  The 1965–66 Indonesian massacres are among the worst atrocities of the 20th century, and it’s horrible that the US intelligence community encouraged and supported them.^[1] I totally agree that the US bears substantial responsibility for this and other atrocities like the Vietnam War, supporting Pinochet’s coup against the democratically elected Allende, arming radical Islamist Mujahideen in Afghanistan (helping enable the Taliban), and orchestrating the 1953 Iranian coup. We briefly mention some of these in the essay (e.g., in the first bullet point here, Appendix F, and the spreadsheet of historical atrocities).
  
  But these atrocities were largely driven by fanatical anti-communism (especially on the US side) that divided the world into an existential struggle between good and evil, dehumanized the enemy, and held that defeating communism was so important that it justified “any means necessary”, including supporting malevolent autocrats and mass killings. They represent failures of liberal democracies to live up to their own principles, not evidence that those principles caused the violence. Classical liberalism provides the intellectual resources to condemn the Jakarta killings.
  
  On the critical theory tradition
  You suggest that ideology should be understood as something structural and embedded rather than merely a matter of overt zealotry. But this is a false dichotomy. We explicitly mention how fanatical ideologies can become normalized and mainstream (that’s partly why we chose the term “fanaticism” over “extremism”).
  
  Several of the thinkers you cite exemplify a reflexive skepticism toward liberal universalism that treats all claims to reason, evidence, and universal moral concern as merely disguised exercises of power. This is its own form of epistemic closure. If every appeal to evidence is just “instrumental rationality serving domination,” you’ve constructed an unfalsifiable framework.
  Of course, power absolutely shapes knowledge production, and Western intellectual traditions have real blind spots. But there’s a crucial difference between “we should be attentive to how power can distort reasoning” (true) and “Enlightenment rationality is always a tool of domination”; an extraordinary claim that, taken seriously, would undermine the very epistemic tools needed to identify and correct injustice, including the injustices you rightly highlighted.

  1. ^

     The Act of Killing, in which some of the perpetrators cheerfully reenact their murders decades later, is one of the most horrifying and interesting documentaries I’ve ever watched.

• David_Althaus 14 Feb 2026 19:58 UTC

  9 points

  3 ∶ 0

  in reply to: MichaelDickens’s comment on: Long-term risks from ide­olog­i­cal fanaticism

  Thanks for the comment, Michael.
  
  On wild animal suffering
  
  You raise a good point, and I do think WAS persisting into the long-term future is a serious concern. That said, I think the distinction between incidental and intentional suffering is absolutely crucial from a longtermist perspective.
  
  Agents who value ecosystems or nature aesthetically don’t have “create suffering” as a terminal value. The suffering is a byproduct—one they might be open to eliminate if they could do so without destroying what they actually care about. That makes this amenable to Pareto improvements: keep the ecology, remove the suffering. It’s at least conceivable that those who value ecosystems would be open to interventions that reduce suffering in nature—though they’d probably dislike doing so via advanced technology like nanobots. (Though they might be open to more “natural” interventions, but more on that in a moment.)
  
  It’s also worth noting that WAS at its current Earthly scale isn’t an s-risk (by definition s-risks entail vastly more suffering than currently exists on Earth). For it to become one, you’d need agents who actively spread it to other star systems and insist that all the animals keep suffering, and refuse any intervention. At that point, you’re arguably describing something that could be called “ecological fanaticism”: dogmatic certainty, a simplistic nature-good/​intervention-evil dichotomy, and willingness to perpetuate vast suffering in service of that ideology. Admittedly, this is a bit of a definitional stretch but it’s at least in the neighborhood.
  
  As an aside, I think it’s worth noting that a lot of people already care about reducing wild animal suffering in certain ways. Videos of people rescuing wild animals—dogs from drowning, deer stuck on ice—get millions of views and enthusiastic responses. There seems to be broad latent demand for reducing animal suffering when it’s made salient. The vast majority of wild animal suffering persists not because people terminally value it, but because we lack the resources and technology to do much about it right now. That will change with ASI.
  
  What’s more, fanatics will resist compromise and moral trade. Someone who likes nature and has a vague preference to keep it untouched, but isn’t fanatically locked into this, would presumably allow you to eliminate the suffering if you offered enough resources in return (and you do it in a way that doesn’t offend their sensibilities—superintelligent agents might come up with ways of doing that). It’s plausible that altruistic agents will own at least some non-trivial fraction of the cosmic endowment and would be happy to spend it on exactly such trades. Fanatical agents, by contrast, won’t trade or compromise.
  
  Where I think the concern about fanaticism becomes most acute is with agents who believe that deliberately creating suffering is morally desirable—e.g., extreme retributivist attitudes wanting to inflict extreme eternal torment. If people with such values have access to ASI, the resulting suffering could dwarf WAS by orders of magnitude, especially factoring in intensity. That’s the type of scenario we’re trying to draw attention to.
  
  On the atrocity table and intentional deaths
  I also received a somewhat similar concern via DM: filtering for intentional deaths and then finding fanaticism is circular reasoning. I don’t think it is because intentional ≠ ideologically fanatical. You can have intentional mass killing driven by strategic interest, resource extraction, personal megalomania, etc. (And the table does indeed include 2 non-fanatical examples). The finding is that among the worst intentional mass killings, most involved ideological fanaticism. This is a substantive empirical result, not a tautology.
  
  Including famines wouldn’t even change the picture that much. You’d add the British colonial famines, the Chinese famine of 1907, and Mao’s Great Leap Forward (though the Great Leap Forward was itself clearly driven by fanatical ideological zealotry, and certain ideologies—colonialism, laissez-faire ideology, etc.—probably also substantially contributed to the British famines in India).
  
  More importantly, once you start including famines, why not also include pandemics? And once you include pandemics, why not deaths from disease more generally—cancer, heart disease, etc.? And why not include deaths from aging then? Obviously, the vast majority of deaths since 1800 were not due to fanaticism; most were from hunger, disease, and aging.
  
  But with sufficiently advanced technology, you won’t have deaths from disease, hunger, or aging. These deaths don’t reveal anything about terminal preferences. Intentional deaths do. That’s why, from a longtermist perspective, focusing on intentional deaths isn’t cherry-picking, it’s studying the thing that actually matters for predicting what the long-term future looks like.
  
  ASI will give agents enormous control over the universe, so the future will be shaped primarily by the terminal values of whoever controls that technology. Unintentional mass death from incompetence or nature (like aging) is terrible, but solvable.
  
  Last, I worry that we’re getting too hung up on the atrocity table. Even in a world where ideological fanaticism had resulted in only a few historical atrocities, I’d still be concerned about it as a long-term risk. The table is just one outside-view /​ historical argument among several for why we should take fanaticism seriously. The core reasons for worrying about fanaticism are mostly discussed in these sections.

• David_Althaus 13 Feb 2026 20:29 UTC

  8 points

  2 ∶ 0

  in reply to: Wei Dai’s comment on: How good would a CCP-dom­i­nated AI fu­ture be?

  I want to point out that the ethical schools of thought that you’re (probably) most anti-aligned with (e.g., that certain behaviors and even thoughts are deserving of eternal divine punishment) are also far more prominent in the West, proportionately even more so than the ones you’re aligned with.

  
  For what it’s worth, we recently ran a cross-cultural survey (n > 1,000 after extensive filtering) on endorsement of eternal extreme punishment, with questions like “If I could create a system that makes deserving people feel unbearable pain forever, I would” and “If hell didn’t exist, or if it stopped existing, we should create it [...]”.
  
  ~16-19% of Chinese respondents consistently endorsed such statements, compared to ~10–14% of US respondents—despite China being majority atheist/​agnostic.^[1]

  Of course, online surveys are notoriously unreliable, especially on such abstract questions. But if these results hold up, concerns about eternal punishment would actually count against a China-dominated future, not in favor of one.

  1. ^

     On individual questions, agreement rates were usually much higher, especially in China and other non-Western countries. The above numbers reflect a conservative conjunctive measure filtering for consistency across multiple questions.

Long-term risks from ide­olog­i­cal fanaticism

• David_Althaus 22 Sep 2025 16:05 UTC

  15 points

  3 ∶ 0

  in reply to: Luke Moore 🔸’s comment on: Up­dates on the effec­tive giv­ing ecosys­tem (MCF 2025 memo)

  Hi Luke,

  Sorry, it seems that we haven’t received your email (where did you send it, btw?). As stated on our website, we planned to give around $30m this year (though by now it seems that we may end up closer to $40m). We’re currently considering our strategy and the available opportunities and may give substantially more (possibly >$100m/​year) in the coming years.

• David_Althaus 1 Aug 2025 13:30 UTC

  3 points

  1 ∶ 0

  in reply to: tommasoquerini’s comment on: Re­duc­ing long-term risks from malev­olent actors

  Hi, makes sense, thanks.
  
  I certainly don’t think investigative journalism is a panacea or tremendously effective. Definitely agree that most voters are motivated by tribalism and economic conditions.
  
  Still, even in this age, investigative journalism sometimes seems to have substantial effects. For example, the book on the Biden’s cognitive decline and cover-up made some waves and hopefully has some positive effects.
  
  More generally, I’m also not sure to what extent our age is highly unusual. It’s not like people in the 1930s and 1940s (in say, Germany) weren’t subject to echo chambers and didn’t exhibit a few “skill issues” in critical thinking.
  
  My sense is that many elections are decided by a few percent of voters (with around 20-40% on each side (the “base”) essentially being completely immovable by any argument) so even shifting a few percent of voters journalism could have an impact.
  
  I think investigative journalism may be particularly effective during more “hingey” moments, before a political party has rallied around a certain candidate and before tribalism kicks in.
  
  I agree that, say, changing the underlying structure of the information eco-system is probably more effective (e.g., changing (social media) recommendation algorithms (e.g., on Youtube, Facebook, or Twitter) to promote truth-seeking and improve the quality of general discourse). (But that’s probably almost impossible to pull off for any given individual unless you happen to be, say, Zuckerberg or Musk). Also, interventions in the AI for epistemics space may be much more scalable and high-leverage.

• David_Althaus 1 Apr 2025 14:49 UTC

  9 points

  1 ∶ 0

  in reply to: NickLaing’s comment on: An­thropic is not be­ing con­sis­tently can­did about their con­nec­tion to EA

  from Their wedding page 2023,

  Not sure if I misunderstand something but the wedding page seems from 2017? (It reads “October 21, 2017” at the top.)

• David_Althaus 6 Mar 2025 14:33 UTC

  7 points

  3 ∶ 2

  in reply to: tlevin’s comment on: levin’s Quick takes

  Very much agree.
  
  Also, some of the more neglected topics tend to be more intellectually interesting and especially appealing if you have a bit of a contrarian temperament. One can make the mistake of essentially going all out on neglectedness and mostly work on the most fringe and galaxy-brained topics imaginable.

  I’ve been there myself: I think I probably spent too much time thinking about lab universes, descriptive population ethics, etc.
  
  Perhaps it connects to a deeper “silver bullet worldview bias”: I’ve been too attracted to worldviews according to which I can have lots of impact. Very understandable given how much meaning and self-worth I derive from how much good I believe I do.

  The real world is rather messy and crowded, so elegant and neglected ideas for having impact can become incredibly appealing, promising both outsized impact and intellectual satisfaction.

• David_Althaus 15 Jan 2025 10:48 UTC

  20 points

  4 ∶ 0

  on: Vol­un­tary Salary Reduction

  Thanks for writing this! I also voluntarily reduced my salary for several years (and lived partly off my savings) and had been meaning to write about this for some time but never got around to it. It’s always been somewhat puzzling why this isn’t more common. While it probably shouldn’t become a norm for the reasons you outline, my sense is that more EAs should consider this option (though I may be underestimating how common it is already).

  I agree with all the downsides you list but I could imagine there are also other upsides to voluntary salary reduction. For example, it can signal your commitment to both your organization and to taking altruistic ideas seriously—following the logic where it leads, even when that means doing unconventional things. This might inspire others.
  
  I also worry that we might be biased to overestimate the downsides of voluntary salary reductions: Donating creates tangible satisfaction—the concrete act of giving, the tax receipt, the social recognition, etc. Taking a lower salary offers none of these psychological benefits and can even feel like a loss in status and recognition.

• David_Althaus 24 Oct 2024 9:09 UTC

  6 points

  0 ∶ 0

  in reply to: Kestrel🔸’s comment on: What is malev­olence? On the na­ture, mea­sure­ment, and dis­tri­bu­tion of dark traits

  Thanks!
  
  I haven’t engaged much with the psychodynamic literature or mostly only indirectly (as some therapy modalities like CFT or ST are quite eclectic and thus reference various psychodynamic concepts) but perhaps @Clare_Diane has. Is there any specific construct, paper/​book or test that you have in mind here?
  
  I’m not familiar with the SWAP but it looks very interesting (though Clare may know it), thanks for mentioning it! As you most likely know, there even exists a National Security Edition developed in collaboration with the US government.
  
  

What is malev­olence? On the na­ture, mea­sure­ment, and dis­tri­bu­tion of dark traits

• David_Althaus 12 Aug 2024 7:49 UTC

  6 points

  1 ∶ 0

  in reply to: David_Althaus’s comment on: David_Althaus’s Shortform

  I just realized that in this (old) 80k podcast episode^[1], Holden makes similar points and argues that aligned AI could be bad.
  
  My sense is that Holden alludes to both malevolence (“really bad values, [...] we shouldn’t assume that person is going to end up being nice”) and ideological fanaticism (“create minds that [...] stick to those beliefs and try to shape the world around those beliefs”, [...] “This is the religion I follow. This is what I believe in. [...] And I am creating an AI to help me promote that religion, not to help me question it or revise it or make it better.”).
  
  Longer quotes below (emphasis added):

  Holden: “The other part — if we do align the AI, we’re fine — I disagree with much more strongly. [...] if you just assume that you have a world of very capable AIs, that are doing exactly what humans want them to do, that’s very scary. [...]

  Certainly, there’s the fact that because of the speed at which things move, you could end up with whoever kind of leads the way on AI, or is least cautious, having a lot of power — and that could be someone really bad. And I don’t think we should assume that just because that if you had some head of state that has really bad values, I don’t think we should assume that that person is going to end up being nice after they become wealthy, or powerful, or transhuman, or mind uploaded, or whatever — I don’t think there’s really any reason to think we should assume that.

  And then I think there’s just a bunch of other things that, if things are moving fast, we could end up in a really bad state. Like, are we going to come up with decent frameworks for making sure that the digital minds are not mistreated? Are we going to come up with decent frameworks for how to ensure that as we get the ability to create whatever minds we want, we’re using that to create minds that help us seek the truth, instead of create minds that have whatever beliefs we want them to have, stick to those beliefs and try to shape the world around those beliefs? I think Carl Shulman put it as, “Are we going to have AI that makes us wiser or more powerfully insane?”

  [...] I think even if we threw out the misalignment problem, we’d have a lot of work to do — and I think a lot of these issues are actually not getting enough attention.”

  Rob Wiblin: Yeah. I think something that might be going on there is a bit of equivocation in the word “alignment.” You can imagine some people might mean by “creating an aligned AI,” it’s like an AI that goes and does what you tell it to — like a good employee or something. Whereas other people mean that it’s following the correct ideal values and behaviours, and is going to work to generate the best outcome. And these are really quite separate things, very far apart.

  Holden Karnofsky: Yeah. Well, the second one, I just don’t even know if that’s a thing. I don’t even really know what it’s supposed to do. I mean, there’s something a little bit in between, which is like, you can have an AI that you ask it to do something, and it does what you would have told it to do if you had been more informed, and if you knew everything it knows. That’s the central idea of alignment that I tend to think of, but I think that still has all the problems I’m talking about. Just some humans seriously do intend to do things that are really nasty, and seriously do not intend — in any way, even if they knew more — to make the world as nice as we would like it to be.

  And some humans really do intend and really do mean and really will want to say, you know, “Right now, I have these values” — let’s say, “This is the religion I follow. This is what I believe in. This is what I care about. And I am creating an AI to help me promote that religion, not to help me question it or revise it or make it better.” So yeah, I think that middle one does not make it safe. There might be some extreme versions, like, an AI that just figures out what’s objectively best for the world and does that or something. I’m just like, I don’t know why we would think that would even be a thing to aim for. That’s not the alignment problem that I’m interested in having solved.

  1. ^

     I’m one of those bad EAs who don’t listen to all 80k episodes as soon as they come out.

  What links here?

  • Long-term risks from ide­olog­i­cal fanaticism by David_Althaus (12 Feb 2026 23:25 UTC; 188 points)
  • Long-term risks from ide­olog­i­cal fanaticism by David Althaus (LessWrong; 12 Feb 2026 23:26 UTC; 97 points)
  • De-em­pha­sise al­ign­ment, em­pha­sise restraint by EuanMcLean (4 Feb 2025 17:43 UTC; 19 points)

• David_Althaus 10 Aug 2024 7:39 UTC

  2 points

  0 ∶ 0

  in reply to: Mike Albrecht’s comment on: David_Althaus’s Shortform

  Thanks Mike. I agree that the alliance is fortunately rather loose in the sense that most of these countries share no ideology. (In fact, some of them should arguably be ideological enemies, e.g., Islamic theocrats in Iran and Maoist communists in China).
  
  But I worry that this alliance is held together by a hatred of (or ressentiment in general) Western secular democratic principles for ideological and (geo-)political reasons. Hatred can be an extremely powerful and unifying force. (Many political/​ideological movements are arguably primarily defined, united, and motivated by what they hate, e.g., Nazism by the hatred of Jews, communism by the hatred of capitalists, racists hate other ethnicities, Democrats hate Trump and racists, Republicans hate the woke and communists, etc.)
  
  So I worry that as long as Western democracies to influence international affairs, this alliance will continue to exist. And I certainly hope that Western democracies will continue to be powerful and worry that the world (and the future) will become a worse place if not.

• David_Althaus 3 Aug 2024 11:07 UTC

  8 points

  0 ∶ 0

  in reply to: David_Althaus’s comment on: David_Althaus’s Shortform

  Another disagreement may be related to the tractability /​ how easy it is to contribute:

  For example, we mentioned above that the three ways totalitarian regimes have been brought down in the past are through war, resistance movements, and the deaths of dictators. Most of the people reading this article probably aren’t in a position to influence any of those forces (and even if they could, it would be seriously risky to do so, to say the least!).

  Most EAs may not be able to directly work on these topics but there are various options that allow you to do something indirectly:
  
  - working in (foreign) policy or politics (or working on financial reforms that make illegal money laundering harder for autocratic states like Russia (again, cf. Autocracy Inc.).
  - becoming a journalist and writing about such topics (e.g., doing investigative journalism on the corruption in autocratic regimes), generally moving the discussion towards more important topics and away from currently trendy but less important topics
  - working at think thanks that protect democratic institutions (Stephen Clare lists several)
  - working on AI governance (e.g., info sec, export controls) to reduce autocratic regimes gaining access to AI. (Again, Stephen Clare already lists this area).
  - probably several more career paths that we haven’t thought of
  
  In general, it doesn’t seem harder to have an impactful career in this area than in, say, AI risk. Depending on your background and skills, it may even be a lot easier; e.g., in order to do valuable work on AI policy, you often need to understand policy/​politics and technical fields like computer science & machine learning. Of course, the area is arguably more crowded (though AI is becoming more crowded every day).