Scott Alexander

• Scott Alexander 11 Apr 2026 16:47 UTC

  37 points

  7 ∶ 0

  on: What it’s like to be an AI safety grant­maker (and why we need more of them)

  Thanks. Can you explain what the bottleneck is to having there be more AI grantmakers? It seems like there are always many bright young people who want to work for EA charities - what prevents the charities that need more grantmakers from hiring some of them?

Alter­na­tive AI Fore­cast­ing Methods

• Scott Alexander 1 Jan 2025 9:09 UTC

  15 points

  1 ∶ 1

  in reply to: MichaelDickens’s comment on: Where I Am Donat­ing in 2024

  2. I agree I’m assuming there will be a slow takeoff (operationalized as let’s say a ~one year period where GPT-integer-increment-level-changes happen on a scale of months, before any such period where they happen on a scale of days).

  3. AI companies being open to persuasion seems kind of trivial to me. They already have alignment teams. They already (I assume) have budget meetings where they discuss how many resources these teams should get. I’m just imagining inputs into this regular process. I agree that issues around politics could be a lesser vs. greater input.

  1. I wouldn’t frame this as alignment is easy/​hard, so much as “alignment is more refractory to 10,000 copies of GPT-6 working for a subjective century” vs. “alignment is more refractory to one genius, not working at a lab, coming up with a new paradigm using only current or slightly-above-current AIs as model organisms, in a sense where we get one roll at this per calendar year”.

• Scott Alexander 31 Dec 2024 13:05 UTC

  127 points

  21 ∶ 2

  on: Where I Am Donat­ing in 2024

  I worry there’s a negative example bias in the section about working with AI companies/​accumulating power and influence, vs. working outside the system.

  You point to cases where something bad happened, and say that some of the people complacent in the bad thing didn’t protest because they wanted to accumulate power/​influence within the system.

  But these should be matched by looking for cases where something good happened because people tried to accumulate power/​influence within a system.

  I think this is a significant percent of all good things that have ever happened. Just to give a trivial example, slavery ended because people like Abraham Lincoln successfully accumulated power within the federal government, which at the time was pro-slavery and an enforcer of slavery. If abolitionists had tried to “stay pure” by refusing to run for office, they probably would have gotten nowhere.

  Or an even clearer example: Jimmy Carter ended segregation in Georgia by pretending to be extremely racist, winning the gubernatorial election on the strength of the racist vote, then showing his true colors and ending segregation.

  (is it cheating to use government as an example? I don’t think so—you mentioned Vietnam)

  You also mention academia and say that maybe the desire of academics to “work within the system” prevents intellectual change. I would argue that any time an academic consensus changes—which historically has been pretty common—it’s been because someone worked within the system, got their PhD, and used their prestige to advocate for the new better position. If nobody who disagreed with an academic consensus ever did that, paradigms would never change, and academia would be much worse.

  (here I think a good example is wokeness—the colleges are full of people who said they decided that such-and-such a field was racist and it was their duty to change it from within. Those people won, and they’ll keep winning until people with alternate ideologies are equally dedicated)

  I also think there’s a bias in this space towards thinking that the current AI situation is maximally cursed compared to all counterfactuals. Suppose nobody who cared about alignment had founded an AI company. We’d still have Moore’s Law and compute costs would still go down. Using modern chips, it costs $20 to train a GPT-2 equivalent (this might be slightly conditioning on chip or algorithmic progress spurred by OpenAI, but I think it’s a useful comparison point). If OpenAI hadn’t done it, eventually someone else would have. So maybe in this world, since OpenAI/​Anthropic/​DeepMind don’t exist, the top AI companies are Google (not Deepmind), Meta, and Baidu, they’re 1-5 years of algorithmic progress and getting-scaling-running behind where they are now, and they all have the Yann LeCunn approach to alignment (or in Baidu’s case have never even heard the term). Is subtracting 1-5 years from timelines in exchange for making most big AI companies have alignment teams and at-least-mildly-concerned CEOs, a good trade? I can’t really say, but I don’t understand everyone else’s strong conviction that it isn’t. What would we have done with 1-5 years extra timeline? MIRI-style agent foundations research? Try to lobby politicians to pause a thing that wasn’t happening?

  (in fact, for this counterfactual to be fair, there can’t be any alignment discussion at all—if there’s alignment discussion, it inspires Sam Altman and the rest. So I think we would just let those 1-5 years pass by without using them strategically, unless we can somehow do the alignment research in secret with no public community to speak of.)

  I don’t want to argue that working within the system is definitely better—I’m on the fence, because of a combination of your considerations and the ones above. My cruxes are -

  1. What is the chance that PauseAI activism will work?

  2. If it does work, is there a plan for what to do with the pause?

  3. Does pro-pause activism now complement or substitute for pro-pause activism later? (eg mid-intelligence-explosion when the case will hopefully be more obvious)

  4. How much goodwill do we burn with AI companies per percent likelihood of an actually-useful AI pause that we gain? Are there different framings /​ forms of activism /​ target laws that would buy us better chances of a useful pause per unit of goodwill burnt?

  5. If pro-pause activism burns goodwill, how effectively can we pull off a good cop /​ bad cop strategy as opposed to having the Unilateralist’s Curse poison the whole movement?

  6. What’s the difference in p(doom) between a world where AI companies have 75th percentile vs. 25th percentile levels of goodwill towards the concept of alignment /​ friendly professional relationship with the alignment community?

  Of these, I find myself thinking most about the last. My gut feeling is that nothing we do is going to matter, and the biggest difference between good outcomes and bad outcomes is how much work the big AI labs put into alignment during the middle of the intelligence explosion when progress moves fastest. The fulcrum for human extinction might look like a meeting between Sam Altman + top OpenAI executives where they decide whether to allocate 10% vs. 20% of their GPT-6 instances to alignment research. And the fulcrum for that fulcrum might be whether the executives think “Oh yeah, alignment, that thing that all the cool Silicon Valley people whose status hierarchy we want to climb agree is really important” vs. “Oh, the ideology of the hated socialist decel outgroup who it would be social suicide to associate ourselves with”. If we get five SB 1047 style bills at the cost of shifting from the first perspective to the second, I’m not sure we’re winning here (even if those bills don’t get vetoed). And the more you think that all past EA interventions have made things worse, the more concerned you should be about this (arguably—I admit it depends how you generalize).

  Right now I lean towards trying to chart a muddy middle course, something like “support activism that seems especially efficient in getting things done per unit of AI company goodwill burnt”. But I am most optimistic about laying the groundwork for a pause campaign that might come later, in the middle of the intelligence explosion, when it will become obvious that something crazy is happening, and when Sam Altman will have all those spare GPT-6 instances which—if paused from doing capabilities research—can be turned to alignment.

  What links here?

  • If EAs ex­isted 100 years ago, would they sup­port Pro­hi­bi­tion? by WillieG (1 Feb 2025 18:49 UTC; 66 points)

• Scott Alexander 29 Dec 2024 0:46 UTC

  11 points

  3 ∶ 0

  on: Mo­ral­ity Is Still Demanding

  “To redeem their version of morality from the demangingness objection, the tweeters assert that some good deeds are supererogatory, which is philosophy for “nice to do, but not obligatory.” The problem is that they do not present a reason why doing more good would ever be supererogatory, other than the implicit convenience of ducking the demandingness objection.”

  I think this might be addressed to me. My reasoning is at https://​​slatestarcodex.com/​​2017/​​08/​​28/​​contra-askell-on-moral-offsets/​​ . Other than that, I’m not sure how you get a coherent theory of obligatory vs. supererogatory.

  What would it mean for a thing which nobody does (donate literally all their money beyond minimum required to live) to be obligatory? I think of “obligatory” as meaning that if someone doesn’t do a thing, then we all agree to consider them a bad person. But we can’t make that agreement regarding donating 100% of income beyond survival level, because we’d never stick to it—I think of the EAs who donate 50% of their income as extremely good people, I can’t self-modify to not do that even if I wanted to, and I don’t want to.

  Without something like that, how do you distinguish “obligatory” from simply “a good thing to do” (which I and IIUC everyone in the discussion agrees that donating more is).

  What links here?

  • Richard Y Chappell🔸's comment on Mo­ral­ity Is Still Demanding by utilistrutil (29 Dec 2024 2:04 UTC; 8 points)

• Scott Alexander 4 Dec 2024 15:31 UTC

  36 points

  10 ∶ 16

  on: What is the ad­mis­sions bar for EA Global?

  If you admit 84% of people, but also feel like many people who you would like to have are turned off by the perception of a high admissions bar, wouldn’t it make sense to admit everyone (or have a default-admit policy that you stray from only in cases of extreme poor culture fit)?

  I won’t quite say “worst case scenario is that there are an extra 16% of people there who you don’t like”, because the worst case scenario is that the marginal applicant lured in by the lack of an admissions bar is much worse than the current applicant pool, but it seems like something like that could be true (ie it doesn’t seem like there’s currently a large pool of unqualified applicants who it would overwhelm the conference to let in).

• Scott Alexander 18 Sep 2024 10:51 UTC

  5 points

  0 ∶ 0

  in reply to: trammell’s comment on: Dispel­ling the An­thropic Shadow

  Habryka referred me to https://​​forum.effectivealtruism.org/​​posts/​​A47EWTS6oBKLqxBpw/​​against-anthropic-shadow , whose “Possible Solution 2” is what I was thinking of. It looks like anthropic shadow holds if you think there are many planets (which seems true) and you are willing to accept weird things about reference classes (which seems like the price of admissions to anthropics). I appreciate the paper you linked for helping me distinguish between the claim that anthropic shadow is transparently true without weird assumptions, vs. the weaker claim in Possible Solution 2 that it might be true with about as much weirdness as all the other anthropic paradoxes.

• Scott Alexander 18 Sep 2024 10:49 UTC

  5 points

  1 ∶ 0

  in reply to: trammell’s comment on: Dispel­ling the An­thropic Shadow

  Thank you, this is helpful.

• Scott Alexander 13 Sep 2024 11:42 UTC

  12 points

  0 ∶ 0

  on: Dispel­ling the An­thropic Shadow

  I’m having trouble understanding this. The part that comes closest to making sense to me is this summary:

  The fact that life has survived so long is evidence that the rate of
  potentially omnicidal events is low...[this and the anthropic shadow effect] cancel out, so that overall the historical record provides evidence for a true rate close to the observed rate.

  Are they just applying https://​​en.wikipedia.org/​​wiki/​​Self-indication_assumption_doomsday_argument_rebuttal to anthropic shadow without using any of the relevant terms, or is it something else I can’t quite get?

  Also, how would they respond to the fine-tuning argument? That is, it seems like most planets (let’s say 99.9%) cannot support life (eg because they’re too close to their sun). It seems fantastically surprising that we find ourselves on a planet that does support life, but anthropics provides an easy way out of this apparent coincidence. That is, anthropics tells us that we overestimate the frequency of things that allow us to be alive. This seems like reverse anthropic shadow, where anthropic shadow is underestimating the frequency of things that cause us to be dead. So is the paper claiming that anthropics does change our estimates of the frequency of good things, but can’t change our estimate of the frequency of bad things? Why would this be?

• Scott Alexander 23 Aug 2024 10:53 UTC

  17 points

  6 ∶ 1

  in reply to: tlevin’s comment on: Re­sults of an in­for­mal sur­vey on AI grantmaking

  I mostly agree with this. The counterargument I can come up with is that the best AI think tanks right now are asking for grants in the range of $2 - $5 million and seem to be pretty influential, so it’s possible that a grantmaker who got $8 million could improve policy by 5%, in which case it’s correct to equate those two.

  I’m not sure how that fits with the relative technical/​policy questions.

• Scott Alexander 23 Aug 2024 10:51 UTC

  2 points

  0 ∶ 0

  in reply to: David_Moss’s comment on: Re­sults of an in­for­mal sur­vey on AI grantmaking

  Yes, I added them partway through after thinking about the question set more.

• Scott Alexander 21 Aug 2024 22:55 UTC

  2 points

  0 ∶ 0

  in reply to: jacobpfau’s comment on: Re­sults of an in­for­mal sur­vey on AI grantmaking

  Sorry, I’ve fixed the Google Form.

Re­sults of an in­for­mal sur­vey on AI grantmaking

• Scott Alexander 20 Jun 2024 6:10 UTC

  16 points

  5 ∶ 2

  on: Against the Guardian’s hit piece on Man­i­fest

  The article was obviously terrible, and I hope the listed mistakes get corrected, but I haven’t seen a request for correction on the claim that CFAR/​Lightcone has $5 million of FTX money and isn’t giving it back. Is there any more information on whether this is true and, if so, what their reasoning is?

• Scott Alexander 22 May 2024 12:07 UTC

  61 points

  14 ∶ 9

  on: A tale of two Sams

  I think this is more over-learning and institutional scar tissue from FTX. The world isn’t divided into Bad Actors and Non-Bad-Actors such that the Bad Actors are toxic and will destroy everything they touch.

  There’s increasing evidence that Sam Altman is a cut-throat businessman who engages in shady practices. This also describes, for example, Bill Gates and Elon Musk, both of whom also have other good qualities. I wouldn’t trust either of them to single-handedly determine the fate of the world, but they both seem like people who can be worked with in the normal paradigm of different interests making deals with each other while appreciating a risk of backstabbing.

  I think “Sam Altman does shady business practices, therefore all AI companies are bad actors and alignment is impossible” is a wild leap. We’re still in the early (maybe early middle) stages of whatever is going to happen. I don’t think this is the time to pick winners and put all eggs in a single strategy. Besides, what’s the alternative? Policy? Do you think politicians aren’t shady cut-throat bad actors? That the other activists we would have to work alongside aren’t? Every strategy involves shifting semi-coalitions with shady cut-throat bad actors of some sort of another, you just try to do a good job navigating them and keep your own integrity intact.

  If your point is “don’t trust Sam Altman absolutely to pursue our interests above his own”, point taken. But there are vast gulfs between “don’t trust him absolutely” and “abandon all strategies that come into contact with him in any way”. I think the middle ground here is to treat him approximately how I think most people here treat Elon Musk. He’s a brilliant but cut-throat businessman who does lots of shady practices. He seems to genuinely have some kind of positive vision for the world, or want for PR reasons to seem like he has a positive vision for the world, or have a mental makeup incapable of distinguishing those two things. He’s willing to throw the AI safety community the occasional bone when it doesn’t interfere with business too much. We don’t turn ourselves into the We Hate Elon Musk movement or avoid ever working with tech companies because they contain people like Elon Musk. We distance ourselves from him enough that his PR problems aren’t our PR problems (already done in Sam’s case; thanks to the board the average person probably thinks of us as weird anti-Sam-Altman fanatics) describe his positive and negative qualities honestly if asked, try to vaguely get him to take whatever good advice we have that doesn’t conflict with his business too much, and continue having a diverse portfolio of strategies at any given time. Or, I mean, part of the shifting semi-coalitions is that if some great opportunity to get rid of him comes, we compare him to the alternatives and maybe take it. But we’re so far away from having that alternative that pining after it is a distraction from the real world.

Ex­cerpts From The EA Talmud

Pause For Thought: The AI Pause Debate

• Scott Alexander 1 Sep 2023 2:35 UTC

  2 points

  0 ∶ 0

  in reply to: Bentham's Bulldog’s comment on: Eliezer Yud­kowsky Is Fre­quently, Con­fi­dently, Egre­giously Wrong

  I thought we already agreed the demon case showed that FDT wins in real life, since FDT agents will consistently end up with more utility than other agents.

  Eliezer’s argument is that you can become the kind of entity that is programmed to do X, by choosing to do X. This is in some ways a claim about demons (they are good enough to predict even the choices you made with “your free will”). But it sounds like we’re in fact positing that demons are that good—I don’t know how to explain how they have 999,999/​million success rate otherwise—so I think he is right.

  I don’t think the demon being wrong one in a million times changes much. 999,999 of the people created by the demon will be some kind of FDT decision theorist with great precommitment skills. If you’re the one who isn’t, you can observe that you’re the demon’s rare mistake and avoid cutting off your legs, but this just means you won the lottery—it’s not a generally winning strategy.

  Decision theories are intended as theories of what is rational for you to do. So it describes what choices are wise and which choices are foolish.

  I don’t understand why you think that the choices that get you more utility with no drawbacks are foolish, and the choices that cost you utility for no reason are wise.

  On the Newcomb’s Problem post, Eliezer explicitly said that he doesn’t care why other people are doing decision theory, he would like to figure out a way to get more utility. Then he did that. I think if you disagree with his goal, you should be arguing “decision theory should be about looking good, not about getting utility” (so we can all laugh at you) rather than saying “Eliezer is confidently and egregiously wrong” and hiding the fact that one of your main arguments is that he said we should try to get utility instead of failing all the time and then came up with a strategy that successfully does that.

• Scott Alexander 30 Aug 2023 3:24 UTC

  6 points

  4 ∶ 1

  in reply to: Bentham's Bulldog’s comment on: Eliezer Yud­kowsky Is Fre­quently, Con­fi­dently, Egre­giously Wrong

  I think rather than say that Eliezer is wrong about decision theory, you should say that Eliezer’s goal is to come up with a decision theory that helps him get utility, and your goal is something else, and you have both come up with very nice decision theories for achieving your goal.

  (what is your goal?)

  My opinion on your response to the demon question is “The demon would never create you in the first place, so who cares what you think?” That is, I think your formulation of the problem includes a paradox—we assume the demon is always right, but also, that you’re in a perfect position to betray it and it can’t stop you. What would actually happen is the demon would create a bunch of people with amputation fetishes, plus me and Eliezer who it knows wouldn’t betray it, and it would never put you in the position of getting to make the choice in real life (as opposed to in an FDT algorithmic way) in the first place. The reason you find the demon example more compelling than the Newcomb example is that it starts by making an assumption that undermines the whole problem—that is, that the demon has failed its omniscience check and created you who are destined to betray it. If your problem setup contains an implicit contradiction, you can prove anything.

  I don’t think this is as degenerate a case as “a demon will torture everyone who believes FDT”. If that were true, and I expected to encounter that demon, I would simply try not to believe FDT (insofar as I can voluntarily change my beliefs). While you can always be screwed over by weird demons, I think decision theory is about what to choose in cases where you have all of the available knowledge and also a choice in the matter, and I think the leg demon fits that situation.

• Scott Alexander 29 Aug 2023 18:22 UTC

  2 points

  0 ∶ 1

  in reply to: Bentham's Bulldog’s comment on: Eliezer Yud­kowsky Is Fre­quently, Con­fi­dently, Egre­giously Wrong

  I guess any omniscient demon reading this to assess my ability to precommit will have learned I can’t even precommit effectively to not having long back-and-forth discussions, let alone cutting my legs off. But I’m still interested in where you’re coming from here since I don’t think I’ve heard your exact position before.

  Have you read https://​​www.lesswrong.com/​​posts/​​6ddcsdA2c2XpNpE5x/​​newcomb-s-problem-and-regret-of-rationality ? Do you agree that this is our crux?

  Would you endorse the statement “Eliezer, using his decision theory, will usually end out with more utility than me over a long life of encountering the sorts of weird demonic situations decision theorists analyze, I just think he is less formally-rational” ?

  Or do you expect that you will, over the long run, get more utility than him?