I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me).
I have a website: https://mdickens.me/ Much of the content on my website gets cross-posted to the EA Forum, but I also write about some non-EA stuff over there.
I used to work as a software developer at Affirm.
Some commentary. I mostly agree with the page, but I will focus on the bits where I see room for improvement:
All 8 gates look correct to me, but they don’t all deserve equal emphasis.
Gate 1 says IPOs have lock-ups. That’s true but I basically don’t think that matters because lock-ups are very predictable: they will announce how long it is, and that’s exactly how long it will be. There’s no uncertainty. The main reason it’s relevant is that a lockup gives more time for AI valuations to fluctuate or collapse, but the text doesn’t even mention this.
Gate 5 and Gate 7 seem like they’re saying the same thing.
Gate 8 (“Bay social incentives”) seems uninteresting since it’s not a claim in the same category as the others. It’s more like a meta-level reason why people might not think about the other 7 gates.
Would be cool for the BOTEC to use distributions rather than point estimates. (Squiggle is good for this, and Squigglehub even has a built-in way to have AI generate models.) IMO distributions are a lot more informative than point estimates.
It looks like the default estimates in the BOTEC are pulled from the sources, but it’s not clear which estimates came from which sources. There should be inline citations.
“Anthropic valuation” variable should specifically be the valuation at the end of the 6-month lockup. Doesn’t matter much for a point estimate but it would increase the variance if there variable were a probability distribution.
Unclear what the “Founder pledge” variable refers to. Is it the % of pledgers’ wealth that they’ve pledged to donate? If so, the default of 80% seems really high?
“Employee committed pool” is defined in terms of dollars rather than as a % of company valuation, which seems weird. Shouldn’t it depend on the value of the equity?
This model is supposed to illustrate how a lot of people are being too optimistic, but even then, I think most of the point estimates in the model are too optimistic. Consider that e.g. the median self-reported earner-to-give only donates (IIRC) 3% of their income.
IMO “Deployment by end-2026” should use a different date. IPO 3-6 months from now plus 6 months lockup means no money will be deployed in 2026, unless Anthropic does a fast IPO + early lockup release. Even by the end of 2027, you’re talking about a 3-9 month turnaround time on lockup ending → grants being disbursed. FTX Foundation donated $190 million (pre-clawbacks) in about 6 months, which was ridiculously fast compared to a typical foundation, and that was still a pretty small % of its long-term budget (or at least, what was believed to be its long-term budget before FTX collapsed).
I would delete the OpenAI Foundation bit because (1) the model has enough parameters already and (2) I doubt OpenAI Foundation will give much money to causes that look good by EA lights.
“Grantmaker capacity multiplier” seems nonsensical as written. Shouldn’t the capacity max out at 1x? If grantmakers are a complete non-bottleneck, then the other parameters will dictate the amount disbursed; if they’re a bottleneck, then the amount disbursed will be less. There’s no way for grantmaker capacity to have a multiplier >1x.
Also this would make more sense as a dollar amount, not a multiplier. Like there’s a fixed total amount that grantmakers can reasonably disburse. You could model it in a more complicated way but IMO a simple cap is the way to do it. Or maybe don’t use this parameter at all. I think it’s probably worth including, but keep it simple.
“Field absorption ceiling” is structured more sensibly than “Grantmaker capacity multiplier”, but these two seem redundant because they’re closely related. If orgs have more capacity to expand, grantmakers can deploy money faster by giving to those orgs. If there are more grantmakers, they can create more and bigger RFPs. etc. I would include one variable or the other, but not both.
“The steelman could be too pessimistic if founders or employees treat liquidity as an urgent moral obligation” – TBH the BOTEC as written seems to me like it’s already pricing in that founders/employees will treat donations as urgent, e.g. it’s implying that Anthropic money will be disbursed faster than FTX Foundation money, which itself was disbursed at historic speed. IMO most likely reason why the model will end up underestimating is that Anthropic market cap ends up being like 10x higher than predicted.
My downward adjustments to the model aren’t even the pessimistic case. The pessimistic case* is that the AI field collapses (investor funding dries up or something) and Anthropic stock is worth $0. Base rate says there’s like a 50% chance that that will happen. Even optimistically, you should expect at least a 10–20% chance that Anthropic stockholders get nothing.
The recommendations under “How to plan if the skeptical case is live” don’t really make sense. AFAIK ~zero orgs are planning as if they’re guaranteed to get a huge pile of donations 1–2 years from now. I believe nonprofits mainly plan based on the money they already have on their books + short-term (<1 year) fundraising expectations. “How to plan if the skeptical case is live” is just “business as usual”.
That section says “Funders and field builders should prioritize grantmaker capacity”, but that’s what to do if the skeptical case is wrong, not if the skeptical case is right.
“What would update this memo?” – as with the gates, no sense of prioritization is given. IMO by far the biggest uncertainty, about which we will get more information in the future, is: What will Anthropic’s valuation be when the lockup ends? “Concrete donor vehicles” is also important evidence, but we won’t get that until probably 6-24 months later.
*this is pessimistic for donations but I would actually prefer that this happen because it would lengthen timelines. so in a way it’s the optimistic outcome
Wouldn’t this be true for any intelligent being, not just LLMs? Almost by definition, you’re disproportionately bad at diagnosing your own mistakes, because otherwise you probably would’ve stopped making them.
I can’t say I’ve ever noticed AI being disproportionately bad at understanding AI psychology. Can you say more about why you believe this?
I’d never thought of this argument, and it’s obviously correct in retrospect.
Although “trying less hard” might not quite be pointing at the right thing. Reflecting on your epistemics / course of action could still be considered a form of “trying hard”, so maybe it would be better to describe it as “rocketing in a particular pre-determined direction less hard”.
Alex Bores is the best politician in a generation.
This is an extraordinary claim without comparably extraordinary evidence!
The origins of the rumor
On May 26, tech news website The Verge published a brief article with the headline, “Did the Pope use AI to write about the dangers of AI?” The Verge’s X post (archived) promoting its article received millions of views.
It annoys me that they cite the Verge article as the “origin”, rather than your article that the Verge article was based on.
RE the first con, my guess is that people will still hate AI even if they stand to make money off it via a SWF. As a basic test, you could look at AI favorability among people who do vs. don’t have 401(k)s / retirement funds (which almost always include some investments in AI), although this test would probably be confounded by wealth and job stability, so there’s probably a better test.
Pros:
Reduces company incentive to dangerously race to build AI (also increases government incentive to do that, but governments don’t really respond to incentives in the same way)
Alleviates economic disenfranchisement due to AI displacing human workers (I’m more concerned about x-risk but this still matters more than zero)
Cons:
Right now the general public hates AI. Putting everyone in a position to make money off AI might change that, and make it harder to get the safety regulations we actually need (this argument sounds right to me, but it also sounds suspiciously like accelerationism)
Something about government overreach stifling capitalism (this is a real concern but it’s dwarfed by x-risk concerns)
Neither-pro-nor-con:
Has basically nothing to do with x-risk
I don’t think it’s currently true that Anthropic is the cultural center of EA, but if it ends up driving most of EA money, then it could become so. On my view that’s plausibly quite bad because typical Anthropic views about AI are far too reckless and too likely to get us all killed.
New title is better!
After making sure we don’t all die, this should be the first priority
Please do not write clickbait titles like this. Put the subject matter in the title.
But I think expecting returns something like 50-100% p.a. would be reasonable, based on historic performance of these funds.
You can’t have it both ways. Either you can extrapolate from historical performance, in which case you should use long-run market returns of ~5%. Or you have a specific view on what’s going to outperform, in which case it’s a question of what your expectations are. You can’t just take a fund that performed extremely well over the last 3–5 years and then say that level of performance will continue. If some thesis did really well in the past, then everyone else can also see that it did well, and you should assume it’s now priced in unless you have some marginal thesis for why it’s not priced in.
In fact, in the long run, stocks have exhibited 3–5 year reversals—that is, stocks that perform particularly well over the last 3–5 years tend to underperform the market going forward.[1] Mutual funds haven’t exhibited reversals, but funds that outperform tend to regress to average performance.[2] At minimum, your thesis needs to have some explanation of why that’s not going to happen this time.
I’m not saying you’re wrong. AI stocks may well be a good investment. But I am pretty unimpressed by most arguments I’ve seen. “The last 5 years saw 50–100% return, therefore that same level of return will continue” is not a good argument—by (weak) default, it’s evidence that future returns will be lower than the market, not higher. (I wrote more on my thoughts in a recent post, especially this section.)
[1] Fama, E. F., & French, K. R. (1988). Permanent and Temporary Components of Stock Prices. Journal of Political Economy, 96(2), 246–273. http://www.jstor.org/stable/1833108
[2] Berk, J. B., & Green, R. C. (2004). Mutual Fund Flows and Performance in Rational Markets. Journal of Political Economy, 112(6), 1269–1295. https://doi.org/10.1086/424739
How would you model the expected investment rate of return? I generally assume a 5% real return, which is small enough that it wouldn’t affect the model in OP very much. If want to incorporate a large expected return on AI investments, then I don’t know how to do that anymore.
I basically agree, although “dangerous approaches are tamped down” is doing most of the work here IMO. By default (i.e. no tamping-down), I expect the situation with a weakly-superhuman Scientist AI to be:
a small number of sane people ask “are we doomed if we go with the following alignment strategy”, and when it says yes, they don’t do it
a lot of people don’t bother to ask at all, they just ask the Scientist AI how to build ASI
a lot of people say “we have to build ASI before the reckless people in group 2”, they build ASI using their best-guess alignment strategy that has an 88.5% chance of failing, and we die with 88.5% probability
(I think Bengio would agree that this is a concern, and would agree that we need global coordination on AI safety to make this work.)
According to a survey of Congressional staffers, they care about personalized emails more than phone calls, and phone calls (much) more than form emails:
https://mdickens.me/assets/images/Congress-influence.png
(the way my brain works is that I’d much rather spend 5 minutes writing a personalized email than spend 5 minutes looking up the name of my representative and a relevant law to reference. YMMV)
I realize this comment makes me sound lazy (that’s because I am lazy), but the way the doc is presented gives me too much work to do, in a way that’s avoidable.
I’m not going to call my representative because I hate phone calls, so I’ll just focus on the email part. The email template requires me to look up my senator’s name and look up relevant laws from a list. I’m too lazy to do that. I would send a letter if there were a website where I could put in my state and it would auto-fill my senator’s name and the list of relevant laws. It would not be very hard to build a web form to do that.
ASPCA has a web form that requires less work on my part. It still require me to fill out a bunch of personal info but I believe that part is a requirement. Although I’m not sure it auto-fills the rep’s name, and it doesn’t give any option to customize the message.
https://ifanyonebuildsit.com/act/letter (unrelated to farm bill) is a good example of a page that auto-fills as much information as possible.
Here are three other responses that I didn’t see addressed. I only skimmed the original article so apologies if these were addressed there.
The saturation view does not obviously avoid the repugnant conclusion. (To be clear, I don’t think the repugnant conclusion should be avoided; I think people’s intuitions are misled by scope insensitivity.) The number of morally distinct experiences is probably very large—perhaps proportional to 2 to the power of the number of neurons, which would mean there are vastly more possible experiences than will ever realize. You could easily have a world with (say) 10^20 barely-worth-living lives that are all sufficiently distinct such that this world, on the saturation view, is better than a world with only (say) 10^7 extremely good lives.
I strongly suspect that the difference between the total view and the saturation view is a factual question, not a moral one. I suspect that if we had a solution to the hard problem of consciousness, then it would follow from that solution whether we should care about repeated experiences equally.
I have an intuition that the value of my own life is innate, and not related to who else might have a similar life. If someone offers me a good experience, I don’t say, “Wait, let me check if anyone else has had an identical experience at some point in the past, because if so, my experience won’t have any value.” Or if I have a really good experience and then after the fact someone tells me that it doesn’t count because a clone of me already had the same experience last week, I’d say: “No, you’re wrong; I just felt that experience, and it was good, and I know it was good because I experienced it being good, and you can’t take that away from me.”
A fourth point, which the article did address:
First, you could reinterpret the intuition as entirely instrumental. Perhaps variety matters only because it tends to support exploration, learning, and other welfare-promoting goods. If so, then our intuition that Variety is better than Homogeneity is a misfire, attributing intrinsic value to something that has only instrumental value.
[the article’s reply:]
Imagine some truly wonderful moment — of falling in love, or a flash of creative insight, or communing with the spiritual. Suppose that this moment is far more wonderful than anything that humanity has experienced to date: you or I would give up years of ordinary happy life just to experience such a peak of joy. But now suppose that this moment is just ever so slightly less good than some other moment that could be produced, with the same resources. So, instead, the world consists of a near-endless reliving of that other moment, and that first moment is never experienced at all.
To me, this feels like a tremendous loss. The world has omitted something wonderful that could have been created.
This doesn’t seem like a satisfying response to me. I could say that your feeling of tremendous loss stems from the instrumental value of novel experiences; you haven’t done anything to dispel that argument.
I would consider it from a first person POV. Suppose I’m given the choice to experience something amazing, or something a bit less amazing. If I experienced either thing in the past, then I have no memory of it (to guarantee that it’s a proper replication of the amazing experience, and not a version altered by memory). I would pick the amazing experience, and not the lesser one.
IMO a “Scientist AI” is more promising in a world where we first get a global ban on superintelligent AI, or something else that prevents anyone from building “high-impact AI”. Then AI developers coalesce around “Scientist AI” as a safe approach and develop it carefully.
(I still think a Scientist AI would result in everyone dying, but at least it’s a better starting point)
Right now I would give very little marginal philanthropic money to compute-based experiments. AI companies already do a lot of those, and I don’t expect them to work anyway. ML experiments are not addressing the fundamental barriers to solving AI misalignment. A core problem is that experiments can’t deal with the sharp left turn.
(I would make an exception for CaML-style alignment-to-animals work, but that’s not about AI safety as it’s normally construed.)
I’d say this is the wrong question. Like, I do not expect that any current alignment approach is going to work. If we do ever figure out what works, it will not look like “pretraining” or “post-training”, it will be something completely different.
Although I guess you could call that “pretraining”?