I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me).
I have a website: https://mdickens.me/ Much of the content on my website gets cross-posted to the EA Forum, but I also write about some non-EA stuff over there.
I used to work as a software developer at Affirm.
After making sure we don’t all die, this should be the first priority
Please do not write clickbait titles like this. Put the subject matter in the title.
But I think expecting returns something like 50-100% p.a. would be reasonable, based on historic performance of these funds.
You can’t have it both ways. Either you can extrapolate from historical performance, in which case you should use long-run market returns of ~5%. Or you have a specific view on what’s going to outperform, in which case it’s a question of what your expectations are. You can’t just take a fund that performed extremely well over the last 3–5 years and then say that level of performance will continue. If some thesis did really well in the past, then everyone else can also see that it did well, and you should assume it’s now priced in unless you have some marginal thesis for why it’s not priced in.
In fact, in the long run, stocks have exhibited 3–5 year reversals—that is, stocks that perform particularly well over the last 3–5 years tend to underperform the market going forward.[1] Mutual funds haven’t exhibited reversals, but funds that outperform tend to regress to average performance.[2] At minimum, your thesis needs to have some explanation of why that’s not going to happen this time.
I’m not saying you’re wrong. AI stocks may well be a good investment. But I am pretty unimpressed by most arguments I’ve seen. “The last 5 years saw 50–100% return, therefore that same level of return will continue” is not a good argument—by (weak) default, it’s evidence that future returns will be lower than the market, not higher. (I wrote more on my thoughts in a recent post, especially this section.)
[1] Fama, E. F., & French, K. R. (1988). Permanent and Temporary Components of Stock Prices. Journal of Political Economy, 96(2), 246–273. http://www.jstor.org/stable/1833108
[2] Berk, J. B., & Green, R. C. (2004). Mutual Fund Flows and Performance in Rational Markets. Journal of Political Economy, 112(6), 1269–1295. https://doi.org/10.1086/424739
How would you model the expected investment rate of return? I generally assume a 5% real return, which is small enough that it wouldn’t affect the model in OP very much. If want to incorporate a large expected return on AI investments, then I don’t know how to do that anymore.
I basically agree, although “dangerous approaches are tamped down” is doing most of the work here IMO. By default (i.e. no tamping-down), I expect the situation with a weakly-superhuman Scientist AI to be:
a small number of sane people ask “are we doomed if we go with the following alignment strategy”, and when it says yes, they don’t do it
a lot of people don’t bother to ask at all, they just ask the Scientist AI how to build ASI
a lot of people say “we have to build ASI before the reckless people in group 2”, they build ASI using their best-guess alignment strategy that has an 88.5% chance of failing, and we die with 88.5% probability
(I think Bengio would agree that this is a concern, and would agree that we need global coordination on AI safety to make this work.)
According to a survey of Congressional staffers, they care about personalized emails more than phone calls, and phone calls (much) more than form emails:
https://mdickens.me/assets/images/Congress-influence.png
(the way my brain works is that I’d much rather spend 5 minutes writing a personalized email than spend 5 minutes looking up the name of my representative and a relevant law to reference. YMMV)
I realize this comment makes me sound lazy (that’s because I am lazy), but the way the doc is presented gives me too much work to do, in a way that’s avoidable.
I’m not going to call my representative because I hate phone calls, so I’ll just focus on the email part. The email template requires me to look up my senator’s name and look up relevant laws from a list. I’m too lazy to do that. I would send a letter if there were a website where I could put in my state and it would auto-fill my senator’s name and the list of relevant laws. It would not be very hard to build a web form to do that.
ASPCA has a web form that requires less work on my part. It still require me to fill out a bunch of personal info but I believe that part is a requirement. Although I’m not sure it auto-fills the rep’s name, and it doesn’t give any option to customize the message.
https://ifanyonebuildsit.com/act/letter (unrelated to farm bill) is a good example of a page that auto-fills as much information as possible.
Here are three other responses that I didn’t see addressed. I only skimmed the original article so apologies if these were addressed there.
The saturation view does not obviously avoid the repugnant conclusion. (To be clear, I don’t think the repugnant conclusion should be avoided; I think people’s intuitions are misled by scope insensitivity.) The number of morally distinct experiences is probably very large—perhaps proportional to 2 to the power of the number of neurons, which would mean there are vastly more possible experiences than will ever realize. You could easily have a world with (say) 10^20 barely-worth-living lives that are all sufficiently distinct such that this world, on the saturation view, is better than a world with only (say) 10^7 extremely good lives.
I strongly suspect that the difference between the total view and the saturation view is a factual question, not a moral one. I suspect that if we had a solution to the hard problem of consciousness, then it would follow from that solution whether we should care about repeated experiences equally.
I have an intuition that the value of my own life is innate, and not related to who else might have a similar life. If someone offers me a good experience, I don’t say, “Wait, let me check if anyone else has had an identical experience at some point in the past, because if so, my experience won’t have any value.” Or if I have a really good experience and then after the fact someone tells me that it doesn’t count because a clone of me already had the same experience last week, I’d say: “No, you’re wrong; I just felt that experience, and it was good, and I know it was good because I experienced it being good, and you can’t take that away from me.”
A fourth point, which the article did address:
First, you could reinterpret the intuition as entirely instrumental. Perhaps variety matters only because it tends to support exploration, learning, and other welfare-promoting goods. If so, then our intuition that Variety is better than Homogeneity is a misfire, attributing intrinsic value to something that has only instrumental value.
[the article’s reply:]
Imagine some truly wonderful moment — of falling in love, or a flash of creative insight, or communing with the spiritual. Suppose that this moment is far more wonderful than anything that humanity has experienced to date: you or I would give up years of ordinary happy life just to experience such a peak of joy. But now suppose that this moment is just ever so slightly less good than some other moment that could be produced, with the same resources. So, instead, the world consists of a near-endless reliving of that other moment, and that first moment is never experienced at all.
To me, this feels like a tremendous loss. The world has omitted something wonderful that could have been created.
This doesn’t seem like a satisfying response to me. I could say that your feeling of tremendous loss stems from the instrumental value of novel experiences; you haven’t done anything to dispel that argument.
I would consider it from a first person POV. Suppose I’m given the choice to experience something amazing, or something a bit less amazing. If I experienced either thing in the past, then I have no memory of it (to guarantee that it’s a proper replication of the amazing experience, and not a version altered by memory). I would pick the amazing experience, and not the lesser one.
IMO a “Scientist AI” is more promising in a world where we first get a global ban on superintelligent AI, or something else that prevents anyone from building “high-impact AI”. Then AI developers coalesce around “Scientist AI” as a safe approach and develop it carefully.
(I still think a Scientist AI would result in everyone dying, but at least it’s a better starting point)
Right now I would give very little marginal philanthropic money to compute-based experiments. AI companies already do a lot of those, and I don’t expect them to work anyway. ML experiments are not addressing the fundamental barriers to solving AI misalignment. A core problem is that experiments can’t deal with the sharp left turn.
(I would make an exception for CaML-style alignment-to-animals work, but that’s not about AI safety as it’s normally construed.)
Superforecasters tend to believe x-risk isn’t a big deal. Regardless of whether they’re using reasonable procedures, they’re getting the wrong object-level answer in this case. FRI’s consulting plausibly made the scaling policies worse. Hard to say without knowing more details.
(I’m thinking particular of XPT, which is from 2023 so it may be outdated at this point. But that tournament had superforecasters predicting only a 1% chance of AI extinction, which is ludicrous and should not be used as the basis for decisions.)
Re: ControlAI, seems that they’re currently fundraising, but only accepting gifts of $100k+. Have updated my post to reflect this.
I missed that, thanks for calling it out!
if you have recs on things to read, would be useful :)
Since I’m biased on the matter, I will start by linking the posts I’ve written:
Where I Am Donating in 2024 – see the comments too, there were some disagreements with my reasoning
Eric Neyman and Zach Stein-Perlman write recommendations on AI risk advocacy, most of their work is non-public but Eric wrote Consider donating to Alex Bores, author of the RAISE Act.
Zvi wrote The Big Nonprofits Post [2024] and 2025. He’s a grantmaker for SFF which is my favorite of the big grantmakers.
Coefficient Giving and Longview Philanthropy often write about their grantmaking decisions (although not in as much detail as I’d like). I tend to have some pretty big disagreements with them, but they’re still worth reading.
Most reasoning on grantmaking/donations happens in private, so there’s not a whole lot to read. If you broaden the question to writings on general strategy (not just donations), there’s a ton of stuff worth reading. I will just link two that align best with my personal views:
No Winners – Q&A on why an international halt on AI development should be the goal
A Narrow Path – ControlAI’s long-term plan for avoiding extinction
All the c3 orgs I mentioned are funding-constrained (except perhaps MIRI). AI x-risk advocacy is unpopular among big funders (with a couple exceptions like SFF), which means both the c3s and c4s are funding-constrained. At least that’s true currently; I’m not sure if that will still be true a year from now.
I agree with you about the structural funding disadvantage to c4s, but empirically it doesn’t look obviously important for driving funding constraints.
I’m also uncertain about the relative value of lobbying vs. other forms of advocacy. My intuition is that direct lobbying is underfunded relative to its impact, partly because the structural barriers to funding it are so high, but I’d love to see more analysis on this.
Yeah this is a difficult question, I don’t know. Another problem is that lobbying is opaque so it’s harder to tell who’s doing a good job. (ControlAI is my favorite lobbying org because they write a lot about what they do and I like what they write. But also I reached out to them and they said they were not accepting donations edit: only accepting donations of $100K or more.)
I’m not sure if there’s any. My concerns are more theoretical than empirical, so it would take theoretical work to significantly change my mind.
Empirical work can provide a small amount of information, e.g. the fact that Claude expresses concern for ethics is a slight positive update relative to the world where Claude doesn’t care about ethics, and I would feel slightly better about a Claude-based ASI than a ChatGPT-based ASI. But only slightly, because I don’t think empirically observable behavior is that relevant to determining whether an AI is aligned. At least not using any empirical methods that we’ve devised so far.
For more on this, see e.g. A central AI alignment problem: capabilities generalization, and the sharp left turn, especially the part starting from ‘How is the “capabilities generalize further than alignment” problem upstream of these problems?’
(ETA: On how various plans miss the hard bits of the alignment challenge is also kind of about this...I was looking around for some writings on why current empirical work isn’t that relevant but it’s hard to find anything that directly makes the argument)
From my POV we are deeply confused about what it would even mean to align ASI. If I could even describe what sort of theoretical work would be good evidence of progress on alignment, we’d be in a better place than we are currently.
A significant reason for my high P(doom) is that most safety researchers at AI companies are ignoring theoretical issues and pretending that alignment is purely an engineering problem. I don’t think they are institutionally capable of solving alignment.
An LLM’s expressed values are not the same thing as its actual values, insofar as it has any.
This paper doesn’t really tell us anything about how ASI values will work. This paper is relevant to the immediate problem of making Claude commercially useful and non-destructive, but it’s not relevant to ASI.
This isn’t something I’ve thought a ton about but I think forecasting should plausibly still receive funding in a specific way:
Funders should either pay forecasters to make predictions on important questions, or subsidize prediction markets on those questions.
I don’t think forecasting is a “solution seeking a problem.” There are tons of important but hard-to-predict questions that I’d like better forecasts on! The problem is that the ecosystem hasn’t done a great job of turning dollars into good forecasts.
For example, most of my Metaculus questions are things I wanted answers to, but I tended not to update on the results because the questions usually don’t receive a lot of forecasts. If someone wanted to pay money to get more predictions on questions, I’d learn something useful!
I’m not sure how valuable this is compared to other uses of money (I wouldn’t pay for it myself) but at least it’s better than more general-purpose research on forecasting.
New title is better!