I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me).
I have a website: https://mdickens.me/ Much of the content on my website gets cross-posted to the EA Forum, but I also write about some non-EA stuff over there.
I used to work as a software developer at Affirm.
Superforecasters tend to believe x-risk isn’t a big deal. Regardless of whether they’re using reasonable procedures, they’re getting the wrong object-level answer in this case. FRI’s consulting plausibly made the scaling policies worse. Hard to say without knowing more details.
(I’m thinking particular of XPT, which is from 2023 so it may be outdated at this point. But that tournament had superforecasters predicting only a 1% chance of AI extinction, which is ludicrous and should not be used as the basis for decisions.)
Re: ControlAI, seems that they’re currently fundraising, but only accepting gifts of $100k+. Have updated my post to reflect this.
I missed that, thanks for calling it out!
if you have recs on things to read, would be useful :)
Since I’m biased on the matter, I will start by linking the posts I’ve written:
Where I Am Donating in 2024 – see the comments too, there were some disagreements with my reasoning
Eric Neyman and Zach Stein-Perlman write recommendations on AI risk advocacy, most of their work is non-public but Eric wrote Consider donating to Alex Bores, author of the RAISE Act.
Zvi wrote The Big Nonprofits Post [2024] and 2025. He’s a grantmaker for SFF which is my favorite of the big grantmakers.
Coefficient Giving and Longview Philanthropy often write about their grantmaking decisions (although not in as much detail as I’d like). I tend to have some pretty big disagreements with them, but they’re still worth reading.
Most reasoning on grantmaking/donations happens in private, so there’s not a whole lot to read. If you broaden the question to writings on general strategy (not just donations), there’s a ton of stuff worth reading. I will just link two that align best with my personal views:
No Winners – Q&A on why an international halt on AI development should be the goal
A Narrow Path – ControlAI’s long-term plan for avoiding extinction
All the c3 orgs I mentioned are funding-constrained (except perhaps MIRI). AI x-risk advocacy is unpopular among big funders (with a couple exceptions like SFF), which means both the c3s and c4s are funding-constrained. At least that’s true currently; I’m not sure if that will still be true a year from now.
I agree with you about the structural funding disadvantage to c4s, but empirically it doesn’t look obviously important for driving funding constraints.
I’m also uncertain about the relative value of lobbying vs. other forms of advocacy. My intuition is that direct lobbying is underfunded relative to its impact, partly because the structural barriers to funding it are so high, but I’d love to see more analysis on this.
Yeah this is a difficult question, I don’t know. Another problem is that lobbying is opaque so it’s harder to tell who’s doing a good job. (ControlAI is my favorite lobbying org because they write a lot about what they do and I like what they write. But also I reached out to them and they said they were not accepting donations edit: only accepting donations of $100K or more.)
I’m not sure if there’s any. My concerns are more theoretical than empirical, so it would take theoretical work to significantly change my mind.
Empirical work can provide a small amount of information, e.g. the fact that Claude expresses concern for ethics is a slight positive update relative to the world where Claude doesn’t care about ethics, and I would feel slightly better about a Claude-based ASI than a ChatGPT-based ASI. But only slightly, because I don’t think empirically observable behavior is that relevant to determining whether an AI is aligned. At least not using any empirical methods that we’ve devised so far.
For more on this, see e.g. A central AI alignment problem: capabilities generalization, and the sharp left turn, especially the part starting from ‘How is the “capabilities generalize further than alignment” problem upstream of these problems?’
(ETA: On how various plans miss the hard bits of the alignment challenge is also kind of about this...I was looking around for some writings on why current empirical work isn’t that relevant but it’s hard to find anything that directly makes the argument)
From my POV we are deeply confused about what it would even mean to align ASI. If I could even describe what sort of theoretical work would be good evidence of progress on alignment, we’d be in a better place than we are currently.
A significant reason for my high P(doom) is that most safety researchers at AI companies are ignoring theoretical issues and pretending that alignment is purely an engineering problem. I don’t think they are institutionally capable of solving alignment.
An LLM’s expressed values are not the same thing as its actual values, insofar as it has any.
This paper doesn’t really tell us anything about how ASI values will work. This paper is relevant to the immediate problem of making Claude commercially useful and non-destructive, but it’s not relevant to ASI.
This isn’t something I’ve thought a ton about but I think forecasting should plausibly still receive funding in a specific way:
Funders should either pay forecasters to make predictions on important questions, or subsidize prediction markets on those questions.
I don’t think forecasting is a “solution seeking a problem.” There are tons of important but hard-to-predict questions that I’d like better forecasts on! The problem is that the ecosystem hasn’t done a great job of turning dollars into good forecasts.
For example, most of my Metaculus questions are things I wanted answers to, but I tended not to update on the results because the questions usually don’t receive a lot of forecasts. If someone wanted to pay money to get more predictions on questions, I’d learn something useful!
I’m not sure how valuable this is compared to other uses of money (I wouldn’t pay for it myself) but at least it’s better than more general-purpose research on forecasting.
Thanks for this post! I broadly agree with your introduction but I’d say the “research vs. advocacy” distinction is more important than “501(c)(3) vs. 501(c)(4)”. There are 501(c)(3) advocacy orgs including MIRI, Palisade, PauseAI US/PauseAI Global, Lightcone Infrastructure (sort of), etc. Lobbying on policy measures is one component of advocacy, but not the only component. I’m uncertain about how valuable it is compared to other kinds of advocacy that 501(c)(3)s can do.
AI systems are being built to do exactly what they were designed to do, which is to faithfully execute human preferences. And those are, in aggregate, to eat cheap meat, conduct research on living organisms when it’s convenient, and prioritise cost and efficiency in agricultural supply chains. AI is reflecting the values of the humans. I don’t think you can sneak those values in, unless there are specific opportunities to tweak things here and there before they get cemented.
This is true in a way. A deeper problem is that we don’t know what values AI is reflecting. If you talk to an LLM, it will express some values, but it gives inconsistent answers depending on what questions you ask. We have no way of knowing whether its expressed values reflect its “true” values, if it has any. And we don’t know how things will change as AI becomes increasingly powerful.
Is voting a Pascal’s mugging?
Is searching for a breakthrough in cancer treatment a Pascal’s mugging?
Marcus also nudged his friend Ege Erdil to start Mechanize, and offered them their first investment.
Mechanize’s mission is to build AGI to replace human labor. They actively deny that AI misalignment risk is a concern. I would not list this as a positive under AI safety track record.
I keep seeing people use the word “dead” in this way and I don’t get it. Today so far I see 8 posts and 2 shortforms. Dead would be 0 posts and 0 shortforms. Do you have some standard for how many posts the EA Forum should have per day?
Here I came up with a short list of orgs who have written legislation or similar:
Center for AI Policy (now mostly defunct)
I also made a list of orgs who talk to policy-makers about AI risk (excluding the above):
Control AI (UK/US)
Good Ancestors (Australia)
Machine Intelligence Research Institute (US/global) (see especially their work on technical governance)
Palisade Research (US)
PauseAI US (US)
A few 501(c)(4)s I’d add:
Americans for Responsible Innovation
AI Policy Network
Public First Action
These don’t all do exactly what you’re describing but they’re in the same ballpark.
Why is the idea that insects have net negative lives dangerous, but the idea that they have net positive lives isn’t dangerous? If you promote the idea that they have net positive lives when in fact they’re net negative, you could end up doing tremendous harm.
I think it’s fine to make arguments that insects’ lives are positive on balance, but I reject the framing of “dangerousness”. You are too quick to jump from “insects’ lives are net positive” to ”...and it should be taboo to argue otherwise”.
Copying from a comment I wrote yesterday:
Either ASI has more than zero values locked in, or it’s fully corrigible. If any values at all are locked in, then we need to have a pretty robust understanding of what the consequences of that will be, because we can’t change it ever. Like I don’t think we know how to encode something like “don’t let people do power grabs, but be fully corrigible in every other way”. I don’t know how much that’s downstream of the facts that (1) we don’t know how to encode any values at all and (2) we don’t know how to encode corrigibility, but my intuition is that even if we solve #1 and #2, the problem of “don’t pick incorrigible values that will screw everything up down the road” is still a hard problem.
This is related to Max Harms’ work on CAST. Part of his argument is that pure corrigibility is a more robust target than any set of values because a near miss fails gracefully. Whereas if you try to encode any values at all, a near miss could be catastrophic. He’s talking more about the “AI kills everyone” flavor of catastrophe, which is valid, but what I’m talking about here is more that a near miss could permanently lock us in to a bad (or maybe just not-that-good) future. Different argument but the concern arises for a similar reason—if you’re specifying values, then you have to get the specification right, beyond just ensuring that the AI does what you want.
I’m in favour of slowing the intelligence explosion (and in particular of “Pause at human-level”.)
Shouldn’t making that happen be a top priority, then? We as a civilization need to get a lot of groundwork in place to make a human-level pause happen: international agreement, defined red lines, enforcement mechanisms (e.g. GPU tracking), etc. Given how neglected these issues are and the catastrophic consequences of building ASI too quickly, it seems that laying the groundwork for a pause is a higher priority than, or at least on par with, the top causes on this post’s “neglected cause areas” list.
I was reading the AI-enabled coups report and one of the mitigations is roughly “build the AI model such that it refuses to do coups”. If you can pull that off then it solves the problem, but it means the model is incorrigible, and therefore you have to exactly specify the correct values up front because you’re locked-in.
There might be some other report saying the way to avoid bad value lock-in is to make AI corrigible, and you can object to that by pointing out how it enables coups.
You can’t look at the problems separately, you have to consider them at the same time and find a solution that works for every problem.
Right now I would give very little marginal philanthropic money to compute-based experiments. AI companies already do a lot of those, and I don’t expect them to work anyway. ML experiments are not addressing the fundamental barriers to solving AI misalignment. A core problem is that experiments can’t deal with the sharp left turn.
(I would make an exception for CaML-style alignment-to-animals work, but that’s not about AI safety as it’s normally construed.)