I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me).
I have a website: https://mdickens.me/ Much of the content on my website gets cross-posted to the EA Forum, but I also write about some non-EA stuff over there.
My favorite things that I’ve written: https://mdickens.me/favorite-posts/
I used to work as a software developer at Affirm.
By default, people don’t talk publicly about their donation plans, especially 1–2 years in advance. It probably just hasn’t occurred to them that they should.
The responsible thing to do is to go look at the balance of what experts in a field are saying, and in this case, they’re fairly split
This is not a crux for me. I think if you were paying attention, it was not hard to be convinced that AI extinction risk was a big deal in 2005–2015, when the expert consensus was something like “who cares, ASI is a long way off.” Most people in my college EA group were concerned about AI risk well before ML experts were concerned about it. If today’s ML experts were still dismissive of AI risk, that wouldn’t make me more optimistic.
SF and Berkeley and south bay (San Jose/Palo Alto area) all have pretty different climates. Going off my memory:
SF: usually cloudy; 40 to 70 degrees
Berkeley/Oakland: usually cloudy in the morning and sunny by mid-day; 50 to 80 degrees
south bay: usually sunny; 40 to 90 degrees
It’s true that SF is usually cloudy but that’s not the case for the whole bay area. Berkeley/Oakland is sunny more often than not.
“EA” isn’t one single thing with a unified voice. Many EAs have indeed denounced OpenAI.
As an EA: I hereby denounce OpenAI. They have greatly increased AI extinction risk. The founding of OpenAI is a strong candidate for the worst thing to happen in history (time will tell whether this event leads to human extinction).
I love hearing that my writing helps! And Palisade is doing important work and I think they’re a good place to donate.
I’m forecasting a 0.00% to 0.02% probability range for AGI by the end of 2034, and that if I were to make 100 predictions of a similar kind, more than 95 of them would have the “correct” probability range
I kinda get what you’re saying but I think this is double-counting in a weird way. A 0.01% probability means that if you make 10,000 predictions of that kind, then about one of them should come true. So your 95% confidence interval sounds like something like “20 times, I make 10,000 predictions that each have a probability between 0.00% and 0.02%; and 19 out of 20 times, about one out of the 10,000 predictions comes true.”
You could reduce this to a single point probability. The math is a bit complicated but I think you’d end up with a point probability on the order of 0.001% (~10x lower than the original probability). But if I understand correctly, you aren’t actually claiming to have a 0.001% credence.
I think there are other meaningful statements you could make. You could say something like, “I’m 95% confident that if I spend 10x longer studying this question, then I would end up with a probability between 0.00% and 0.02%.”
My confidence interval is over 95%.
What do you mean by this? What is it that you’re 95% confident about?
I don’t think this should be downvoted. It’s a perfectly fine example of reasoning transparency. I happen to disagree, but the disagree-vote button is there for a reason.
I won’t go through this whole post but I’ll pick out a few representative bits to reply to.
Deutsch’s idea of explanatory universality helps clarify the mistake. Persons are universal explainers. They create new explanations that were not contained in past data. This creativity is not extrapolation from a dataset. It is invention.
LLMs do not do this. They remix what exists in their training corpus. They do not originate explanatory theories.
This statement expresses a high degree of confidence in a claim that has, as far as I can tell, zero supporting evidence. I would strongly bet against the prediction that LLMs will never be able to originate an explanatory theory.
Until we understand how humans create explanatory knowledge, we cannot program that capacity.
We still don’t know how humans create language, or prove mathematical conjectures, or manipulate objects in physical space, and yet we created AIs that can do those things.
The AI 2027 paper leans heavily on forecasting. But when the subject is knowledge creation, forecasting is not just difficult. It is impossible in principle. This was one of Karl Popper’s central insights.
I am not aware of any such insight? This claim seems easily falsified by the existence of superforecasters.
And: if prediction is impossible in principle, then you can’t confidently say that ASI won’t kill everyone, therefore you should regard it as potentially dangerous. But you seem to be quite confident that you know what ASI will be like.
The rationalist story claims a superintelligent AI will likely be a moral monster. This conflicts with the claim that such a system will understand the world better than humans do.
Also this is an aside but I find it telling that this post specifically focuses on the murder of women, even though men are more likely than women to be victims of murder. You say “why the silence?” but in my experience, it is the opposite—female victims get disproportionate attention relative to male victims. See also how breast cancer is the most-researched type of cancer even though it’s not the most harmful.
The relevant question isn’t “how big is this issue?” The question is, “how cost-effective is it to make progress on this issue?” EA focuses on schistosomiasis and vitamin A deficiency because they are very cheap to fix, and fixing them provides a big improvement in quality of life.
EAs don’t focus on lung cancer, even though lung cancer is responsible for a lot more DALYs than schistosomiasis (or femicide), because lung cancer interventions aren’t as cost-effective as schistosomiasis interventions.
Oh maybe that doc wasn’t supposed to be public, I will remove the reference.
Key Uncertainties
The listed uncertainties do not include my biggest question, which is: “do Talos fellows do more to reduce AI x-risk than the counterfactual hire?”
I see, I took the chart under “The compensation schedule’s structure” to imply that the Axiom of Continuity held for suffering, based on the fact that the X axis shows suffering measured on a cardinal scale.
If you reject Continuity for suffering then I don’t think your assumptions are self-contradictory.
I think a downvote-as-disapproval wouldn’t work in this case, I would want to use upvote+disagree to express “this is an important change that the community should know about, but I disagree with the decision”. A downvote de facto communicates “I don’t want people to read this post”.
Yeah it’s also something I want to get more clarity on. This post is about the step of the chain that goes from “donate money to campaign” → “candidates gets elected”, but it’s harder to say what happens after that. I’m working on some future posts that I hope will help me get a better understanding.
Some thoughts:
Overall I think I am more confident than you that extinction risk is more important than catastrophic risk, but I agree that this topic is worth exploring and I’m glad you’re doing it. We have too much of a tendency to fall into “X is more important than Y” [with 90% probability] and then spend so few resources on Y that Y is now more valuable on the margin even though it only has a 10% chance of mattering. (I’m not saying 10% is the correct number in this case, that’s just an example)
In your Parfit-esque categorization of three scenarios, I agree that “the difference in value between 3 and 2 is very much greater than the difference between 2 and 1” and I think this is important, although I would also note that #3 could be much worse than #2 if #3 entails spreading wild animal suffering.
I’m having a hard time wrapping my head around what the “1 unit of extinction” equation is supposed to represent.
By my interpretation, the parable of the apple tree is more about P(recovery) than it is about P(flourishing|recovery). If the low-hanging apples are eaten, that makes it harder to rebuild the tall ladders. But once you’ve rebuilt them, you’re okay. I think your central argument of “flourishing might be harder after recovery than if the catastrophe had been avoided” is still good, but the parable doesn’t seem to support that argument.
The model of P(success) = (1 - r)^N seems much clearer, but it also seems a bit different than what I understood your argument to be. I understood the fruit-picking argument to be something like “Resources get used up, so getting back to a level of technology the 2nd time is harder than the 1st time.” But this equation is embedding an argument more like “A higher probability of catastrophe means there’s a higher chance that civilization keeps getting set back by catastrophes without ever expanding to the stars.” Which is also an interesting idea but it seems different than what you wrote in prose. It chains into the first argument because the more times civilization collapses, the more resources get used up; but you could also make the 1st argument without making the 2nd one (i.e., even a single catastrophe may be enough to permanently prevent interstellar expansion).
“If AGI is significantly more than a decade away and higher annual risk estimates of other catastrophes are plausible, then we might still have a greater expected loss of value from those catastrophes”—This seems unlikely to me, but I’d like to see some explicit modeling. What range of assumptions would make us want to prioritize [non-extinction, non-AI] catastrophe reduction over AI x-risk?
“Reclaimed technology makes tech progress easy for future civilisations”—I agree that this seems outweighed by the opposite consideration, although I’m not confident and I don’t think anyone has a great understanding of this.
“But by far the strongest and surely least controversial consequence of this model is that longtermists should seek to reduce non-extinction catastrophic risk almost as urgently as they do extinction risks” – This conclusion depends on the numbers. The model converted catastrophic risk into “extinction units” but it’s not clear from the model that catastrophic risk is of similar importance to extinction risk (and my sense is that it isn’t).
Good question, it’s a good idea to think about this in advance.
Some thoughts:
It’s pretty unlikely that Anthropic goes public at a time that matters. Much more likely is that either AGI happens and everyone dies; or AGI happens and the economy is radically transformed to the point that things like “going public” don’t make sense anymore; or AI progress fizzles and Anthropic can’t support the growth that investors expect, and it shuts down or radically contracts.
A more likely scenario is that Anthropic has a tender offer where early employees can sell a lot of equity.
But tender offers are still limited in scope. Plus, the people with the most equity—founders/execs—often don’t sell much equity because they want to signal confidence to investors. If you multiply the discounts due to (1) tender offer will have a cap on sales, (2) biggest equity holders probably won’t sell much, (3) sellers will only donate some fraction of their equity, I would be surprised if a tender offer resulted in more than $1 billion for EA causes—which is great, but wouldn’t change the ecosystem the way FTX did. On the other hand, Anthropic donors might want to spend down very quickly due to short timelines. $1 billion spent over two years would definitely change the funding ecosystem.
There a high chance that we won’t figure out how to align AI and it will kill everyone, and the best move is a global ban on ASI. People at Anthropic will disproportionately disagree with me about this, because (1) people who believe that will mostly not go to work at an AI company, and (2) among those who do believe that, many will be influenced by their colleagues to change their minds. So I don’t expect to see much increase in funding for the interventions that look best by my lights.
If you’re an AI safety donor, the value of different donation opportunities can depend a lot on your worldview; every grantmaker (except Survival and Flourishing Fund, which doesn’t take donations from small donors AFAIK) has a worldview in a particular cluster*; many individuals don’t share that worldview; and if you don’t share that worldview, most of the grants that big grantmakers make don’t look ideal.
At one point the same was true in animal welfare where grantmakers didn’t make grants to “weird” stuff like wild animal welfare or shrimp welfare, but that’s becoming less true lately because grantmakers have been branching out more (which IMO is a positive development).
*my best attempt at concisely summarizing it is on the spectrum from Eliezer Yudkowsky to Paul Christiano, all the grantmakers (except for SFF) are much more on the Christiano side.
Good note. Also worth keeping in mind the base rate of companies going under. FTX committing massive fraud was weird; but a young, fast-growing, unprofitable company blowing up was decidedly predictable, and IMO the EA community was banking too hard on FTX money being real.
Plus the planning fallacy, i.e., if someone says they want to do something by some date, then it’ll probably happen later than that.
My off-the-cuff guess is
30% chance Anthropic IPOs by end of 2028
20% chance Anthropic IPOs in 2029 or later
50% chance Anthropic never IPOs—because they go under for normal business-y reasons, or they build AGI first, or we’re all dead, or whatever