I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me).
I have a website: https://mdickens.me/ Much of the content on my website gets cross-posted to the EA Forum, but I also write about some non-EA stuff over there.
My favorite things that I’ve written: https://mdickens.me/favorite-posts/
I used to work as a software developer at Affirm.
Good note. Also worth keeping in mind the base rate of companies going under. FTX committing massive fraud was weird; but a young, fast-growing, unprofitable company blowing up was decidedly predictable, and IMO the EA community was banking too hard on FTX money being real.
Plus the planning fallacy, i.e., if someone says they want to do something by some date, then it’ll probably happen later than that.
My off-the-cuff guess is
30% chance Anthropic IPOs by end of 2028
20% chance Anthropic IPOs in 2029 or later
50% chance Anthropic never IPOs—because they go under for normal business-y reasons, or they build AGI first, or we’re all dead, or whatever
By default, people don’t talk publicly about their donation plans, especially 1–2 years in advance. It probably just hasn’t occurred to them that they should.
The responsible thing to do is to go look at the balance of what experts in a field are saying, and in this case, they’re fairly split
This is not a crux for me. I think if you were paying attention, it was not hard to be convinced that AI extinction risk was a big deal in 2005–2015, when the expert consensus was something like “who cares, ASI is a long way off.” Most people in my college EA group were concerned about AI risk well before ML experts were concerned about it. If today’s ML experts were still dismissive of AI risk, that wouldn’t make me more optimistic.
SF and Berkeley and south bay (San Jose/Palo Alto area) all have pretty different climates. Going off my memory:
SF: usually cloudy; 40 to 70 degrees
Berkeley/Oakland: usually cloudy in the morning and sunny by mid-day; 50 to 80 degrees
south bay: usually sunny; 40 to 90 degrees
It’s true that SF is usually cloudy but that’s not the case for the whole bay area. Berkeley/Oakland is sunny more often than not.
“EA” isn’t one single thing with a unified voice. Many EAs have indeed denounced OpenAI.
As an EA: I hereby denounce OpenAI. They have greatly increased AI extinction risk. The founding of OpenAI is a strong candidate for the worst thing to happen in history (time will tell whether this event leads to human extinction).
I love hearing that my writing helps! And Palisade is doing important work and I think they’re a good place to donate.
I’m forecasting a 0.00% to 0.02% probability range for AGI by the end of 2034, and that if I were to make 100 predictions of a similar kind, more than 95 of them would have the “correct” probability range
I kinda get what you’re saying but I think this is double-counting in a weird way. A 0.01% probability means that if you make 10,000 predictions of that kind, then about one of them should come true. So your 95% confidence interval sounds like something like “20 times, I make 10,000 predictions that each have a probability between 0.00% and 0.02%; and 19 out of 20 times, about one out of the 10,000 predictions comes true.”
You could reduce this to a single point probability. The math is a bit complicated but I think you’d end up with a point probability on the order of 0.001% (~10x lower than the original probability). But if I understand correctly, you aren’t actually claiming to have a 0.001% credence.
I think there are other meaningful statements you could make. You could say something like, “I’m 95% confident that if I spend 10x longer studying this question, then I would end up with a probability between 0.00% and 0.02%.”
My confidence interval is over 95%.
What do you mean by this? What is it that you’re 95% confident about?
I don’t think this should be downvoted. It’s a perfectly fine example of reasoning transparency. I happen to disagree, but the disagree-vote button is there for a reason.
I won’t go through this whole post but I’ll pick out a few representative bits to reply to.
Deutsch’s idea of explanatory universality helps clarify the mistake. Persons are universal explainers. They create new explanations that were not contained in past data. This creativity is not extrapolation from a dataset. It is invention.
LLMs do not do this. They remix what exists in their training corpus. They do not originate explanatory theories.
This statement expresses a high degree of confidence in a claim that has, as far as I can tell, zero supporting evidence. I would strongly bet against the prediction that LLMs will never be able to originate an explanatory theory.
Until we understand how humans create explanatory knowledge, we cannot program that capacity.
We still don’t know how humans create language, or prove mathematical conjectures, or manipulate objects in physical space, and yet we created AIs that can do those things.
The AI 2027 paper leans heavily on forecasting. But when the subject is knowledge creation, forecasting is not just difficult. It is impossible in principle. This was one of Karl Popper’s central insights.
I am not aware of any such insight? This claim seems easily falsified by the existence of superforecasters.
And: if prediction is impossible in principle, then you can’t confidently say that ASI won’t kill everyone, therefore you should regard it as potentially dangerous. But you seem to be quite confident that you know what ASI will be like.
The rationalist story claims a superintelligent AI will likely be a moral monster. This conflicts with the claim that such a system will understand the world better than humans do.
Also this is an aside but I find it telling that this post specifically focuses on the murder of women, even though men are more likely than women to be victims of murder. You say “why the silence?” but in my experience, it is the opposite—female victims get disproportionate attention relative to male victims. See also how breast cancer is the most-researched type of cancer even though it’s not the most harmful.
The relevant question isn’t “how big is this issue?” The question is, “how cost-effective is it to make progress on this issue?” EA focuses on schistosomiasis and vitamin A deficiency because they are very cheap to fix, and fixing them provides a big improvement in quality of life.
EAs don’t focus on lung cancer, even though lung cancer is responsible for a lot more DALYs than schistosomiasis (or femicide), because lung cancer interventions aren’t as cost-effective as schistosomiasis interventions.
Oh maybe that doc wasn’t supposed to be public, I will remove the reference.
Key Uncertainties
The listed uncertainties do not include my biggest question, which is: “do Talos fellows do more to reduce AI x-risk than the counterfactual hire?”
I see, I took the chart under “The compensation schedule’s structure” to imply that the Axiom of Continuity held for suffering, based on the fact that the X axis shows suffering measured on a cardinal scale.
If you reject Continuity for suffering then I don’t think your assumptions are self-contradictory.
I think a downvote-as-disapproval wouldn’t work in this case, I would want to use upvote+disagree to express “this is an important change that the community should know about, but I disagree with the decision”. A downvote de facto communicates “I don’t want people to read this post”.
Didn’t Sam tell several straightforward lies? e.g. claiming that they had enough assets to fully cover the account values of all users, which they didn’t; claiming that Alameda never borrowed users’ deposits, which it did.