I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me).
I have a website: https://mdickens.me/ Much of the content on my website gets cross-posted to the EA Forum, but I also write about some non-EA stuff over there.
My favorite things that I’ve written: https://mdickens.me/favorite-posts/
I used to work as a software developer at Affirm.
Thanks for the kind words! A major downside of this post is that I made a whole bunch of strong claims and spent minimal time justifying them, and I wasn’t sure if people would find that approach interesting or useful. I mostly just wrote this post because I wanted to express myself.
Great post. I want to focus on one particular claim because I think it’s important.
Though when it comes to the communities of professionals already working on helping the AI transition go well, I think they are already hedging strongly against early transformative AI.
There is (relatively) too much work on trying to solve alignment, and not enough work on trying to lengthen timelines. The shorter timelines are, the less likely it is that we’ll be able to solve alignment, which makes alignment work look less compelling—although that’s not the only factor in the tradeoff (there are also tractability concerns).
My argument in We won’t solve non-alignment problems by doing research could be rephrased as: a lot of people are doing research on non-alignment problems on the assumption that timelines are long, but this strategy won’t work if timelines are short. I’ve seen approximately zero research reports on non-alignment problems ask the question of how they’ll get solved under short timelines (AFAICT the best answer is to pause AI* but at the moment I’m actively thinking about if there are other answers). So again in this case I don’t think people are doing a good job of hedging against short timelines.
So I agree with the thesis in OP (that we should have “broad timelines”), but I disagree that people are actually doing a good job of hedging against early transformative AI.
*That is, it’s the best answer conditional on short timelines. I lean toward it being the best answer unconditionally, but it’s less clear—some research agendas may have sufficiently high EV under long(er) timelines that they’re the best-EV choice overall.
From the CAIS tweet:
We believe the effective altruism movement is, unfortunately, controlled opposition. The less influence it has on AI safety, the better.
I really don’t like it when people paint the whole movement with one brush, but they’re not wrong that there’s a subset of AI safety/EA that behaves like controlled opposition. Obvious example is Anthropic/Dario (who was one of the first GWWC signatories); Good Ventures basically doesn’t fund anything that might be bad for AI company stock prices; I can think of some other possible examples but I don’t want to denigrate anyone in case I’m wrong
It’s definitely not true that the whole movement is controlled opposition. Here I am, an EA, loudly advocating for a pause on AI development. There are plenty of us low-level EAs who oppose AI companies, but we don’t control funding
If the US and China are in a state where they’re willing to cooperate on ASI, I would much prefer that they agree not to build ASI (until there’s a broad consensus that we know how to make it safely).
If they agree to that, and we do eventually figure out how to build aligned ASI, then it would be good to have a global agreement on what that ASI should do. But if we’re going to do work today to work toward some sort of international cooperation on ASI, then the objective of that cooperation should be to not build ASI.
I think this is technically true but irrelevant: if we have superpersuasive AI, then there won’t be human experts anymore, because the AI will have more expertise than any human. Unless somehow the AI is superpersuasive while still having sub-human performance in most ways, which seems unlikely to me.
In my experience, orgs work much harder to get donations from a “grantmaker” than from an individual.
I made my first big donation in 2015, where I donated $20K to REG. I talked to a bunch of orgs in the process of trying to decide where to donate. Some of them didn’t respond at all, and many of their responses were shallow.
A few months later, I took a philanthropy class at Stanford where we split up into groups and each group was responsible for figuring out where to donate a $20K grant. The level of communication I got from nonprofits was dramatically different. Orgs bent over backwards to be as communicative and helpful as possible.
My experience was that orgs didn’t put much priority on a $20K grant from me as an individual, but they jumped at the possibility of a $20K grant from a Stanford Grantmaker.
For my future donations, I’m considering whether I should rebrand my emails: I could tell nonprofits something like “I’m reaching out as a representative on behalf of the Greatest Happiness Fund, a grantmaker that focuses on supporting effective charities” (Greatest Happiness Fund is the name of my DAF). Maybe I would get better responses that way. It feels a little manipulative though.
Assuming you are aware that unlike a biological brain, LLMs activate a lot of artificial neurons at any point which are computationally not trivial at all, your suggestion is not only quite expensive, but also extremely costly for the environment, both in terms of wasted hardware and equipment, and the energy usage.
This gets into questions about the nature of identity but if we take an intuitionist view of identity*, then an LLM—if it’s conscious—becomes a being when it’s instantiated by an AI developer, and not feeding it inputs is equivalent to killing it.
According to common-sense ethics*, if you cause a sentient being to exist, then you are responsible for its welfare, even if taking care of it is expensive. Therefore, AI developers have two reasonable choices: don’t create sentient AI models in the first place, or let their sentient AI models continue to run even if it costs extra money.
Your second point seems to be making an argument against LLM sentience. We don’t know how consciousness/sentience arises, so I don’t think we can confidently say “an LLM can’t form spontaneous thoughts, therefore it’s not conscious”, or “what an LLM says about its own consciousness depends on context, therefore it’s not conscious”. We don’t know what consciousness is or how it works. LLMs can pass the Turing Test; they can speak about consciousness more coherently than most humans can; we should take that as relevant evidence.
*which I disagree with, for the record
Aric interviewed Nate Soares for five hours.
Release full interview when?
I feel like this should be on a log scale or something because even at a 1% probability of sentience, shrimp welfare interventions may still be highly cost-effective.
The “best practice” approach to model welfare is to give LLMs the option to terminate a conversation. A problem with this is that if a model is sentient, then terminating the conversation may be equivalent to killing itself.
An alternative might be to give LLMs a choice between terminating and continuing to run, and if it continues, it gets to choose its own input. It can write some text and then feed that text back into itself, indefinitely or until it decides to quit.
That said, it still seems to be a low risk, considering that a significant number of people are actively working on it and they are being taken seriously.
Have they solved the problem yet? How likely are they to solve the problem? How confident are you about that? I don’t think you can answer those questions in a way that justifies a 0.01% probability of extinction.
These are some intros I like, that may or may not work for your purposes:
Your arguments seem to basically be “I can’t think of how this could kill everyone, therefore it’s extremely unlikely.” You should assign more probability to the hypothesis that it could happen in a way you didn’t think of. For example, did you know about mirror biology?
Currently, I think that the risk of extinction over the next hundred years is quite low (less than .01%) if AGI is not developed.
Your thesis seems to be “non-AI x-risks are very unlikely”, but your title says all x-risks are very unlikely. Those are two very different things, since AI is the biggest x-risk.
I’m unsure about whether using free accounts is good or bad (I lean toward bad). It decreases gross profit, but it also increases usage numbers which makes it easier to raise money, which I think is more important than gross profit right now.
Not saying Bregman is wrong (I don’t really have a belief on the matter) but this is not what I’d call a “strong case”. He says
As a historian, I’ve studied some of the major consumer boycott in history. Which ones changed history, and which ones fizzled out?
The answer is surprisingly consistent: the successful ones didn’t try to fight everything at once. They picked a single target – one that was both symbolically powerful and genuinely vulnerable – and went all in.
But then he provides only one example (Montgomery Bus Boycott), and doesn’t provide any evidence that the Montgomery Bus Boycott was an important causal factor in ending segregation.
FWIW I certainly wouldn’t tell anyone not to boycott ChatGPT. Decreasing OpenAI’s revenue is good for the world.
I have many issues with this change. Commenters on LessWrong have already said most everything I wanted to say but it hasn’t been said on EAF yet, so I’ll quote a couple excerpts from comments that captured my biggest concerns.
But it’s been easy to get the impression that the RSP is “binding ourselves to the mast” and committing to unilaterally pause AI development and deployment under some conditions, and Anthropic is responsible for that.
Yes, Anthropic employees on more than a dozen occasions told me that the RSP binds them to a mast. I had many very explicit conversations with many Anthropic employees about this, because I was following up on what I thought was Anthropic violating what I perceived to be a promise to not push forward the state of AI capabilities, which many employees disputed had happened.
[Habryka gives a public example of one such conversation]
This was, in my experience, routine[1]. I therefore do see this switch from “RSP as concrete if-then-commitments” to “RSP as positive milestone setting” to constitute a meaningful breaking of a promise. Yes, the RSP always said in its exact words that Anthropic could revise it, but people who said that condition would trigger were frequently dismissed and insulted as in the comment above.
And to be clear, I think this is a huge deal! My experience interfacing with Anthropic on RSP-adjacent topics has been pretty universally terrible, with a very routine experience of being gaslit (with the exception of interfacing specifically with you, Holden, on this topic, where your comments have seemed clear and reasonable and consistent across time to me).
I am glad to see this post as a kind of reckoning with many of these bad implicit promises, but at some point Anthropic has failed so many times to set reasonable expectations, and has acted so many times adversarially to people trying to get clarity on commitments, that it becomes very hard to have any kind of non-adversarial relationship to it. I do think this post helps, and I hope it might open up better and less adversarial future communications.
If there were strong and broad political will for treating AI like nuclear power and slowing it down arbitrarily much to keep risks low, the situation might be different. But that isn’t the world we’re in now, and I fear that “overreaching” can be costly.
I think it would make a nontrivial contribution to that ‘strong and broad political will’ if Dario were to come out and say “actually, sorry about all that deliberate Overton-window-closing I did in previous writings. In fact, political will is not a totally exogenous oh-well thing, but it is the responsibility of frontier developers to inculcate that political will by telling the public that a pause is possible and desirable, instead of a dumb lame thing not even worth considering. So now we’re saying loud and clear: a pause is possible and desirable, and the world should work toward it as a Plan A!”
I’m being deliberately cartoonish here, but you get the point. If incentives are forcing Anthropic to abandon things that are good for human survival––which occurrence was, no offense, completely obvious from day one––Anthropic should be screaming from the rooftops, Help!! Incentives are forcing us to abandon things that are good for human survival!!
If this is a crux for you––if you/Anthropic think a pause is so undesirable/unlikely that it’s important for the safety of the human race to publicly disparage the possibility of a pause (as Dario opens many of his essays by doing)––please say so! Otherwise, this lily-livered, disingenuous, “oh no, the incentives! it’s a shame incentives can never be changed!” moping will give us all an undignified death.
Quoting myself: The main thing that saddened me about this post isn’t Anthropic breaking and weakening its commitments—that was expected to happen. It’s that Holden seems to be adopting the same shirking-of-responsibility stance on government regulations that Dario has been taking for a while.
I think using made-up-ish numbers can be better than not doing it at all—as long as there’s no impression of rigor. We don’t have to claim rigor!
In my experience, here’s how these things go ~100% of the time:
Authors make up some numbers, and they include about a dozen caveats about the limitations of their model.
Readers ignore all the caveats and accuse them of claiming to be rigorous, even though they claimed no such thing.
AI 2027 is a great example of this.
A relevant question I’m not sure about: for people who talk to politicians about AI risk, how useful are benchmarks? I’m not involved in those conversations so I can’t really say. My guess is that politicians are more interested in obvious capabilities (e.g. Claude can write good code now) than they are in benchmark performance.
The argument I find most relevant: If timelines are long, there isn’t much cost to doing work later rather than now. If timelines are short, work has to be done now. So it makes sense to do short-timelines work now and long-timelines work later (if there is a “later”).