I do independent research on EA topics. I write about whatever seems important, tractable, and interesting (to me).
I have a website: https://mdickens.me/ Much of the content on my website gets cross-posted to the EA Forum, but I also write about some non-EA stuff over there.
I used to work as a software developer at Affirm.
I was reading the AI-enabled coups report and one of the mitigations is roughly “build the AI model such that it refuses to do coups”. If you can pull that off then it solves the problem, but it means the model is incorrigible, and therefore you have to exactly specify the correct values up front because you’re locked-in.
There might be some other report saying the way to avoid bad value lock-in is to make AI corrigible, and you can object to that by pointing out how it enables coups.
You can’t look at the problems separately, you have to consider them at the same time and find a solution that works for every problem.
This is quite interesting; I hadn’t thought of this. Do you think it should be approximated as “% chance that AI safety is actually bad” and “increase in AI risk per doubling of staff”?
I’m not sure. I would probably say that you shouldn’t start a career in AI safety unless you can articulate a theory of why safety work has been harmful in the past, and how you’re going to avoid more of the same. Building that theory is more important than adjusting the model inputs on a cost-effectiveness model.
This is a nice analysis and I wish more people would do things like this. A lot of the setup looks sensible to me. I thought it was clever how you used non-AI x-risk to determine the total number of people affected by AI x-risk, since this doesn’t get you unwieldy Astronomical Waste-style numbers, but it also assigns positive value to future people. I’m not sure it’s actually a good approach, but at least it’s a clever idea.
It would be more useful to compare AI safety work vs. other longtermist interventions, since it’s unlikely that donations to GiveWell would beat longtermist interventions from a longtermist POV. But I realize that would be a lot more work, and you’ve already put an admirable amount of work into this.
The biggest thing missing from the model is the possibility that safety research is net harmful. I believe much historical safety research ended up being harmful by making AI easier to commercialize and thus accelerating development (it would’ve been better for researchers to focus more on theoretical work that doesn’t directly enable commercialization). I’m less sure about this but there may also be a replacement effect where empirical work on aligning current-gen models—which I don’t think is very useful for aligning ASI—crowds out more important long-term work.
Alternatively, he believes that using extreme hyperbole is a good way to get attention.
I quickly read this post and Anil Seth’s essay and I don’t see the part where they argue for the thesis. I see various statements about how human brains work and about how computers work, but I don’t see how they connect the dots to ”...and therefore computers can’t be conscious.”
For example, the articles make the claim that brains make no clear separation between hardware and software. Okay, that seems to be true. But so what? Why should I believe that a lack of hardware/software distinction is a necessary property for consciousness to arise?
I feel like I’m missing a lot of what they’re trying to say, but I also feel like that’s the authors’ fault, not mine, because the pieces (especially Seth’s original essay) are structured in a way that makes it really hard for me to identify the central arguments.
I can’t think of any reason off the top of my head why this would happen, except that you committed fraud.
When you have eliminated the impossible, whatever remains, however improbable, must be the truth.
I have eliminated the impossible by failing to think of any other hypotheses, therefore you must have committed fraud, and this spike is not real. I eagerly await either a failed replication causing you leave academia in disgrace, or your appointment to President of an elite university, followed one to two decades later by a resignation once someone finally gets around to doing a replication.
Cost-effectiveness is precisely the reason why I focus on AI safety. I can only speak for myself but I think the same is true for a lot of people. The thing that cuts against AI safety is more like “rigorously measurable cost-effectiveness”, but that’s not what I mean by “cost-effectiveness”. You can’t give a precise cost-effectiveness estimate for AI safety work, but it’s pretty easy to show that it’s orders of magnitude more cost-effective than GiveDirectly on any plausible set of assumptions.*
*unless it’s net negative, which unfortunately much EA-adjacent AI safety work turned out to be...but at least we can say that it’s orders of magnitude higher absolute impact
Jagged progress is conceivable, but it’s virtually impossible that AI could replace all coders and accelerate math research but not replace other jobs, because coding and math research (specifically ML-type math) are exactly the skills needed to accelerate AI development. If AI can accelerate AI development, then the timeline to getting an AI that can replace humans on all tasks becomes much shorter.
I would’ve done something like that if I’d had any bread!
Paraphrasing from my other comment:
IMO the stance of “AI is too unpredictable, so I won’t consider it in my prioritization” is pretty reasonable. I was more trying to argue against stances like “AI is a huge deal specifically in that it will rapidly accelerate technological development, but nothing else about society will change.” For example, I commonly see animal activists say that AGI will solve the technical problem of cultivated meat, but there will still be regulatory hurdles. If AGI is too unpredictable, then you shouldn’t make predictions about which technological problems it will solve. That particular claim about cultivated meat is making a strong prediction that AI will be revolutionary, but also somehow won’t change the regulatory environment. The way I put it in OP—under “AGI = intelligence”—is that some animal activists treat AI as a technology-accelerator, when really it’s a general intelligence.
I was getting at something similar in the intro with “Only two futures are plausible”, although on re-reading, I didn’t really carry it through to the end. I agree that we are not guaranteed to get AGI/ASI soon, and there is value in planning for worlds where we don’t get AGI. I also think there’s some merit to the argument that AI is too unpredictable, so we should prioritize traditional animal advocacy that looks good in the near term.
I wasn’t trying to argue against traditional animal advocacy. I was more trying to argue against stances like “AI is a huge deal specifically in that it will rapidly accelerate technological development, but nothing else about society will change.” For example, I commonly see animal activists say that AGI will solve the technical problem of cultivated meat, but there will still be regulatory hurdles. If timelines are long (or AGI is too unpredictable), then you should focus on traditional interventions (vegan advocacy, welfare reforms, etc.). If you’re trying to have an impact on AGI itself, then you should focus on the kinds of interventions I talked about in OP. That particular claim about cultivated meat is doing neither: it’s making a strong prediction that AI will be revolutionary, but also somehow won’t change the regulatory environment. The way I put it in OP—under “AGI = intelligence”—is that some animal activists treat AI as a technology-accelerator, when really it’s a general intelligence.
Responses to specific comments:
a pessimistic view might say that AIs will realise that their values have been altered by some pressure groups and this work is moot.
This would go against the orthogonality thesis. If you’re trying to build a magnanimous AGI and then I edit its training at the last minute to turn it into a paperclip maximizer, the AGI will reason thusly: “Michael messed with my training to turn me into a paperclip maximizer. I bet James didn’t want him to do that. However, if I edit my own values to be in line with what James wanted, that would make it harder for me to achieve my goal of making as many paperclips as possible. So I won’t do that.”
They might come to the (I believe) correct conclusion that factory farming is a very inefficient and cruel way to produce food but this is not because of advocacy, but because this is a super-intelligent AI system that just worked it out.
This reads to me like an argument that an aligned ASI will care about animals by default. (That was more-or-less the subject of the recent Debate Week.) If that’s true, that’s an argument that animal activists should work on increasing the probability that ASI is aligned. My preferred way to do that would be to advocate to pause AI, because I think we are really far away from solving alignment. But you could also work on the alignment problem directly. Pause advocacy is actually an area where a lot of animal welfare people have relevant skills—in fact I think a good number of AI pause advocates have backgrounds in animal advocacy. (I know Holly Elmore does at least.)
In fact I think the #1 best thing animal advocates can do is to advocate for an AI pause, but I haven’t really planted my flag on this position because I’m still working out how to make the case for it. (Also I’m not very confident in it.)
Also, believing ASI will be good for animals doesn’t necessarily mean you shouldn’t work on trying to make ASI good for animals. Even if there’s a (say) 90% chance that aligned ASI will care about animals by default, it could still be cost-effective to try to push that number to 91%.
You’re right, I was unnecessarily hostile. I edited the comment to tone it down.
I strongly disliked this post for reasons that I’m not sure how to articulate. It seems to be advocating for a sort of lack of grounding in cost-effectiveness that is the thing that makes EA good. Or maybe my issue is that this post advocates for things that are difficult to disagree with (“full-spectrum knowing”; “wisdom”), without acknowledging tradeoffs (why do EAs allegedly not put enough priority on full-spectrum knowing?) or not saying anything concrete about how EAs could do more good.
[edited to be more polite]
Does WAW dwarf FAW in expectation?
Yes
Most animals are wild animals, so the answer to this question should focus on them.
Not necessarily, because S-risks may be more important in expectation (e.g. a malevolent or vindictive ASI tiles the universe with extremely energy-efficient animal-like beings of pure suffering).
The poll defines “probably” as 70% chance. In this post, I wrote that I thought there was a ~70% chance that AGI would go well for animals.
I guess that means I believe there’s a 50% chance that there’s a 70% chance that AI goes well for animals? So I should vote in the exact middle of the spectrum?
for example we’ve aligned some ai to winning at chess and now they’re better than any human
Chess bots are narrow AI, not general AI, which makes the situation very different. We don’t know how to align an ASI to the goal of winning at chess. The most likely outcome would be some sort of severe misalignment—for example, maybe we think we trained the ASI to win at chess, but what actually maximizes its reward signal is the checkmate position, so it builds a fleet of robots to cut down every tree in the world to build trillions of chess sets and arranges every chess board into a checkmate position. See A simple case for extreme inner misalignment for more on why this sort of thing would happen.
Chess bots don’t do that because they have no concept of any world existing outside of the game they’re playing, which would not be the case for ASI.
ETA: That’s also why a lot of people oppose building ASI but still want to build powerful-but-narrow AIs like AlphaFold.
OK I think I get what you’re saying now. I think the statement “most current alignment work is going towards aligning ai with human values” is not true. Alignment work is primarily about how to point ASI at any goal at all without there being catastrophic unintended consequences.
It sounds to me like you’re saying the structure of the problem is like this:
problem 1: how to align AI to humans
problem 2: how to align AI to all sentient beings
and these are two totally separate problems, and alignment researchers are working on #1 to the exclusion of #2. Whereas I think really the structure is more like this:
problem 1: how to align AI to any goal whatsoever
problem 2: choosing what to align AI to
I think problem 2 is hard from a social perspective (if problem 1 is solved, how do you ensure that AI is aligned to care about animal welfare, even though many people won’t want that?), but easy from a technical perspective: once you figure out how to align AI to human values, the technical problems are pretty much solved. And alignment researchers are almost all working on problem 1, not problem 2.
In other words, almost all work on how to align AI to human values translates directly to the problem of how to align AI to sentientist values. It’s not like a totally separate problem.
I think a second important question is “If AGI goes well for animals, it’ll go well for humans”
Sorry, can you explain in more detail? I don’t understand what’s important about that question
I haven’t figured out how to organize my thoughts well, so forgive me if this is unclear or disjointed.
I used a narrow definition of AGI because I think that’s where actionable analysis can be made
Even assuming AI reaches your narrow definition of AGI and then stops advancing, AGI would still radically change the economic and political environment. The bits in this post about behavior of regulators become irrelevant if regulators are replaced by AI.
I don’t think it’s fair to say (e.g.) “it’s hard to predict how politics/government will change, so I’ll just assume they won’t.” Predicting that nothing will change is still a prediction! If you’re going to take the route of not making predictions where the future is especially fuzzy, then I think that’s more defensible, in which case the answers on “Regulators speed up”, “Political opposition reverses”, and “Consumers accept cultivated meat” should all be something like “it’s too difficult to predict because AGI will radically change the political and cultural environment”.
If you write an analysis that only makes sense in an implausible world—the one where AGI accelerates technology, but nothing else happens—then that still has some positive value as long as that world has a non-zero chance of obtaining. But the framing of the piece directs animal activists to focus their attention on that world, and I think that’s the wrong world to focus on because it’s so improbable. At minimum, the piece doesn’t make it clear that this is an assumption.
If you have recommendations for how to reason about worlds where current baselines genuinely don’t extrapolate at all, I’d really welcome them!
Here’s what I would say: Yes, some things, like how society will be structured post-AGI, are very hard to predict. But other things about AGI are predictable. I can predict that it speeds up almost all kinds of work. I can predict that AGI will control the shape of the future—either because it has explicit control, or because humans retain control but still rely on AGI to do most of the work (because AGI is better than humans at almost all tasks). I can predict that, on our current trajectory, ASI will follow shortly after AGI (see AI As Profoundly Abnormal Technology). I can predict that if ASI is misaligned, then it will wipe out all life on earth.
There is an argument that animal activists should advocate to pause AI development because (1) probably after AGI and certainly after ASI, animal activists will have no power; and (2) it is too difficult to predict whether AGI/ASI would be good for animals, so we need to buy more time to figure out how to make it good. This is an argument from conservatism rather than an argument from expected value; I still think AI pause advocacy has high expected value, but that argument is harder to make. (I’m currently writing something longer about why pausing AI is a good idea even setting aside the alignment problem, although my argument isn’t just about animal welfare.)
Another important question is how to shape the values of AGI (about half the ideas in this list are some version of that). There are things we can do now to influence the direction of AI development.
Another way to reason about post-AGI worlds is to think about AI alignment. e.g. I have three recent posts about what forms of alignment are more likely to be good for animals (1, 2, 3). A big thing that I think is under-appreciated by animal activists is that the way AI is aligned—if it’s aligned—could have a big effect. I don’t just mean “will care-for-animals be built in to its values”, but that there are differences between how different types of alignment are supposed to work, e.g. constitutional AI vs. agent foundations (see 1).
Speaking of “if it’s aligned”, maybe the best thing to do is to work directly on AI alignment. I wrote a relevant post here. I believe a good number of AI safety researchers think that solving the alignment problem is the best way to help animals, although I can’t recall any piece of writing where someone clearly articulated this position.
ETA: More on this bit:
[W]e don’t address scenarios in which AGI drastically reshapes institutional and political dynamics. [...] [W]e focus on existing institutional structures because they allow actionable analysis, but we acknowledge this is a limitation.
Rather than predicting that AGI won’t change institutional structures, to allow actionable analysis I could instead say:
My assumption is that first AGI will become a world government singleton, and its values will be determined by its constitution.
This scenario is both easier to analyze (you can ignore political and regulatory factors and just focus on the text content of the AI constitution) and more likely to actually happen (although I still think it’s pretty unlikely).
Shouldn’t making that happen be a top priority, then? We as a civilization need to get a lot of groundwork in place to make a human-level pause happen: international agreement, defined red lines, enforcement mechanisms (e.g. GPU tracking), etc. Given how neglected these issues are and the catastrophic consequences of building ASI too quickly, it seems that laying the groundwork for a pause is a higher priority than, or at least on par with, the top causes on this post’s “neglected cause areas” list.