I work on AI alignment. Right now, I’m using ideas from decision theory to design and train safer artificial agents.
I also do work in ethics, focusing on the moral importance of future generations.
You can email me at thornley@mit.edu.
Elliott Thornley (EJT)
Karma: 1,095
I’m not sure but I think maybe I also have a different view than you on what problems are going to be bottlenecks to AI development. e.g. I think there’s a big chance that the world would steam ahead even if we don’t solve any of the current (non-philosophical) problems in alignment (interpretability, shutdownability, reward hacking, etc.).
try to make them “more legible” to others, including AI researchers, key decision makers, and the public
Yes, I agree this is valuable, though I think it’s valuable mainly because it increases the probability that people use future AIs to solve these problems, rather than because it will make people slow down AI development or try very hard to solve them pre-TAI.
I don’t think philosophical difficulty is that much of an increase to the difficulty of alignment, mainly because I think that AI developers should (and likely will) aim to make AIs corrigible assistants rather than agents with their own philosophical views that they try to impose on the world. And I think it’s fairly likely that we can use these assistants (if we succeed in getting them and aren’t disempowered by a misaligned AI instead) to help a lot with these hard philosophical questions.
I didn’t meant to imply that Wei Dai was overrating the problems’ importance. I agree they’re very important! I was making the case that they’re also very intractable.
If I thought solving these problems pre-TAI would be a big increase to the EV of the future, I’d take their difficulty to be a(nother) reason to slow down AI development. But I think I’m more optimistic than you and Wei Dai about waiting until we have smart AIs to help us on these problems.
I’m a philosopher who’s switched to working on AI safety full-time. I also know there are at least a few philosophers at Anthropic working on alignment.
With regards to your Problems in AI Alignment that philosophers could potentially contribute to:
I agree that many of these questions are important and that more people should work on them.
But a fair amount of them are discussed in conventional academic philosophy, e.g.:
How to resolve standard debates in decision theory?
Infinite/multiversal/astronomical ethics
Fair distribution of benefits
What is the nature of philosophy?
What constitutes correct philosophical reasoning?
How should an AI aggregate preferences between its users?
What is the nature of normativity?
And these are all difficult, controversial questions.
For each question, you have to read and deeply think about at least 10 papers (and likely many more) to get a good understanding of the question and its current array of candidate answers.
Any attempt to resolve the question would have to grapple with a large number of considerations and points that have previously been made in relation to the question.
Probably, you need to write something at least book-length.
(And it’s very hard to get people to read book-length things.)
In trying to do this, you probably don’t find any answer that you’re really confident in.
I think most philosophers’ view on the questions they study is: ‘It’s really hard. Here’s my best guess.’
Or if they’re confident of something, it’ll be a small point within existing debates (e.g. ‘This particular variant of this view is subject to this fatal objection.’).
And even if you do find an answer you’re confident in, you’ll have a very hard time convincing other philosophers of that answer.
They’ll bring up some point that you hadn’t thought of.
Or they’ll differ from you in their bedrock intuitions, and it’ll be hard for either of you to see any way to argue the other out of their bedrock intuition.
In some cases—like population ethics and decision theory—we have proofs that every possible answer will have some unsavory implication. You have to pick your poison, and different philosophers will make different picks.
And on inductive grounds, I suspect that many other philosophical questions also have no poison-free answers.
Derek Parfit is a good example here.
He spent decades working on On What Matters, trying to settle the questions of ethics and meta-ethics.
He really tried to get other philosophers to agree with him.
But very few do. The general consensus in philosophy is that it’s not a very convincing book.
And I think a large part of the problem is a difference of bedrock intuitions. For example, Bernard Williams simply ‘didn’t have the concept of a normative reason,’ and there was nothing that Parfit could do to change that.
It also seems like there’s not much of an appetite among AI researchers for this kind of work.
If there were, we might see more discussions of On What Matters, or any of the other existing works on these questions.
When I decided to start working on AI, I seriously considered working on the kinds of questions you list. But due to the points above, I chose to do my current work instead.
Makes sense! Unfortunately any x-risk cost-effectiveness calculation has to be a little vibes-based because one of the factors is ‘By how much would this intervention reduce x-risk?’, and there’s little evidence to guide these estimates.
Whether longtermism is a crux will depend on what we mean by ‘long,’ but I think concern for future people is a crux for x-risk reduction. If future people don’t matter, then working on global health or animal welfare is the more effective way to improve the world. The more optimistic of the calculations that Carl and I do suggest that, by funding x-risk reduction, we can save a present person’s life for about $9,000 in expectation. But we could save about 2 present people if we spent that money on malaria prevention, or we could mitigate the suffering of about 12.6 million shrimp if we donated to SWP.
Oops yes, fundamentals between my and Bruce’s cases are very similar. Should have read Bruce’s comment!
The claim we’re discussing—about the possibility of small steps of various kinds—sounds kinda like a claim that gets called ‘Finite Fine-Grainedness’/‘Small Steps’ in the population axiology literature. It seems hard to convincingly argue for, so in this paper I present a problem for lexical views that doesn’t depend on it. I sort of gestured at it above with the point about risk without making it super precise. The one-line summary is that expected welfare levels are finitely fine-grained.
Oh yep nice point, though note that—e.g. - there are uncountably many reals between 1,000,000 and 1,000,001 and yet it still seems correct (at least talking loosely) to say that 1,000,001 is only a tiny bit bigger than 1,000,000.
But in any case, we can modify the argument to say that S* feels only a tiny bit worse than S. Or instead we can modify it so that S is degrees celsius of a fire that causes suffering that just about can be outweighed, and S* is degrees celsius of a fire that causes suffering that just about can’t be outweighed.
Also you might be interested in this paper from Andreas Mogensen which discusses a similar idea.
Nice post! Here’s an argument that extreme suffering can always be outweighed.
Suppose you have a choice between:
(S+G): The most intense suffering S that can be outweighed, plus a population that’s good enough to outweigh it G, so that S+G is good overall: better than an empty population.
(S*+nG): The least intense suffering S* that can’t be outweighed, plus a population that’s n times better than the good population G.
If extreme suffering can’t be outweighed, we’re required to choose S+G over S*+nG, no matter how big n is. But that seems implausible. S* is only a tiny bit worse than S, and n could be enormous. To make the implication seem more implausible, we can imagine that the improvement nG comes about by extending the lives of an enormous number of people who died early in G, or by removing (non-extreme) suffering from the lives of an enormous number of people who suffer intensely (but non-extremely) in G.
We can also make things more difficult by introducing risk into the case (in this sort of way). Suppose now that the choice is between:
(S+G): The most intense suffering S that can be outweighed, plus a population that’s good enough to outweigh it G, so that S+G is good overall: better than an empty population.
(Risky S*+nG): With probability , the most intense suffering S that can be outweighed. With probability , the least intense suffering S* that can’t be outweighed. Plus (with certainty) a population that’s n times better than the good population G.
We’ve amended the case so that the move from S+G to Risky S*+nG now involves just a increase in the probability of a tiny increase in suffering (from S to S*). As before, the move also improves the lives of those in the good population G by as much as you like. Plausibly, each increase (for very small ) in the probability of getting S* instead of S (together with an n increase in the quality of G, for very large n) is an improvement. Then with Transitivity, we get the result that S*+nG is better than S+G, and therefore that extreme suffering can always be outweighed.
I think the view that extreme suffering can’t always be outweighed has some counterintuitive prudential implications too. It implies that basically we should never think about how happy our choices would make us. Almost always, we should think only about how to minimize our expected quantities of extreme suffering. Even when we’re—e.g. - choosing between chocolate and vanilla at the ice cream shop, we should first determine which choice minimizes our expected quantity of extreme suffering. Only if we conclude that these quantities are exactly the same should we even consider which of chocolate and vanilla tastes nicer. That seems counterintuitive to me.
Note also that you can accept outweighability and still believe that extreme suffering is really bad. You could—e.g. - think that 1 second of a cluster headache can only be outweighed by trillions upon trillions of years of bliss. That would give you all the same practical implications without the theoretical trouble.
Nice point, but I think it comes at a serious cost.
To see how, consider a different case. In X, ten billion people live awful lives. In Y, those same ten billion people live wonderful lives. Clearly, Y is much better than X.
Now consider instead Y* which is exactly the same as Y except that we also add one extra person, also with a wonderful life. As before, Y* is much better than X for the original ten billion people. If we say that the value of adding the extra person is undefined and that this undefined value renders the value of the whole change from X to Y* undefined, we get the implausible result that Y* is not better than X. Given plausible principles linking betterness and moral requirements, we get the result that we’re permitted to choose X over Y*. That seems very implausible, and so it counts against the claim that adding people results in undefined comparisons.
Thanks!
Wait, all the LLMs get 90+ on ARC? I thought LLMs were supposed to do badly on ARC.
You should read the post! Section 4.1.1 makes the move that you suggest (rescuing PAVs by de-emphasising axiology). Section 5 then presents arguments against PAVs that don’t appeal to axiology.
I think my objections still work if we ‘go anonymous’ and remove direct information about personal identity across different options. We just need to add some extra detail. Let the new version of One-Shot Non-Identity be as follows. You have a choice between: (1) combining some pair of gametes A, which will eventually result in the existence of a person with welfare 1, and (2) combining some other pair of gametes B, which will eventually result in the existence of a person with welfare 100.
The new version of Expanded Non-Identity is then the same as the above, except it also has available option (3): combine the pair of gametes A and the pair of gametes B, which will eventually result in the existence of two people each with welfare 10.
Narrow views claim that each option is permissible in One-Shot Non-Identity. What should they say about Expanded Non-Identity? The same trilemma applies. It seems implausible to say that (1) is permissible, because (3) looks better. It seems implausible to say that (3) is permissible, because (2) looks better. And if only (2) is permissible, then narrow views imply the implausible-seeming Losers Can Dislodge Winners.
Now consider wide views and Two-Shot Non-Identity, again redescribed in terms of combining pairs of gametes A and B. You first choose whether to combine pair A (which would eventually result in the existence of a person with welfare 1), and then later choose whether to combine pair B (which would eventually result in the existence of a person with welfare 100). Suppose that you know your predicament in advance, and suppose that you choose to combine pair A. Then (your view implies) you’re required to combine pair B, even if that choice occurs many decades later, and even though you wouldn’t be required to combine pair B if you hadn’t (many decades earlier) chose to combine pair A. Now consider a slightly different case: you first choose whether to combine pair C (which would eventually result in the existence of a person with welfare 101), then later choose whether to combine pair B. Suppose that you know your predicament in advance, and suppose that you decline to combine pair C. Many decades later, you face the choice of whether to combine pair B. Your view seems to imply that you’re not permitted to do so. There are thus cases where (all else being equal) you’re not even permitted to create a person who would enjoy a wonderful life.
Here’s my understanding of the dialectic here:
Me: Some wide views make the permissibility of pulling both levers depend on whether the levers are lashed together. That seems implausible. It shouldn’t matter whether we can pull the levers one after the other.
Interlocutor: But lever-lashing doesn’t just affect whether we can pull the levers one after the other. It also affects what options are available. In particular, lever-lashing removes the option to create both Amy and Bobby, and removes the option to create neither Amy nor Bobby. So if a wide view has the permissibility of pulling both levers depend on lever-lashing, it can point to these facts to justify its change in verdicts. These views can say: it’s permissible to create just Amy when the levers aren’t lashed because the other options are on the table; it’s wrong to create just Amy when the levers are lashed because the other options are off the table.
Me: (Side note: this explanation doesn’t seem particularly satisfying. Why does the presence or absence of these other options affect the permissibility of creating just Amy?). If that’s the explanation, then the resulting wide view will say that creating just Amy is permissible in the four-button case. That’s against the spirit of wide PAVs, so wide views won’t want to appeal to this explanation to justfiy their change in verdicts given lever-lashing. So absent some other explanation of some wide views’ change in verdicts occasioned by lever-lashing, this implausible-seeming change in verdicts remains unexplained, and so counts against these views.
Thanks! I’d like to think more at some point about Dasgupta’s approach plus resolute choice.
In Parfit’s case, we have a good explanation for why you’re rationally required to bind yourself: doing so is best for you.
Perhaps you’re morally required to bind yourself in Two-Shot Non-Identity, but why? Binding yourself isn’t better for Amy. And if it’s better for Bobby, it seems that can only be because existing is better for Bobby than not-existing, and then there’s pressure to conclude that we’re required to create Bobby in Just Bobby, contrary to the claims of PAVs.
And suppose that (for whatever reason) you can’t bind yourself in Two-Shot Non-Identity, so that the choice to create Bobby (having previously created Amy) remains open. In that case, it seems like our wide view must again make permissibility depend on lever-lashing or past choices. If the view says that you’re required to create Bobby (having previously created Amy), permissibility depends on past choices. If the view says that you’re permitted to decline to create Bobby (having previously created Amy), permissibility depends on lever-lashing (since, on wide views, you wouldn’t be permitted to pull both levers if they were lashed together).
I said a little in another thread. If we get aligned AI, I think it’ll likely be a corrigible assistant that doesn’t have its own philosophical views that it wants to act on. And then we can use these assistants to help us solve philosophical problems. I’m imagining in particular that these AIs could be very good at mapping logical space, tracing all the implications of various views, etc. So you could ask a question and receive a response like: ‘Here are the different views on this question. Here’s why they’re mutually exclusive and jointly exhaustive. Here are all the most serious objections to each view. Here are all the responses to those objections. Here are all the objections to those responses,’ and so on. That would be a huge boost to philosophical progress. Progress has been slow so far because human philosophers take entire lifetimes just to fill in one small part of this enormous map, and because humans make errors so later philosophers can’t even trust that small filled-in part, and because verification in philosophy isn’t much quicker than generation.