Paul_Christiano

Karma: 3,572

Paul_Christiano Apr 24, 2023, 8:25 PM
4 points
2 ∶ 4
in reply to: Greg_Colbourn ⏸️ ’s comment on: The Cruel Trade-Off Between AI Misuse and AI X-risk Concerns
I’m pushing back against the framing: “this is a suicide race with no benefit from winning.”
If there is a 10% chance of AI takeover, then there is a real and potentially huge benefit from winning the race. But we still should not be OK with someone unilaterally taking that risk.
I agree that AI developers should have to prove that the systems they build are reasonably safe. I don’t think 100% is a reasonable ask, but 90% or 99% seem pretty safe (i.e. robustly reasonable asks).
(Edited to complete cutoff sentence and clarify “safe.”)

Paul_Christiano Apr 23, 2023, 6:57 PM
21 points
6 ∶ 3
in reply to: Greg_Colbourn ⏸️ ’s comment on: The Cruel Trade-Off Between AI Misuse and AI X-risk Concerns
I’ve seen this a few times but I’m skeptical about taking this rhetorical approach.
I think a large fraction of AI risk comes from worlds where the ex ante probability of catastrophe is more like 50% than 100%. And in many of those worlds, the counterfactual impact of individual developers move faster is several times smaller (since someone else is likely to kill us all in the bad 50% of worlds). On top of that, reasonable people might disagree about probabilities and think 10% in a case where I think 50%.
So putting that together they may conclude that racing faster increases the risk of doom by 0.03% for every 1% that it increases your share of the future (whether measured in profit, or reduced opportunity for misuse of frontier systems). And that’s just not going to be compelling.
I think you will have an extremely hard time convincing people that the race is obviously suicidal. I know some folks are confident about this, but I don’t really find that position credible today and I’ve spent a very long time thinking about the problem and engaging with pessimistic people. Maybe it will become obvious tomorrow, and maybe it’s OK for some people to be betting their chips on that, but I don’t want to get lumped in with them (because I think their political position is going to become increasingly untenable over time).
On the flip side, I don’t think it’s controversial to say: “If the probability of AI takeover is 10%, AI developers need to stop racing.”
It’s a tiny bit unclear what that means, so to be a bit more precise: “If people didn’t stop AI development until things look significantly more dangerous than they do today, then then the probability of takeover would be more than 10%.” I don’t think that’s true today, but will likely become true.

Paul_Christiano Apr 9, 2023, 7:20 PM
12 points
4 ∶ 0
in reply to: Anthony DiGiovanni’s comment on: GPTs are Predictors, not Imitators
As an analogy: consider a variant of rock paper scissors where you get to see your opponent’s move in advance—but it’s encrypted with RSA. In some sense this game is much harder than proving Fermat’s last theorem, since playing optimally requires breaking the encryption scheme. But if you train a policy and find that it wins 33% of the time at encrypted rock paper scissors, it’s not super meaningful or interesting to say that the task is super hard, and in the relevant intuitive sense it’s an easier task than proving Fermat’s last theorem.

Paul_Christiano Apr 8, 2023, 8:43 PM
12 points
4 ∶ 0
in reply to: Paul_Christiano’s comment on: GPTs are Predictors, not Imitators
Smaller notes:
- The conditional GAN task (given some text, complete it in a way that looks human-like) is just even harder than the autoregressive task, so I’m not sure I’d stick with that analogy.
- I think that >50% of the time when people talk about “imitation” they mean autoregressive models; GANs and IRL are still less common than behavioral cloning. (Not sure about that.)
- I agree that “figure out who to simulate, then simulate them” is probably a bad description of the cognition GPT does, even if a lot of its cognitive ability comes from copying human cognitive processes.

Paul_Christiano Apr 8, 2023, 8:38 PM
79 points
22 ∶ 1
on: GPTs are Predictors, not Imitators
I agree that it’s best to think of GPT as a predictor, to expect it to think in ways very unlike humans, and to expect it to become much smarter than a human in the limit.
That said, there’s an important further question that isn’t determined by the loss function alone—does the model do its most useful cognition in order to predict what a human would say, or via predicting what a human would say?
To illustrate, we can imagine asking the model to either (i) predict the outcome of a news story, (ii) predict a human thinking step-by-step about what will happen next in a news story. To the extent that (ii) is smarter than (i), it indicates that some significant part of the model’s cognitive ability is causally downstream of “predict what a human would say next,” rather than being causally upstream of it. The model has learned to copy useful cognitive steps performed by humans, which produce correct conclusions when executed by the model for the same reasons they produce correct conclusions when executed by humans.
(In fact (i) is smarter than (ii) in some ways, because the model has a lot of tacit knowledge about news stories that humans lack, but (ii) is smarter than (i) in other ways, and in general having models imitate human cognitive steps seems like the most useful way to apply them to most economically relevant tasks.)
Of course in the limit it’s overdetermined that the model will be smart in order to predict what a human would say, and will have no use for copying along with the human’s steps except insofar as this gives it (a tiny bit of) additional compute. But I would expect to AI to be transformative well before approaching that limit, so that this will remain an empirical question.
GPT-4 is still not as smart as a human in many ways, but it’s naked mathematical truth that the task GPTs are being trained on is harder than being an actual human.
I don’t think this is totally meaningful. Getting perfect loss on the task of being GPT-4 is obviously much harder than being a human, and so gradient descent on its loss could produce wildly superhuman systems. But:
- Given that you can just keep doing better and better essentially indefinitely, and that GPT is not anywhere near the upper limit, talking about the difficulty of the task isn’t super meaningful.
- To the extent that GPT-4 and humans are both optimizing a loss function, getting a nearly perfect genetic fitness is probably harder than getting a nearly perfect log loss.
- Getting a GPT-4 level loss on GPT-4′s task is probably much easier than getting a human-level loss on the human task.
What links here?

Paul_Christiano Mar 31, 2023, 7:38 PM
28 points
5 ∶ 1
in reply to: Dan H’s comment on: Critiques of prominent AI safety labs: Redwood Research
The better reference class is adversarially mined examples for text models. Meta and other researchers were working on a similar projects before Redwood started doing that line of research. https://github.com/facebookresearch/anli is an example
I agree that’s a good reference class. I don’t think Redwood’s project had identical goals, and would strongly disagree with someone saying it’s duplicative. But other work is certainly also relevant, and ex post I would agree that other work in the reference class is comparably helpful for alignment
Reader: evaluate your model’s consistency for what counts as alignment research—does this mean non-x-risk-pilled Meta researchers do some alignment research, if we believe RR project constituted exciting alignment research too?
Of course! I’m a bit unusual amongst the EA crowd in how enthusiastic I am about “normal” robustness research, but I’m similarly unusual amongst the EA crowd in how enthusiastic I am this proposed research direction for Redwood, and I suspect those things will typically go together.
Separately, I haven’t seen empirical demonstrations that pursuing this line of research can have limited capabilities externalities or result in differential technological progress.
I’m still not convinced by this perspective. I would frame the situation as:
- There’s a task we really want future people to be good at—finding places where models behave in obviously-undesirable ways, and understanding the limitations of such evaluations and the consequences of training on adversarial inputs.
- That task isn’t obviously improving automatically with model capabilities, it seems like something that requires knowledge and individual+institutional expertise.
- So maybe we should practice a lot to get better at that task, sharing what we learn and building a larger community of researchers and engineers with relevant experience.
Your objection sounds like: “That may be true but there’s not a lot of evidence that this doesn’t also make models more capable, which would be bad.” And I don’t find that very persuasive—I don’t think there is such a strong default presumption that generic research accelerates capabilities enough to be a meaningful cost.
On the question of what generates differential technological progress, I think I’m comparably skeptical of all of the evidence on offer for claims of the form “doing research on X leads to differential progress on Y,” and the best guide we have (both in alignment and in normal academic research!) is basically common-sense arguments along the lines of “investigating and practicing doing X tends to make you better at doing X.”

Paul_Christiano Feb 22, 2023, 5:28 PM
7 points
2 ∶ 0
in reply to: DavidW’s comment on: Deceptive Alignment is <1% Likely by Default
I’ve argued about this point with Evan a few times but still don’t quite understand his take. I’d be interested in more back and and forth. My most basic objection is that the fine-tuning objective is also extremely simple—produce actions that will be rated highly, or even just produce outputs that get a low loss. If you have a picture of the training process, then all of these are just very simple things to specify, trivial compared to other differences in complexity between deceptive alignment and proxy alignment. (And if you don’t yet have such a picture, then deceptive alignment also won’t yield good performance.)

Paul_Christiano Feb 22, 2023, 5:21 PM
5 points
2 ∶ 0
in reply to: DavidW’s comment on: Deceptive Alignment is <1% Likely by Default
Yes, I think that’s how people have used the terms historically. I think it’s also generally good usage—the specific thing you talk about in the post is important and needs its own name.
Unfortunately I think it is extremely often misinterpreted and there is some chance we should switch to a term like “instrumental alignment” instead to avoid the general confusion with deception more broadly.

Paul_Christiano Feb 22, 2023, 7:07 AM
7 points
1 ∶ 0
in reply to: DavidW’s comment on: Deceptive Alignment is <1% Likely by Default
Because they lead to good performance on the pre-training objective (via deceptive alignment). I think a similarly big leap is needed to develop deceptive alignment during fine-tuning (rather than optimization directly for the loss). In both cases the deceptively aligned behavior is not cognitively similar to the intended behavior, but is plausibly simpler (with similar simplicity gaps in each case).
What links here?
- Deceptive Alignment is <1% Likely by Default by DavidW (LessWrong; Feb 21, 2023, 3:09 PM; 89 points)
- Deceptive Alignment is <1% Likely by Default by DavidW (Feb 21, 2023, 3:07 PM; 54 points)

Paul_Christiano Feb 22, 2023, 7:05 AM
6 points
0 ∶ 0
in reply to: David Johnston’s comment on: Deceptive Alignment is <1% Likely by Default
I don’t know how common each view is. My guess would be that in the old days this was the more common view, but there’s been a lot more discussion of deceptive alignment recently on LW.
I don’t find the argument about “take actions with effects in the real world” --> “deceptive alignment,” and my current guess is that most people would also back off from that style of argument if they thought about the issues more thoroughly. Mostly though it seems like this will just get settled by the empirics.

Paul_Christiano Feb 22, 2023, 12:38 AM
10 points
2 ∶ 0
on: Deceptive Alignment is <1% Likely by Default
Models that are only pre-trained almost certainly don’t have consequentialist goals beyond the trivial next token prediction.
If a model is deceptively aligned after fine-tuning, it seems most likely to me that it’s because it was deceptively aligned during pre-training.
“Predict tokens well” and “Predict fine-tuning tokens well” seem like very similar inner objectives, so if you get the first one it seems like it will move quickly to the second one. Moving to the instrumental reasoning to do well at fine-tuning time seems radically harder. And generally it’s quite hard for me to see real stories about why deceptive alignment would be significantly more likely at the second step than the first.
(I haven’t read your whole post yet, but I may share many of your objections to deceptive alignment first emerging during fine-tuning.)
I’ve gotten the vague vibe that people expect deceptive alignment to emerge during fine-tuning (and perhaps especially RL fine-tuning?) but I don’t fully understand the alternative view. I think that “deceptively aligned during pre-training” is closer to e.g. Eliezer’s historical views.
What links here?
- Deceptive Alignment is <1% Likely by Default by DavidW (LessWrong; Feb 21, 2023, 3:09 PM; 89 points)
- Deceptive Alignment is <1% Likely by Default by DavidW (Feb 21, 2023, 3:07 PM; 54 points)

Paul_Christiano Feb 22, 2023, 12:34 AM
6 points
3 ∶ 0
on: Deceptive Alignment is <1% Likely by Default
There is another form of deceptive alignment in which agents become more manipulative over time due to problems with training data and eventually optimize for reward, or something similar, directly.
I think “deceptive alignment” refers only to situations where the model gets a high reward at training for instrumental reasons. This is a source of a lot of confusion (and should perhaps be called “instrumental alignment”) but worth trying to be clear about.
I might be misunderstanding what you are saying here. I think the post you link doesn’t use the term “deceptive alignment” at all so am a bit confused about the cite. (It uses the term “playing the training game” for all models that understand what is happening in training and are deliberately trying to get a low loss, which does include both deceptively aligned models and models that intrinsically value reward or something sufficiently robustly correlated.)

Paul_Christiano Feb 20, 2023, 7:26 AM
7 points
3 ∶ 0
in reply to: splinter’s comment on: Should ChatGPT make us downweight our belief in the consciousness of non-human animals?
Rather, I don’t think that GPUs performing parallel searches through a probabilistic word space by themselves are likely to support consciousness.
This seems like the crux. It feels like a big neural network run on a GPU, trained to predict the next word, could definitely be conscious. So to me this is just a question about the particular weights of large language models, not something that can be established a priori based on architecture.

Paul_Christiano Feb 19, 2023, 6:51 AM
35 points
13 ∶ 2
on: Should ChatGPT make us downweight our belief in the consciousness of non-human animals?
It seems reasonable to guess that modern language models aren’t conscious in any morally relevant sense. But it seems odd to use that as the basis for a reductio of arguments about consciousness, given that we know nothing about the consciousness of language models.
Put differently: if a line of reasoning would suggest that language models are conscious, then I feel like the main update should be about consciousness of language models rather than about the validity of the line of reasoning. If you think that e.g. fish are conscious based on analysis of their behavior rather than evolutionary analogies with humans, then I think you should apply the same reasoning to ML systems.
I don’t think that biological brains are plausibly necessary for consciousness. It seems extremely likely to me that a big neural network can in principle be conscious without adding any of these bells or whistles, and it seems clear that SGD could find conscious models.
I don’t think the fact that language models say untrue things show they have no representation of the world (in fact for a pre-trained model that would be a clearly absurd inference—they are trained to predict what someone else would say and then sample from that distribution, which will of course lead to confidently saying false things when the predicted-speaker can know things the model does not!)
That all said, I think it’s worth noting and emphasizing that existing language models’ statements about their own consciousness are not evidence that they are conscious, and that more generally the relationship between a language model’s inner life and its utterances is completely unlike the relationship between a human’s inner life and their utterances (because they are trained to produce these utterances by mimicking humans, and they would make similar utterances regardless of whether they are conscious). A careful analysis of how models generalize out of distribution, or about surprisingly high accuracy on some kinds of prediction tasks could provide evidence of consciousness, but we don’t have that kind of evidence right now.

Paul_Christiano Dec 27, 2022, 7:31 PM
5 points
0 ∶ 0
in reply to: Tyler Johnston’s comment on: May The Factory Farms Burn
I don’t think those objections to offsetting really apply to demand offsetting. If I paid someone for a high-welfare egg, I shouldn’t think about my action as bringing an unhappy hen into existence and then “offsetting” that by making it better off. And that would be true even if I paid someone for a high-welfare egg, but then swapped my egg with someone else’s normal egg. And by the same token if I pay someone to sell a high-welfare egg on the market labeled as a normal egg, and then buy a normal egg from the market, I haven’t increased the net demand for normal eggs at all and so am not causally responsible for any additional factory-farmed hens.

Paul_Christiano Dec 27, 2022, 7:54 AM
2 points
0 ∶ 2
in reply to: Tyler Johnston’s comment on: May The Factory Farms Burn
The idea is: if I eat an egg and buy a certificate, I don’t increase demand for factory farmed eggs, and if I buy 2 certificates per egg I actually decrease demand. So I’m not offsetting and causing harm to hens, I’m directly decreasing the amount of harm to hens. I think this is OK according to most moral perspectives, though people might find it uncompelling. (Not sure which particular objections you have in mind.)

Paul_Christiano Dec 27, 2022, 12:25 AM
3 points
0 ∶ 2
in reply to: Tyler Johnston’s comment on: May The Factory Farms Burn
I’m particularly interested in demand offsetting for factory farming. I think this form of offsetting makes about as much sense as directly purchasing higher-welfare animal products.

Paul_Christiano Dec 2, 2022, 12:01 AM
8 points
2 ∶ 1
in reply to: RyanCarey’s comment on: SBF Dealbook Thread
The picture I was suggesting is: FTX leaves customer assets in customer accounts, FTX owes and is owed a ton of $ as part of its usual operation as a broker + clearinghouse, Alameda in particular owes them a huge amount of $ and is insolvent (requiring negligence on both risk management and accounting), FTX ends up owing customers $ that FTX can’t pay, FTX starts grabbing customer assets to try to meet withdrawals.
I’m still at maybe 25-50% overall chance that their conduct is similar to a conventional broker+exchange+clearinghouse except for the two points I raised: (i) extremely and probably willfully negligent risk management + accounting, (ii) selling customer assets to raise $ once they were insolvent, rather than owning up to the shortfall. I do think it’s probably worse than that, but (i) is already bad enough that I expect SBF could end up in prison indefinitely.
I’m interested in having more answers, but expect it will all become clear in time and it’s easier to just wait.
(Small point: it was bad for FTX to take FTT as collateral but I think they did so openly from all of their clients, see here.)

Paul_Christiano Dec 1, 2022, 7:05 PM
6 points
0 ∶ 0
in reply to: Lorenzo Buonanno🔸’s comment on: SBF Dealbook Thread
None of this is decision-relevant for me and waiting a few months or years makes complete sense, but alas “interesting” != “decision-relevant.”

Paul_Christiano Dec 1, 2022, 7:03 PM
11 points
2 ∶ 1
in reply to: Habryka [Deactivated]’s comment on: SBF Dealbook Thread
Another issue is the fiat that FTX owed customers. I think FTX did not say that the $ were being held as deposits (e.g. I think the terms of service explicitly say they aren’t). Those numbers in accounts just represent the $ that FTX owes its customers. And if the main problem was a -$8B fiat balance, then it’s less clear any of the expectations about assets and lending matters.
The statement “we never invest customer funds” is still deeply misleading but it’s more clear how SBF could spin this as not being an outright lie: he could ask (i) “How much cash are we owed?” (ii) “How much cash do we owe?” (iii) “How good is collateral for the cash we are owed?” If the first number is larger than the second, and the collateral is good enough, then I think it’s not too surprising if you aren’t even tracking individual customer $ or thinking about where money is coming from, you are just adding up entries in a spreadsheet and saying “we are a net lender to customers, not a net borrower.”
I find this particularly unsurprising if lots of customers have negative fiat balances (which I expect was the case). For a reasonable broker I expect the sum of all the customer fiat balances to be equal the amount of fiat that the broker has on hand. I don’t particularly expect FTX to have more $ on hand than the sum of all positive customer balances. (E.g. I wouldn’t expect my broker to have so much $ on hand.)
But again, I think that the willfully negligent accounting and risk management is a big deal, and the distinction between “customers owe money, with reasonable collateral posted that we could liquidate at any time” and “we’ve bought a bond and have no collateral at all” is entirely about the quality and liquidity of the collateral. So this whole discussion is just about whether there needs to be additional fraud, or whether there’s a reasonable description of the whole thing as willfully negligent risk management.