Paul_Christiano

Karma: 3,572

Paul_Christiano Dec 1, 2022, 7:05 PM
6 points
0 ∶ 0
in reply to: Lorenzo Buonanno🔸’s comment on: SBF Dealbook Thread
None of this is decision-relevant for me and waiting a few months or years makes complete sense, but alas “interesting” != “decision-relevant.”

Paul_Christiano Dec 1, 2022, 7:03 PM
11 points
2 ∶ 1
in reply to: Habryka [Deactivated]’s comment on: SBF Dealbook Thread
Another issue is the fiat that FTX owed customers. I think FTX did not say that the $ were being held as deposits (e.g. I think the terms of service explicitly say they aren’t). Those numbers in accounts just represent the $ that FTX owes its customers. And if the main problem was a -$8B fiat balance, then it’s less clear any of the expectations about assets and lending matters.
The statement “we never invest customer funds” is still deeply misleading but it’s more clear how SBF could spin this as not being an outright lie: he could ask (i) “How much cash are we owed?” (ii) “How much cash do we owe?” (iii) “How good is collateral for the cash we are owed?” If the first number is larger than the second, and the collateral is good enough, then I think it’s not too surprising if you aren’t even tracking individual customer $ or thinking about where money is coming from, you are just adding up entries in a spreadsheet and saying “we are a net lender to customers, not a net borrower.”
I find this particularly unsurprising if lots of customers have negative fiat balances (which I expect was the case). For a reasonable broker I expect the sum of all the customer fiat balances to be equal the amount of fiat that the broker has on hand. I don’t particularly expect FTX to have more $ on hand than the sum of all positive customer balances. (E.g. I wouldn’t expect my broker to have so much $ on hand.)
But again, I think that the willfully negligent accounting and risk management is a big deal, and the distinction between “customers owe money, with reasonable collateral posted that we could liquidate at any time” and “we’ve bought a bond and have no collateral at all” is entirely about the quality and liquidity of the collateral. So this whole discussion is just about whether there needs to be additional fraud, or whether there’s a reasonable description of the whole thing as willfully negligent risk management.

Paul_Christiano Dec 1, 2022, 6:51 PM
6 points
1 ∶ 0
in reply to: Habryka [Deactivated]’s comment on: SBF Dealbook Thread
One important accounting complexity is that there might be a lot of futures contracts that effectively represent borrow/lend without involving physical borrowing or lending. My impression is that FTX offered a lot of futures so this could be significant.
For example, a bunch of people might have had long BTC futures contracts, with Alameda having the short end of that contract. If Alameda is insolvent then it can’t pay its side of the contract; usually FTX (which is both broker and clearinghouse) would make a margin call to ensure that Alameda has enough collateral to pay, but it is clear that FTX was completely irresponsible about Alameda’s collateral.
So it would be nice to know the volume of futures contracts, in addition to the $3B of borrow/lend. CoinGecko currently lists open interest of $2.5B, but I don’t know if that number is accurate or if there are other subtleties, and moreover I could imagine that the number was significantly larger prior to the collapse.
No matter how large the futures volume, this would still leave point 2 above as a complaint. If the FTX clearinghouse was failing, then I think morally they should just admit it. Probably what would happen instead is that they would desperately try to make customers whole including by liquidating other customers whose assets weren’t theirs to liquidate (especially given that if they do so successfully and FTX mostly recovers then they will ultimately be solvent).

Paul_Christiano Dec 1, 2022, 6:34 AM
16 points
4 ∶ 0
in reply to: Habryka [Deactivated]’s comment on: SBF Dealbook Thread
It seems significant if only $3B of customers were earning interest on cash or assets, and the other customers had the option to opt in to lending but explicitly did not. I think all the other customers would have an extremely reasonable expectation of their assets just sitting there. I’m not super convinced by the close reading of the terms of service but it seems like the common-sense case is strong.
I’m interested in understanding that $3B number and any relevant subtleties in the accounting. I feel like if that number had been $15B then this would plausibly just be a failure of risk management, in which case I guess that number is central to the clear-cut fraud vs willful negligence question. The $3B estimate seems plausible but all I’ve seen is an out of context screenshot which is not great.

Paul_Christiano Dec 1, 2022, 1:40 AM
9 points
3 ∶ 0
in reply to: RyanCarey’s comment on: SBF Dealbook Thread
Doesn’t FTX pay interest on deposits and prominently offer margin loans? Do you have a citation for the claim that the terms of service excluded the prospect of lending? (All I’ve seen are some out-of-context screenshots.)
Why do you say “Alameda (FTX trading)”? Aren’t these just separate entities?

Paul_Christiano Dec 1, 2022, 1:35 AM
30 points
9 ∶ 3
in reply to: Habryka [Deactivated]’s comment on: SBF Dealbook Thread
It seems like FTX offered borrowing and lending to its clients, and this was prominently marketed. I don’t think you can call FTX offering margin loans to Alameda a “straightforward case of fraud” if they publicly offer margin loans to all of their clients. (There may be other ways in which their behavior was straightforwardly fraudulent, especially as they were falling apart, but I don’t know.)
In general brokers can get wiped out by risk management failures. I agree this isn’t just an “ordinary bank run,” the bank run was just an exacerbating feature once the damage had already been done.
“Never invest customer deposits” sounds like a misleading tweet. In the conversation with Kelsey, SBF clarifies that he meant that FTX never invests customer deposits, that it is just acting as an exchange that facilitates borrowing and lending between its customers one of whom was Alameda. I think this is probably technically true but misleading.
It seems like there are two distinctive problems:
1. By November 2 Alameda’s collateral looks like it was very bad. Nominally the market value might have exceeded their liabilities, but it was exceptionally illiquid even before considering the fact that a lot of it was FTT and hence correlated with FTX solvency. They were evidently not automatically liquidated as any normal customer would be, and it sounds like their liabilities were not tracked by the normal system at all.
  
  If this was just a huge risk management and accounting failure that would be merely bad, but the beneficiary was a hedge fund mostly owned by SBF whose leadership he had a close relationship with. From the outside it looks quite likely that they knew this was a possibility, but didn’t want to liquidate Alameda after its losses in 2022 (or perhaps deliberately avoided getting clarity about the accounting situation?) because they had a good chance of making money during a market recovery and wanted to take one last gamble at the customer’s expense.
  
  This isn’t technically FTX investing customer’s money, but the risk management is so bad, the joint ownership so extreme, and the organizational boundaries so blurred by bad accounting, that it seems likely to be willful negligence and fair to say that they were effectively investing customer money.
2. I assume FTX also didn’t pay back customers who weren’t participating in asset lending and who weren’t getting paid interest on their deposits. I don’t actually know details here, either about FTX or about finance in general, and they would affect my view about how bad this part was (though point 1 is bad at any rate). My guess is that it this outcome was basically inevitable once the exchange was having liquidity problems (I’d have been surprised if they halted withdrawals for margin accounts while they still had a ton of $ and liquid coins in other accounts) and the question is just how much of a betrayal that is.
  
  For normal brokerage accounts, my vague sense of expectations is that margin accounts would be at risk of going down with the exchange, whereas normal accounts wouldn’t. I don’t know if that expectation is grounded in legal reality. I don’t know if FTX had any such distinction.
  
  I would guess that FTX customers had the choice to opt in to lending, though I wouldn’t be surprised if it was opt-out or if uptake of lending was very large. And legally I don’t know if customers who don’t use margin accounts or who don’t click the “lend asset” box or whatever are supposed to be protected (kind of like more senior creditors). Morally it seems like they have a reasonable expectation of not losing it but I expect this is also not straightforward fraud.
Overall my sense is that this is probably not straightforward fraud, but I don’t know and would appreciate a bit more clarity.

Paul_Christiano Nov 29, 2022, 2:44 AM
52 points
8 ∶ 5
on: Why Giving What We Can recommends using expert-led charitable funds
If someone is strongly considering donating to a charitable fund, I think they should usually instead participate in a donor lottery up to say 5-10% of the annual money moved by that fund. If they win, they can spend more time deciding how to give (whether that means giving to the fund that they were considering, giving to a different fund, changing cause areas, supporting a charity directly, participating in a larger lottery, saving in a donor-advised fund, or doing something altogether different).
I’m curious how you feel about that advice. Obviously some donors won’t be comfortable with the idea of a donor lottery and they can continue to give directly. I personally remain very excited about the idea of donor lotteries and think it would be healthy for the EA community to use more extensively.
For example, I think it would be healthy if funds were accountable to a smaller number of randomly selected donors who had the time to investigate more deeply, rather than spending <10% as much time and being more likely to pick based on a quick skim of fund materials and advertising/social dynamics/etc. And it seems like there’s no way to escape from that regress by having GWWC evaluate evaluators, since then the donor must evaluate GWWC’s evaluations. From this perspective a donor lottery is really like a “free lunch” that’s hard to get in other ways.
Using a fund is similar to using an actively managed investment fund instead of trying to pick individual stocks to invest in: in both cases, you let experts decide what to do with your money. This analogy helps explain the structure of a charitable fund, but it likely understates its benefits.
There is also one major way in which it overstates the benefits: for financial investments it is very valuable to diversify across at least dozens of firms and a few asset classes. Evaluating so many investments would take a huge amount of time, and so even if evaluating individual investments was easier than evaluating funds you’d still probably want to invest in a fund. In contrast, a charitable donor needs to find just one charity that they want to support, and so the case for evaluators really rests on it being easier to evaluate an evaluator than to evaluate a charity.
That comparison is most favorable for organizations like GiveWell, whose main role is to produce reasoning that would clearly be valuable to an individual donor trying to evaluate a charity. But “evaluate funds” vs “evaluate charities” is more apples-to-apples when you are primarily relying on funder judgment, since you could just as well rely on the judgment of people who run the charities they support.
(However the point about charities preferring to engage with fewer big funders is still very relevant and suggests using either a fund or a lottery.)

Paul_Christiano Nov 23, 2022, 9:28 PM
17 points
5 ∶ 4
in reply to: Linda Linsefors’s comment on: EA should blurt
I would prefer more people give through donor lotteries rather than deferring to EA funds or some vague EA vibe. Right now I think EA funds do like $10M / year vs $1M / year through lotteries, and probably in my ideal world the lottery number would be at least 10x higher.
I think that EAs are consistently underrating the value of this kind of decentralization. With all due respect to EA funds I don’t think it’s reasonable to say “thinking more about how to donate wouldn’t help because obviously I should just donate 100% to EA funds.” (That said, I don’t have a take about whether EA funds should shut down. I would have guessed not.)
I think that’s probably the lowest hanging fruit, though it might only help modestly. The effect size depends on how much the problem is lacking diverse sources of money vs lacking diverse sources of attention.

Paul_Christiano Nov 17, 2022, 6:59 PM
24 points
0 ∶ 0
in reply to: Paul_Christiano’s comment on: How Much Leverage Should Altruists Use?
Following up on this. Since May 2020 when these comments were written:
- VT is up 13.5%/year.
- VWO (emerging markets ETF) is up 3.6%/year.
- VMOT is up 7%/year.
Comparing to your predictions:
- You expected VMOT to outperform by about 6%, and suggesting cut that in half to 3% to be conservative, but it underperformed by 6.5%. So that’s another 1.5-3 standard deviations of underperformance.
- You expected VWO to significantly outperform, but it underperformed by about 10%/year.

Paul_Christiano Nov 16, 2022, 5:54 PM
31 points
0 ∶ 0
on: Integrity for consequentialists
I still broadly endorse this post. Here are some ways my views have changed over the last 6 years:
- At the time I wrote the OP I considered consequentialist evaluation the only rubric for judging principles like this, and the only reason we needed anything else was because of the intractability of consequentialist reasoning or moral uncertainty. I’m now more sympathetic to other moral intuitions and norms, and think my previous attempts to shoehorn them into a consequentialist justification involve some motivated cognition and philosophical error.
- That said I’m now more sympathetic to evidential cooperation in large worlds and a bit less confused about decision theory. So overall I’m more convinced by a range of consequentialist arguments for common-sense moral judgments, including the principle expressed in this post. I don’t think this is the most important form of justification, but it does slightly strengthen those intuitions and plays a role when trying to clarify them in weird cases (e.g. when considering our obligations towards AI systems rather than humans).
- I’m more hesitant about retaliation than when I wrote the OP, and am mostly unwilling to “do malicious things that have no direct good consequences for me” except in cases where people have opted in to retaliation for bad behavior (e.g. by agreeing to a contract or putting down a deposit).
- Although I still endorse this post and think that some relevant arguments have become stronger, I’m more sensitive to a bunch of ways it’s complicated and incomplete. Overall I have less conviction about everything in this space. I do still try to just behave with integrity in a straightforward way, and do think that this is an unusually robust ethical conclusion despite acknowledging more uncertainty about it.

Paul_Christiano Nov 15, 2022, 9:54 PM
77 points
10 ∶ 3
on: We must be very clear: fraud in the service of effective altruism is unacceptable
Earlier this year ARC received a grant for $1.25M from the FTX foundation. We now believe that this money morally (if not legally) belongs to FTX customers or creditors, so we intend to return $1.25M to them.
It may not be clear how to do this responsibly for some time depending on how bankruptcy proceedings evolve, and if unexpected revelations change the situation (e.g. if customers and creditors are unexpectedly made whole) then we may change our decision. We’ll post an update here when we have a more concrete picture; in the meantime we will set aside the money and not spend it.
We feel this is a particularly straightforward decision for ARC because we haven’t spent most of the money and have other supporters happy to fill our funding gap. I think the moral question is more complex for organizations that have already spent the money, especially on projects that they wouldn’t have done if not for FTX, and who have less clear prospects for fundraising.
(Also posted on our website.)
What links here?
- Erich_Grunewald 🔸's comment on If you received FTX grant money you should return it by Thurgood (Nov 28, 2022, 9:51 PM; 3 points)

Paul_Christiano Nov 14, 2022, 4:57 PM
93 points
18 ∶ 3
in reply to: Aaron C’s comment on: NY Times on the FTX implosion’s impact on EA
I think the point of most non-profit boards is to ensure that donor funds are used effectively to advance the organization’s charitable mission. If that’s the case, then having donor representation on the board seems appropriate. Why would this represent a conflict of interest? My impression is that this is quite common amongst non-profits and is not considered problematic. (Note that Holden is on ARC’s board.)
I’m also not sure this what the NYT author is objecting to. I think they would be equally unhappy with SBF claiming to have donated a lot, but it secretly went to a DAF he controlled that he could potentially use to have influence later. The problem is more like trying to claim credit for good works despite not having actually given up the influence yet, not a COI issue.
(I don’t think it’s plausible to call “I gave my money to foundation or DAF, and then I make 100% of the calls about how the foundation donates” a COI issue. )
What links here?
- Dancer's comment on CEA/EV + OP + RP should engage an independent investigator to determine whether key figures in EA knew about the (likely) fraud at FTX by Tyrone-Jay Barugh 🔸 (Nov 14, 2022, 10:00 PM; 4 points)

Paul_Christiano Nov 2, 2022, 4:14 PM
4 points
0 ∶ 0
in reply to: Jeff Kaufman 🔸’s comment on: Mildly Against Donor Lotteries
I think that (normal charity) vs (lottery) is a clear improvement for a much wider range of worldviews than (normal charity) vs (defer to GiveWell).
I do agree that “defer to GiveWell” is easier to explain though. Or slightly more precisely: I think it’s easier to explain what GiveWell does well enough that someone can understand why you might think it’s the best option, but harder to explain what GiveWell does in enough detail that someone can verify for themselves that it’s actually better than their alternatives.

Paul_Christiano Nov 2, 2022, 4:09 PM
32 points
6 ∶ 0
on: Mildly Against Donor Lotteries
- I think most people should at least have a best guess about where they would donate if they hadn’t participated in a lottery, and if it’s easy then they should plausibly give a token amount directly (like 10%). This makes “give to lottery” much more comparable to “give directly” since you’ve still spent the time thinking about it, have your head in the game, and can talk from experience about that decision. (Another simple reason to do this is to confirm that actually diminishing returns aren’t a big deal wherever you were planning to donate.)
- If someone really means to ask “where should I donate?” and “not where did you donate?” then it seems like you can just answer that question. Giving 10% to the object level makes that even easier but I feel like I can basically do that even when I don’t give directly to the particular charity I’m discussing. (That said, I probably get asked this question much less than you and by a very different distribution of people.)
- I particularly like donor lotteries because I think they are one of the cleanest “wins” amongst all EA recommendations for small donors (though not one of the largest wins). I think it’s really complicated to assess whether you should defer to EAs that you don’t really know, but that there are great arguments that most donors would be better served by using a lottery. So I consider it one of the most legible ways to clearly demonstrate that EA is just better at donating than other communities (rather than merely having complex methodological disagreements).
- I think that donor lotteries seem better for the health of the community than having a bunch of small donors who have no option other than deferring to a centralized recommender (whose work they often can’t check very thoroughly). It empowers more people to have meaningful influence over how charity is done. “Small donors should just defer to professionals” might be right but I don’t think it’s very legibly right, and it relies on a bunch of trust. In practice if you are donating $10-100k I think that a lot of that has to be social proof rather than having the time to dig into claims enough to reliably tell that GiveWell knows their stuff. And even if everyone defers to an evaluator, the form of accountability seems important and I’d prefer evaluators be accountable to 10x fewer donors with 10x more time. Relatedly, I think this makes it easier in practice to start a competitor and market to a smaller number of large donors based on quality product.
- I do think a significant fraction of people (especially in tech or finance) respond positively to the idea of donor lotteries, and having the option value to talk about the object level or lotteries based on the audience seems good. Again, I think this might differ because we are talking to different audiences in a different way. I also think that I’m more interested in weird stuff than you, so I’m maybe particularly willing to lose out on prospective donors who aren’t willing to spend 30 minutes thinking about a plausible-looking but weird idea, or who can’t handle expected value calculations. And in general I more want EA to be the kind of place that does weird but good stuff.
- I don’t think that “lots of people are thinking about the allocation of EA funds” has made it that much easier to be a small donor. Which of those people am I supposed to defer to? Also, I qualitatively don’t feel like the ratio of (money moved) vs (time spent thinking about allocation) is that much better than it was in 2016. I wouldn’t be surprised if both of those have grown ~4x in parallel.
- I agree some lottery-winners give to the same kinds of things that they would have given to otherwise, but in the two cases where I have the most information it seems like the outcome was very different. (For several of these grants I think that giving smaller amounts would not have been an option.) It’s also worth noting that most small donors give much worse than you, and for many of them it would be great if “spend more time to think” got them up to “defer to GiveWell.”
- I’d guess that your earning to give situation was similar to a small lottery winner but not a large lottery winner, and if you donated somewhere without significant diminishing returns then I think it might have made sense to go to an even larger scale. (Right now I personally feel like it’s pretty plausible that a philanthropist with a $10M budget should do a lottery with a big EA foundation up to a $100M budget, if they’d offer it, though I’d defer to funders and charities about what they think would make for a healthier ecosystem.)

Paul_Christiano Oct 12, 2022, 10:13 PM
32 points
4 ∶ 0
in reply to: Davidmanheim’s comment on: Cultural EA considerations for Nordic folks
The near-exception is donation of significantly appreciated assets. I think you virtually always still lose money, but it can get close.
You can donate up to 30% of your annual income as appreciated assets valued at market price. If I have that option in a relatively high-tax state like CA, then the decision can be between me keeping $1 or the charity getting >$6.
Suppose I pay $1 for an asset, it goes up to $100, then I donate it. I’d avoid paying 20% (federal) + 12.3% (state) + 3.8% (obamacare) = $36 in taxes on the $99 of capital gains. On top of that, I’d get to take a tax deduction of $100 which will save me about 37% (federal)+ 12.3% = $49 on my tax bill. So overall I’m saving $85 in taxes for every $100 donated.
(Not sure if those calculations are wrong or if I’m missing something.)

Paul_Christiano Jul 17, 2022, 6:42 PM
5 points
0 ∶ 0
in reply to: David Johnston’s comment on: How would a language model become goal-seeking?
Here is my story, I’m not sure if this is what you are referring to (it sounds like it probably is).
Any prediction algorithm faces many internal tradeoffs about e.g. what to spend time thinking about and what to store in memory to reference in the future. An algorithm which makes those choices well across many different inputs will tend to do better, and in the limit I expect it to be possible to do better more robustly by making some of those choices in a consequentialist way (i.e. predicting the consequences of different possible options) rather than having all of them baked in by gradient descent or produced by simpler heuristics.
If systems with consequentialist reasoning are able to make better predictions, then gradient descent will tend to select them.
Of course all these lines are blurry. But I think that systems that are “consequentialist” in this sense will eventually tend to exhibit the failure modes we are concerned about, including (eventually) deceptive alignment.
I think making this story more concrete would involve specifying particular examples of consequentialist cognition, describing how they are implemented in a given neural network architecture, and describing the trajectory by which gradient descent learns them on a given dataset. I think specifying these details can be quite involved both because they are likely to involve literally billions of separate pieces of machinery functioning together, and because designing such mechanisms is difficult (which is why we delegate it to SGD). But I do think we can fill them in well enough to verify that this kind of thing can happen in principle (even if we can’t fill them in in a way that is realistic, given that we can’t design performant trillion parameter models by hand).

Paul_Christiano Jul 16, 2022, 8:42 PM
14 points
0 ∶ 0
in reply to: Paul_Christiano’s comment on: How would a language model become goal-seeking?
Some examples of more exotic sources of consequentialism:
- Some consequentialist patterns emerge within a large model and deliberately acquire more control over the behavior of the model such that the overall model behaves in a consequentialist way. These could emerge randomly, or e.g. while a model is explicitly reasoning about a consequentialist (I think this latter example is discussed by Eliezer in the old days though I don’t have a reference handy). They could either emerge within a forward pass, over a period of “cultural accumulation” (e.g. if language models imitate each other’s outputs), or during gradient descent (see gradient hacking).
- An attacker publishes github repositories containing traces of consequentialist behavior (e.g. optimized exploits against the repository in which they are included). They also place triggers in these repositories before the attacks, like stretches of low-temperature model outputs, such that if we train a model on github and then sample autoregressively the model may eventually begin imitating the consequentialist behavior included in these repositories (since long stretches of low-temperature model outputs occur rarely in natural github but occur just before attacks in the attacker’s repositories). This is technically a special case of “#1 imitating consequentialists” but it behaves somewhat strangely since the people training the system weren’t aware of the presence of the consequentialist.
- An attacker selects an input on which existing machinery for planning or prediction within a large model is repurposed for consequentialist behavior. If we have large language models that are “safe” only because they aren’t behaving as consequentialists, this could be a bad situation. (Compromised models could themselves design and deploy similar attacks to recruit still more models; so even random failures at deployment time could spread like a virus without any dedicated attacker. This bleeds into the first failure mode.)
- A language model can in theory run into the same problem as described in what does the universal prior actually look like?, even if it is only reasoning abstractly about how to predict the physical universe (i.e. without actually containing malign consequentialists). Technically this is also a special case of #1 imitating a consequentialist, but again it can be surprising since e.g. the consequentialist wasn’t present at training time and the person deploying the system didn’t realize that the system might imitate a consequentialist.
I find it interesting to think about the kind of dynamics that can occur in the limit of very large models, but I think these dynamics are radically less important than #1-#5 in my original answer (while still not being exhaustive). I think that they are more speculative, will occur later if they do occur, and will likely be solved automatically by solutions to more basic issues. I think it’s conceivable some issues of this flavor will occur in security contexts, but even there I think they likely won’t present an alignment risk per se (rather than just yet another vector for terrible cybersecurity problems) for a very long time.
What links here?
- Training goals for large language models by Johannes Treutlein (LessWrong; Jul 18, 2022, 7:09 AM; 28 points)

Paul_Christiano Jul 16, 2022, 4:55 PM
142 points
0 ∶ 0
on: How would a language model become goal-seeking?
Here are five ways that you could get goal-directed behavior from large language models:
1. They may imitate the behavior of an agent.
2. They may be used to predict which actions would have given consequences, decision-transformer style (“At 8 pm X happened, because at 7 pm ____”).
3. A sufficiently powerful language model is expected to engage in some goal-directed cognition in order to make better predictions, and this may generalize in unpredictable ways.
4. You can fine-tune language models with RL to accomplish a goal, which may end up selecting and emphasizing one of the behaviors above (e.g. the consequentialism of the model is redirected from next-word prediction to reward maximization; or the model shifts into a mode of imitating an agent who would get a particularly high reward). It could also create consequentialist behavior from scratch.
5. An outer loop could use language models to predict the consequences of many different actions and then select actions based on their consequences.
In general #1 is probably the most common ways the largest language models are used right now. It clearly generates goal-directed behavior in the real world, but as long as you imitate someone aligned then it doesn’t pose much safety risk.
#2, #4, and #5 can also generate goal-directed behavior and pose a classic set of risks, even if the vast majority of training compute goes into language model pre-training. We fear that models might be used in this way because it is more productive than #1 alone, especially as your model becomes superhuman. (And indeed we see plenty of examples.)
We haven’t seen concerning examples of #3, but we do expect them at a large enough scale. This is worrying because it could result in deceptive alignment, i.e. models which are pursuing some goal different from next word prediction which decide to continue predicting well because doing so is instrumentally valuable. I think this is significantly more speculative than #2/4/5 (or rather, we are more unsure about when it will occur relative to transformative capabilities, especially if modest precautions are taken). However it is most worrying if it occurs, since it would tend to undermine your ability to validate safety—a deceptively aligned model may also be instrumentally motivated to perform well on validation. It’s also a problem even if you apply your model even to an apparently benign task like next-word prediction (and indeed I’d expect this to be a particularly plausible if you try to do only #1 and avoid #2/4/5 for safety reasons).
The list #1-#5 is not exhaustive, even of the dynamics that we are currently aware of. Moreover, a realistic situation is likely to be much messier (e.g. involving a combination of these dynamics as well as others that are not so succinctly described). But I think these capture many of the important dynamics from a safety perspective, and that it’s a good list to have in mind if thinking concretely about potential risks from large language models.
What links here?

Paul_Christiano Jul 9, 2022, 1:56 AM
6 points
0 ∶ 0
in reply to: anonymous_ea’s comment on: On Deference and Yudkowsky’s AI Risk Estimates
I’m not sure either of the quotes you cited by Eliezer require or suggest ridiculous overconfidence.
If I’ve seen some photos of a tiger in town, and I know a bunch of people in town who got eaten by an animal, and we’ve all seen some apparent tiger-prints near where people got eaten, I may well say “it’s obvious there is a tiger in town eating people.” If people used to think it was a bear, but that belief was formed based on priors when we didn’t yet have any hard evidence about the tiger, I may be frustrated with people who haven’t yet updated. I may say “The only question is how quickly people’s views shift from bear to tiger. Those who haven’t already shifted seem like they are systematically slow on the draw and we should learn from their mistakes.” I don’t think any of those statements imply I think there’s a 99.9% chance that it’s a tiger. It’s more a statement rejecting the reasons why people think there is a bear, and disagreeing with those reasons, and expecting their views to predictably change over time. But I could say all that while still acknowledging some chance that the tiger is a hoax, that there is a new species of animal that’s kind of like a tiger, that the animal we saw in photos is different from the one that’s eating people, or whatever else. The exact smallness of the probability of “actually it wasn’t the tiger after all” is not central to my claim that it’s obvious or that people will come around.
I don’t think it’s central to this point, but I think 99% is a defensible estimate for many-worlds. I would probably go somewhat lower but certainly wouldn’t run victory laps about that or treat it as damning of someone’s character. The above is mostly a bad analogy explaining why I think it’s pretty reasonable to say things like Eliezer did even if your all-things-considered confidence was 99% or even lower.
To get a sense for what Eliezer finds frustrating and intends to critique, you can read If many-worlds had come first (which I find quite obnoxious). I think to the extent that he’s wrong it’s generally by mischaracterizing the alternative position and being obnoxious about it (e.g. misunderstanding the extent to which collapse is proposed as ontologically fundamental rather than an expression of agnosticism or a framework for talking about experiments, and by slightly misunderstanding what “ontologically fundamental collapse” would actually mean). I don’t think it has much to do with overconfidence directly, or speaks to the quality of Eliezer’s reasoning about the physical world, though I think it is a bad recurring theme in Eliezer’s reasoning about and relationships with other humans. And in fairness I do think there are a lot of people who probably deserve Eliezer’s frustration on this point (e.g. who talk about how collapse is an important and poorly-understood phenomenon rather than most likely just being the most boring thing) though I mostly haven’t talked with them and I think they are systematically more mediocre physicists.

Paul_Christiano Jul 6, 2022, 5:40 PM
4 points
0 ∶ 0
in reply to: Habryka [Deactivated]’s comment on: On Deference and Yudkowsky’s AI Risk Estimates
I think my views about takeoff speeds are generally similar to Robin’s though neither Robin nor Eliezer got at all concrete in that discussion so I can’t really say. You can read this essay from 1998 with his “outside-view” guesses, which I suspect are roughly in line with what he’s imagining in the FOOM debate.
I think that doc implies significant probability on a “slow” takeoff of 8, 4, 2… year doublings (more like the industrial revolution), but a broad distribution over dynamics which also puts significant probability on e.g. a relatively fast jump to a 1 month doubling time (more like the agricultural revolution). In either case, over the next few doublings he would by default expect still further acceleration. Overall I think this is basically a sensible model.
(I agree that shorter timelines generally suggest faster takeoff, but I think either Robin or Eliezer’s views about timelines would be consistent with either Robin or Eliezer’s views about takeoff speed.)