PhD candidate at Goldsmiths College, University of London. Title: ‘Reasons for Persons, or the Good Successor Problem’. Abstract: AI alignment aims for advanced machine intelligences that preserves and enhances human welfare (appropriately defined). In a narrow sense, this includes not reflecting existing biases in society or destabilising political systems; more broadly, it could also mean not creating conditions that result in the extinction or disempowerment of humanity. This project tries to define a alternative, speculative vision of alignment: one that relaxes the assumption (tacit in some alignment discourse) that humans must indefinitely retain control over the future. My exploration instead aims for some ambitious and catholic notion of value (that is, avoiding maximisers of the squiggle/paperclip/hedonium varieties and fleshing out sources of value that don’t hinge on a biological human subject). I draw upon philosophy (moral realism, population ethics, and decision theory), aesthetic theory (can the human tendency to make and appreciate aesthetic products, something broadly shared across cultures, be generalised to AIs or is it some contingent, and evolutionarily-useful, practice that arose amongst a particular set of primates?) This project has an empirical aspect: to the extent possible, I want to enrich speculations with experiments using LLMs in multi-agent setups.
ukc10014
Linkpost: Epistle to the Successors
Reza Negarestani’s Intelligence & Spirit
There is also this—the Future Claimant’s Representative—it is apparently a phrase from US bankruptcy/tort law that has been applied in an environmental and museumology context by Ian Baucom, a US academic. This is probably out of context for your question, but I’m interested in fleshing out: what would a FCR that represents interests of future generations of AIs that are more likely to enter the moral circle (i.e. when we turn off a GPT-n or make big changes to an advanced/human-level AI, are we doing something we wouldn’t be happy doing to a human or other entity that is (possibly/roughly) morally equivalent) look like? I think Bostrom might have mentioned something like this in one of his digital minds papers.
If anyone has any thoughts/wants to work on this with me, please get in touch (I’m thinking of it as a video and/or paper/essay).
This is really useful in that it examines critically what I think of as the ‘orthodox view’: alignment is good because it ‘allows humans to preserve control over the future’. This view feels fundamental but underexamined, in much of the EA/alignment world (with notable exceptions: Rich Sutton, Robin Hanson, Joscha Bach who seem species-agnostic; Paul Christiano has also fleshed out his position e.g. this part of a Dwarkesh Patel podcast).
A couple of points I wasn’t sure I understood/agreed with FWIW:
a) A relatively minor one is
To the extent you think that future AIs would not be capable of creating massive wealth for humans, or extending their lifespans, this largely implies that you think future AIs will not be very powerful, smart, or productive. Thus, by the same argument, we should also not think future AIs will be capable of making humanity go extinct.
I’m not sure about about this symmetry—I can imagine an LLM (~GPT-5 class) integrated into a nuclear/military decision-making system that could cause catastrophic death/suffering (millions/billions of immediate/secondary deaths, massive technological setback, albeit not literal extinction). I’m assuming the point doesn’t hinge on literal extinction.
b) Regarding calebp’s comment on option value: I agree most option value discussion (doesn’t seem to be much outside Bostrom and the s-risk discourse) assumes continuation of the human species, but I wonder if there is room for a more cosmopolitan framing: ‘Humans are our only example of an advanced technological civilisation, that might be on the verge of a step change in their evolution. The impact of this evolutionary step-change on the future can arguably be (on balance) good (definition of “good” tbd). The “option value” we are trying to preserve is less the existence of humans per-se, but rather the possibility of such an evolution happening at all. Put another way, we don’t an to prematurely introduce an unaligned or misaligned AI (perhaps a weak one) that causes extinction, a bad lock-in, or prevents emergence of more capable AIs that could have achieved this evolutionary transition.’
In other words, the option value is not over the number of human lives (or economic value) but rather over the possible trajectories of the future...this does not seem particularly species-specific. It just says that we should be careful not to throw these futures away.
c) point (b) hinges on why human evolution is ‘good’ in any broad or inclusive sense (outside of letting current and near-current generations live wealthier, longer lives, if indeed those are good things).
In order to answer this, it feels like we need some way of defining value ‘from the point of view of the universe’. That particular phrase is a Sidgwick/Singer thing, and I’m not sure it is directly applicable in this context (like similar phrases e.g. Nagel’s ‘view from nowhere’), but without this it is very hard to talk about non-species based notions of value (i.e. standard utilitarianism, deontological/virtue approaches all basically rely on human on animal beings).
My candidate for this ‘cosmic value’ is something like created complexity (which can be physical or not, and can include things that are not obviously economically/militarily/reproductively valuable like art). This includes having trillions of diverse computing entities (human or otherwise).
This is obviously pretty hand-wavey, but I’d be interested in talking to anyone with views (it’s basically my PhD :-)
Few more things on FTX/Alameda (from Financial Times’ Alphaville blog)
Reuters on what may have happened (ie possible transactions connecting FTX and Alameda as well as Binance’s opaque history)
An old story on SBF & FTX’s history & connections to EA
https://www.sequoiacap.com/article/sam-bankman-fried-spotlight/
Probably less directly relevant now (except as it impacts crypto values generally), but some info on Binance’s opaque financial arrangements
This is Matt Levine’s (Bloomberg) take on what’s happening with FTX (copied from a newsletter). I haven’t included the footnotes/links. I thought he explained well how (even absent fraud) fragile financial institutions are. His crypto primer is also very good on the ecosystem around the various currencies etc: https://www.bloomberg.com/features/2022-the-crypto-story/
——————
So how could this happen? I don’t know, but let me speculate a little bit.
Let’s start with Coinbase. Coinbase Global Inc. runs a cryptocurrency exchange. When FTX.com, one of the largest crypto exchanges, was instantaneously vaporized yesterday, Coinbase put out a statement, the gist of which was “don’t worry, we are not going to be instantaneously vaporized.” The part that I want to focus on is this paragraph:
There can’t be a “run on the bank” at Coinbase. As you can review in our publicly filed, audited financial statements, we hold customer assets 1:1. Any institutional lending activity at Coinbase is at the discretion of the customer and backed by collateral. We have no gating for client loan recalls or withdrawals.
The way it works is roughly that you open an account and send dollars to Coinbase, and then you tell Coinbase “I’d like to buy some Bitcoin with those dollars,” and Coinbase buys Bitcoin and holds on to it for you and charges you a fee for that transaction. You can check your account balance, and Coinbase says “you have 0.5 Bitcoin” or whatever. That 0.5 Bitcoin is, in the general case, held by Coinbase; it has possession of the Bitcoin.[1] But it is held in a custody account for you. Coinbase says:
Your funds are your funds, and your crypto is your crypto: Coinbase maintains internal systems, like a bank or a broker. Our fully audited ledger identifies your account, your fiat and crypto holdings, and tracks your account activity in real time. There’s never a situation where customer funds could be confused with corporate assets.
We will never repurpose your funds: We do not lend or take any action with your assets, unless you specifically instruct us to. Many banks and financial institutions use customer funds for commercial purposes including lending and trading, meaning that they often hold only a fraction of their customer assets at any given time. Coinbase always holds customer assets 1:1. This means that funds are available to our customers 24 hours a day, 7 days a week, 365 days of the year.
The analogy is: Imagine a weird sort of bank. You come to the bank with $100 in paper bills, and you deposit it in the bank, and the bank takes your paper bills and sticks them in an envelope with your name on it. Then it sticks the envelope in a vault, and if at any point you ask for your money back, it opens the vault and hands you your envelope. This sounds like a bad business model: The bank needs to pay for real estate and tellers and vaults, and it is not doing anything with your money. But the other weird thing about this bank is that, every day, you come in and say “hey I’d like to exchange my dollars for euros” or “my euros for pounds” or whatever, and each time you do that the bank charges you a dollar. So you have $100, which you exchange for €99, which you exchange for £98, which you exchange for $97, etc.,[2] paying the bank $1 each time. If all of the bank’s customers do this every day, then the bank makes plenty of money to pay for real estate and tellers and vaults and executive bonuses, without doing anything else with your money. It just takes the $100 out of your envelope and replaces it with €99, etc., always keeping exactly the right amount of money (in whatever currency you like that day) in exactly your envelope.[3]
And then if one day every single customer walked into the bank at the same time and said “we would like our money back,” the bank would just hand them all their envelopes. Don’t get me wrong, this would be a catastrophe for the bank: If everyone took their envelopes back, then presumably they would stop changing money at the bank and paying fees, and the bank would stop making money, and it would no longer be able to pay for real estate or tellers or vaults or executive bonuses. It would go out of business in fairly short order. But it would not go out of business that minute. It would actually have enough money to give all the customers their money back, because it kept all the customers’ money in their own envelopes the whole time.
No actual bank works that way. Real banks take deposits but don’t keep the money in envelopes; they lend it out.[4] Most classically, they borrow short to lend long, taking checking deposits that can be withdrawn at any time, and using them to make long-term mortgages. This makes them vulnerable to runs, Diamond-Dybvig, It’s a Wonderful Life, etc., everyone knows all this.
But in theory a cryptocurrency exchange could work that way, and at a high level of generality Coinbase sort of does.[5] Historically — not so much now, but until early this year anyway — cryptocurrencies were volatile and exciting and people were jazzed to trade them a lot, so you could make a lot of money by just charging fees without doing anything else with customer assets. And that is a run-proof business. If everyone takes their money out at once, you have the money.
But then one day a customer comes to you and says “I have $10,000, but I am really bullish on Bitcoin, so I would like to buy $20,000 worth of Bitcoin. Why don’t you lend me $10,000 so I can buy $20,000 of Bitcoin, so I can get more excitement?” This is called a margin loan.
Or — equivalently — a customer comes to you and says “I have $20,000 of Bitcoin in my account, and I need some cash this month. I don’t want to sell my Bitcoin, because I am a true believer and also do not want to realize gains for tax purposes. Could you just lend me $10,000, secured by my $20,000 of Bitcoin? You know I’m good for it: If I don’t pay you back, you can sell my Bitcoin and pay yourself back from the proceeds.”
You might just say “no, that’s dumb, Bitcoin is volatile, buying $10,000 of Bitcoin is plenty of excitement.” (In fact Coinbase shut down margin trading in 2020.) But your competitors probably offer loans, and it is tempting for you to do it too. So you say, sure, fine, I’ll take your $10,000 and put $20,000 of Bitcoin in your account.
But where do you get the money that you are lending to the customer? Well, you have to borrow it too. Ordinarily the way that you will borrow it is by putting up the customer’s Bitcoin as collateral to your lender, just as the customer puts up its Bitcoin as collateral to you. If the customer defaults, you still have to pay your lender (and then you get the Bitcoin back and can sell it to pay off your customer’s liability to you); if you default, the lender sells the Bitcoin.
But who are the lenders? Oh, various possibilities. But one general point is that while some customers will want to borrow dollars to buy Bitcoin, other customers will want to borrow Bitcoin. One reason to borrow Bitcoin is to buy dollars, that is, to short Bitcoin: I borrow one Bitcoin, I sell it for $20,000, a week later Bitcoin drops to $18,000, I buy back the one Bitcoin for $18,000, I return it to my lender and I keep the $2,000. There are variations on this trade (I borrow Bitcoin and sell it for Ethereum, betting on the relative value between the tokens, etc.). It is necessarily a leveraged trade; I can’t short Bitcoin without borrowing it.[6]
If you are a crypto exchange, this is a nice opportunity. You have Customer A who has Bitcoin and wants to borrow dollars, and Customer B who has dollars and wants to borrow Bitcoin. (By “dollars,” for a crypto exchange, I mostly mean “dollar-denominated stablecoins,” though potentially also dollars.) You take some of Customer A’s Bitcoin and lend it to Customer B, and you take some of Customer B’s dollars and lend them to Customer A. Each of them is overcollateralized — you only lend Customer A half the value of her Bitcoin, and you only lend Customer B half the value of his dollars — so you feel pretty safe. And they both pay you interest.
But there are risks. One day Customer A might come in, pay off her dollar loan, and ask to take her Bitcoin back. You don’t have her Bitcoin, or not all of them anyway; some of them are with Customer B. Customer B owes them to you — ultimately you’re good for it — but you don’t have them now. There is a timing problem.
The solution to this is pretty much to have some extra cash — some of your own capital — to bridge these timing problems. Eventually you’ll get the rest of the Bitcoin back from Customer B, but for now you just pay Customer A out of your own Bitcoin stash.
But the timing problem is also connected to a real economic risk. If the price of Bitcoin falls by 90%, Customer B will be thrilled. He will come to you and say “here’s my Bitcoin back, I’d like to withdraw my dollars.” But you don’t have his dollars, or not all of them; half of them are with Customer A. Your dollar loan to Customer A is now underwater: You loaned her 50% of the value of her Bitcoin, but Bitcoin fell by 90%, so she owes you more than her collateral is worth. You call her up and ask her for more money — a “margin call” — but she, sensibly, doesn’t answer the phone.[7] You have to pay Customer B out of your own capital, and you don’t get it back from Customer A. You’ve just lost money. Actually that’s the best outcome. The worst outcome is that you don’t have enough capital, you go bankrupt, and Customer B does not get his money back.
Everyone knows this, which is why crypto exchanges — and securities broker-dealers, who have the same basic business model — spend most of their time thinking about risk management. Before the price of Bitcoin drops too far, you will be calling up Customer A for more margin, and if she doesn’t answer the phone you will liquidate her position to pay back the loan you made. If you are a sophisticated modern crypto exchange like FTX, you will have automated 24⁄7 margining systems that automatically liquidate trades that have gotten too risky, so that only the rarest catastrophic market moves could get you in trouble.
But sometimes market moves are catastrophic, and in particular, sometimes securities broker-dealers and crypto exchanges will have “run on the bank” risks. If everyone knows that you are in this situation — that you have a lot of Bitcoin collateral and Bitcoin prices are falling — people will expect you to have to liquidate your Bitcoin collateral, so they will expect Bitcoin prices to fall, so they will sell Bitcoin, which will cause Bitcoin prices to fall, which will cause your long-Bitcoin customers to default, which will cause you to liquidate Bitcoin at lower and lower prices, etc., until you are bankrupt.
Now let’s add one more crypto element. If you are a crypto exchange, you might issue your own crypto token. FTX issues a token called FTT. The attributes of this token are, like, it entitles you to some discounts and stuff, but the main attribute is that FTX periodically uses a portion of its profits to buy back FTT tokens. This makes FTT kind of like stock in FTX: The higher FTX’s profits are, the higher the price of FTT will be.[8] It is not actually stock in FTX — in fact FTX is a company and has stock and venture capitalists bought it, etc. — but it is a lot like stock in FTX. FTT is a bet on FTX’s future profits.
But it is also a crypto token, which means that a customer can come to you and post $100 worth of FTT as collateral and borrow $50 worth of Bitcoin, or dollars, or whatever, against that collateral, just as they would with any other token. Or something; you might set the margin requirements higher or lower, letting customers borrow 25% or 50% or 95% of the value of their FTT token collateral.
If you think of the token as “more or less stock,” and you think of a crypto exchange as a securities broker-dealer, this is completely insane. If you go to an investment bank and say “lend me $1 billion, and I will post $2 billion of your stock as collateral,” you are messing with very dark magic and they will say no.[9] The problem with this is that it is wrong-way risk. (It is also, at least sometimes, illegal.) If people start to worry about the investment bank’s financial health, its stock will go down, which means that its collateral will be less valuable, which means that its financial health will get worse, which means that its stock will go down, etc. It is a death spiral. In general it should not be possible to bankrupt an investment bank by shorting its stock. If one of the bank’s main assets is its own stock — is a leveraged bet on its own stock — then it is easy to bankrupt it by shorting its stock.
The worst case is something like:
You have 100 Customer As who are long Bitcoin on margin: They each have 1 Bitcoin in their accounts and owe you $10,000. You have 100 Customer Bs who are short Bitcoin on margin: They each have $20,000 in their account and owe you 0.5 Bitcoin. You have loaned 50 of the Customer As’ Bitcoins to the Customer Bs, and $1 million of the Customer Bs’ dollars to the Customer As. You keep the other 50 Bitcoins and $1 million as collateral. Your accounts show that you owe clients 100 Bitcoins and $2 million, and that they owe you back 50 Bitcoins and $1 million, and you have 50 Bitcoins and $1 million on hand, so everything balances. You have one Customer C who says “hi I would like to borrow 50 Bitcoins and $1 million, I will secure that loan with 150,000 FTT, each of which is worth $20.” You say “sure, sounds good,” and hand over all your collateral. Now you have 150,000 of FTT, worth $3 million, as collateral (and no Bitcoins or dollars). Your accounts show that you owe clients 100 Bitcoins and $2 million and 150,000 FTT, and they owe you back 100 Bitcoins and $2 million, and you have 150,000 FTT of collateral, so everything balances. But then if the value of FTT drops to zero, you have nothing. You have no Bitcoins to give to the customers to whom you owe Bitcoins, no dollars to give to the customers to whom you owe dollars. You just have to call up Customer C and say “hey we need all those dollars and Bitcoins back.” But Customer C will not want to give you back all those valuable dollars and Bitcoins in exchange for now-worthless FTT. Also the fact that Customer C had all that FTT in the first place is not a great sign. It is an FTT whale, and FTT is now worthless. Has it been borrowing elsewhere against FTT? Are all those debts coming due?
Now let’s add a few more FTX-specific elements. One is that FTX is an exchange for levered traders, offering products like perpetual futures and leveraged tokens that build in margin lending. So whereas the basic model of Coinbase is “they buy Bitcoin for you and put it in an envelope,” the basic model of FTX has to be “they lend you money to buy crypto and then make use of your crypto to get the money.” In financial terms, they have to rehypothecate your collateral; you can’t expect them to just keep it in an envelope if they’re lending you the money to buy it.
The other is that FTX is closely associated with a hedge fund called Alameda Research. Sam Bankman-Fried founded Alameda to do crypto arbitrage and market-making trades, and then he founded FTX to basically have a better exchange for Alameda to trade on. Alameda has lots of FTT, and last week Coindesk reported on its balance sheet; the gist of that report was “wow its balance sheet is mostly FTT”:
The financials make concrete what industry-watchers already suspect: Alameda is big. As of June 30, the company’s assets amounted to $14.6 billion. Its single biggest asset: $3.66 billion of “unlocked FTT.” The third-largest entry on the assets side of the accounting ledger? A $2.16 billion pile of “FTT collateral.”
There are more FTX tokens among its $8 billion of liabilities: $292 million of “locked FTT.” (The liabilities are dominated by $7.4 billion of loans.)
That is not in itself a reason for a run on FTX! It might be a reason for the price of FTT to go down, if you think that Alameda has too much of it and might need to sell it.
The reason for a run on FTX is that you think that Alameda is, in my terminology, Customer C. The reason for a run on FTX is if you think that FTX loaned Alameda a bunch of customer assets and got back FTT in exchange. If that’s the case, then a crash in the price of FTT will destabilize FTX. If you’re worried about that, you should take your money out of FTX before the crash. If everyone is worried about that, they will all take their money out of FTX. But FTX doesn’t have their money; it has FTT, and a loan to Alameda. If they all take their money out, that’s a bank run.
And all of this is self-fulfilling: If you are worried about FTX’s business, then the price of FTT should go down. If the price of FTT goes down, then FTX’s business is riskier, because it has less collateral. If, say, the operator of the biggest crypto exchange gently raises one eyebrow and says “FTT, eh?” that can be enough to topple FTX. FTT goes down, leaving FTX undercapitalized, leading to customer withdrawals, leading to ruin.
Anyway it is still early and confusing but that seems to be the story of FTX. Coindesk reported on Alameda’s FTT exposure, and then Changpeng “CZ” Zhao, the founder of Binance Holdings Ltd., the largest crypto exchange, raised eyebrows by tweeting that Binance would sell its FTT holdings “due to recent revelations.” People worried that this would tank the price of FTT and put pressure on FTX, so they started withdrawing money from FTX. FTX didn’t have the money, and Bankman-Fried started calling around asking for a loan or a bailout. Eventually he called CZ himself, and they announced a non-binding letter of intent for Binance to acquire FTX and make customers whole. Bankman-Fried’s fortune basically vanished, as did his “ emperor aura.” Venture capital investors in FTX — which last raised money at a $32 billion valuation — are probably getting zeroed, the price of FTT collapsed, and now regulators are investigating.
In this description I have drawn on Twitter threads from Jon Wu, Lucas Nuzzi and an anonymous “Wassie Lawyer,” who make arguments along these lines, as well as this Substack post from Byrne Hobart. But the most informed view is probably that of CZ himself, who tweeted this morning:
Two big lessons:
1: Never use a token you created as collateral.
2: Don’t borrow if you run a crypto business. Don’t use capital “efficiently”. Have a large reserve.
Binance has never used BNB for collateral, and we have never taken on debt.
“Never use a token you created as collateral” suggests, to me, that FTX accepted its FTT token as collateral, probably from Alameda, probably in exchange for borrowing assets that it owes to customers. And that that went wrong in roughly the way I have outlined.
One other point here is that if this is the story, then it is not a liquidity crisis but a solvency one. That is, the problem is not a timing mismatch, in which FTX’s customers asked for their cash back but FTX did not have enough ready cash because it had long-term but money-good loans out. The problem is that FTX took its customers’ money and traded it for a pile of magic beans, and now the beans are worthless and there’s a huge hole in the balance sheet. On that note:
Changpeng Zhao moved fast when Sam Bankman-Fried’s FTX.com was on the brink, offering to take it over and stem any further crypto contagion.
Within hours, he was forced to reconsider.
For starters, Binance executives quickly found themselves staring into a financial black hole—a gap between liabilities and assets at FTX that’s probably in the billions, and possibly more than $6 billion, according to a person familiar with the matter.
On top of that, US regulators are circling FTX, investigating whether the firm properly handled customer funds, as well as its relationship with other parts of Bankman-Fried’s crypto empire, Bloomberg News reported Wednesday.
It makes for a tricky decision for Zhao, known in the crypto world as CZ: Follow through with rescuing his onetime top rival and shoulder the financial and regulatory burdens, or let FTX crumble and sort through the potential wreckage? Zhao himself admits there was no “master plan” to take over FTX.
His answer, at least for now, is that the financial hole appears too deep. Binance is unlikely to follow through on its takeover of FTX, according to the person familiar, who wasn’t authorized to publicly discuss the matter.
Analysing a 2036 Takeover Scenario
Defective Altruism article in Current Affairs Magazine
If you haven’t seen his site, it’s quite useful—also has lots of adjacent commentary/arguments against. I found Bostrom’s FAQs useful, and essays by Hanson, Chalmers, and Steinhart interesting.
Thank you—my bad—changed !
That’s true, and perhaps working with a graphic designer is a good place to start (i.e. with website design, logo).
This is great—really useful to have calm, researched responses to bundle/bungle/shibboleths like Torres/Gebru seem to have come up with. I have heard the eugenicist critique a lot, and unfortunately some of it is influential via culture/media eg this book (the author has a CS background and makes many good points (as does Torres in their other writing on x-risk), but the media tagline ends up being reductive and senationalist)