CS student at the University of Southern California. Previously worked for three years as a data scientist at a fintech startup. Before that, four months on a work trial at AI Impacts. Currently working with Professor Lionel Levine on language model safety research.
aogara
Karma: 2,212
Separately, I disagree voted because it seems wrong about the facts. HLI’s analysis relies on many different studies with a sample size >30K, as Michael explains below.
My sense is that this is the kind of result that you’d tend to see in a world with shorter rather than longer timelines.
Agreed. I’m working on this area and I thought it would be another few years before we saw human-level AI in Diplomacy with natural language communication. Diplomacy without natural language was already at human level, but we don’t have many examples of game-playing agents successfully using natural language. This seems to show there’s not much of a challenge there.
One small personal experience: I worked a non-EA job for three years. None of my close friends were interested in EA, and my job wasn’t in a highly impactful cause area. I developed some other interests during those years, reading a lot about startups and VC and finance. Despite my enthusiasm when I first read Peter Singer and Doing Good Better, I think my interest in working on EA topics could have slowly faded and been replaced with other interesting ideas.
The EA community was a big part of what kept me engaged with EA. This forum was a steady stream of information about how to do good in the world, and one that allowed me to voice my own opinions and have lots of interesting conversations. I attended two online EA Globals which mostly made me identify more as an EA. Later I went back to school, where the university EA group leader reached out and encouraged me to join a reading group. We had weekly dinners and great conversations, and only a few months later, I quit my part-time job at a for-profit startup and began working on AI Safety.
It’s hard to say what the counterfactual is, but I think the odds I’d be working on AI Safety right now would have been much lower without the identity, personal connections, and intellectual engagement from the EA community. Part of it is nerdsniping — it’s not always easy to find smart, sensible conversations about the world, but I’ve always found EA to provide plenty of them. There are real downsides — I used to think that I deferred way too much on my opinions about cause prioritization (I think I’ve improved, but maybe I’ve just lost my independent thinking). Your post is a great analysis of those dynamics and I’m not trying to argue for a bottom line, but just wanted to share one personal benefit of the community.
This is a really nice perspective. Great reminder that working on EA dollars is not the only way to do good, there’s a whole world of ways to have an impact outside of the community itself.
This was my understanding as well, that he actually believes in utilitarianism and was only cynical about individual public stances like the value of regulation or the importance of transparency.
Kelsey gave him three thumbs up when he blamed hundreds of millions moved out of FTX after the bankruptcy filing on a hacker. I doubt she fully believes that explanation—he has no credibility and offered no evidence. More likely is that she’s being friendly to get access. That’s not necessarily wrong, maybe more statements from SBF is a good thing, but we can admit that he’s looking for a friendly conversation and she provided it. Same as the recent NYT interview.
Normally I would downvote a claim without evidence. But if this is true, it would help explain why FTX bailed out Alameda using customer funds. And it would have important implications for how EA views conflicts of interest from romantic relationships in other areas.
AI Safety has plenty of romantic relationships between important people. To name just one relationship that has been made fully public and transparent, Holden Karnofsky is married to Daniella Amodei, President of Anthropic and sister to Dario Amodei, an advisor to OpenPhil on AI Safety. I think it’s reasonable to believe that Dario has had more influence over OpenPhil’s institutional views, and that OpenPhil has a higher opinion of Anthropic, than would be true without this romantic relationship.
My perception is that many EAs think romantic relationships don’t often cause problematic conflicts of interest. Sabs’ comment was downvoted heavily for reasons such as Lukas’s below. I have previously drafted posts similar to this one and not published them because of fear of backlash.
I don’t want to name other relationships mostly because it wouldn’t solve the problem. I don’t think romance between coworkers is always wrong — I’m currently dating someone I used to work with, though our relationship only started after she left the company. But when grantmakers are taking advice from or giving grants to people they’re romantically involved with, I think there’s a lot of room for compromised decision making. These conflicts of interest should at least be public, and better yet avoided entirely.
Julia Wise writes about this a bit here: https://forum.effectivealtruism.org/posts/fovDwkBQgTqRMoHZM/power-dynamics-between-people-in-ea
My understanding is that the rise and fall of FTX was a positive feedback loop turned negative. When FTX had money, they used it to drive up the price of their own crypocurrency, FTT. One of the biggest holders of FTT was Alameda Research, SBF’s hedge fund. They profit nominally from rising FTT, but the only big buyer of FTT is FTX, so how does Alameda turn FTT into cash? It looks like FTX gave Alameda loans on the premise that Alameda’s FTT was strong collateral. It’s a money printer when times are good: FTX profits on customer transactions and loans to Alameda, and Alameda profits on trades.
But it’s an extremely precarious position. If the price of FTT drops, FTX loses money. People get worried about the money they deposited in FTX and try to withdraw. To pay them back, FTX can try to sell FTT coin, but FTT coin is already trading below where FTX bought it. FTX could also try to call back their loans to Alameda, but those loans are collateralized by FTT, so Alameda can’t pay. All of a sudden, customers want their money back and FTX can’t provide it. FTT, FTX, and Alameda collapse.
That’s my current best guess at what happened. We’re going to find out a lot more in the weeks to come, but this is the story that’s currently being reported. Here’s Byrne Hobart’s timeline:
November 1, 9:44am ET: Coindesk writes a story on a leaked Alameda balance sheet showing large exposure to FTX’s FTT token.
November 4, 5:38am: a Substack post asks if the company is insolvent, and rumors spread over the weekend.
November 6th, 10:47am: Changpeng “CZ” Zhao, CEO of rival exchange Binance, announces plans to sell his FTT tokens.
16 minutes later: Alameda Research’s CEO offers to buy them.
November 6th, 6:28pm: Sam Bankman-Fried tweets that the exchange is solvent. Many people remember their Bagehot.
November 7th, 7:38am: SBF reiterates that the exchange is fine. The Bagehot-posting intensifies.
November 8, 6:00am: Tech news sites cover slow withdrawals at FTX.
November 8th, 11:03am: SBF announces that FTX has signed a letter of intent to be acquired by Binance, which has been confirmed by CZ. By this time, total losses based on the positions in the Coindesk report are at least $2.3bn, mostly from FTT depreciation. (Those losses will roughly double in the next few hours as FTT continues to depreciate.)
November 9th: Binance withdraws from deal to acquire FTX.
Byrne speculates that FTX’s profitability could be as manufactured as it sounds:
This could have been a completely cynical perpetual motion machine where Alameda makes FTX a better exchange by providing cheap liquidity, and the growth of FTX increases the value of the underlying FTT, giving Alameda more collateral to borrow against for providing more liquidity.
Matt Levine compares it to a bank providing a loan with its own stock as collateral. This is stupid and often illegal because if the loan goes bad, the stock price goes down, making it a double-whammy:
If you go to an investment bank and say “lend me $1 billion, and I will post $2 billion of your stock as collateral,” you are messing with very dark magic and they will say no. The problem with this is that it is wrong-way risk. (It is also, at least sometimes, illegal.) If people start to worry about the investment bank’s financial health, its stock will go down, which means that its collateral will be less valuable, which means that its financial health will get worse, which means that its stock will go down, etc. It is a death spiral. In general it should not be possible to bankrupt an investment bank by shorting its stock. If one of the bank’s main assets is its own stock — is a leveraged bet on its own stock — then it is easy to bankrupt it by shorting its stock.
CZ, CEO of Binance, says enough to help put the pieces together:
Two big lessons:
Never use a token you created as collateral.
Don’t borrow if you run a crypto business.
Jon Wu has a great thread:
tl;dr
- CZ dumps $FTT
- Alameda faces external margin calls, looks to FTX for liquidity
- FTX depositors have withdrawn; piggy bank is broken
- FTX’s assets are loans Alameda can’t repay, against real customer liabilities
- FTX in the hole for billions
- CZ buys FTXIf this is what happened, then SBF took an incredible gamble with other people’s money and lost big, billions of dollars in customer deposits big. It’s a shame for FTX and the EA community.
Thanks, best one I’ve read.
Since releasing his report on existential risk from power-seeking AI, his estimate of AI x-risk has increased from 5% to >10%. Why?
Why do you think there will be lawsuits? Have other crypto exchanges that went under resulted in lawsuits? Not disagreeing, just not up to date on this stuff.
Matt Levine seems to agree. Some quotes from his article:
Is Binance paying FTX tens of billions of dollars for its equity? I would be very, very, very surprised!
My main assumption is that if you are a crypto exchange facing a “significant liquidity crunch,” and you call up a bigger crypto exchange to ask for help, and you sign a deal for them to buy you the same day, then the price that they are paying you is, roughly speaking, zero.
There is precedent. In my world, the most famous precedent is probably JPMorgan Chase & Co. buying Bear Stearns Cos. for $2 per share one Sunday in 2008, “less than one-tenth the firm’s market price on Friday.” (Later the price was revised up.) If you need a bailout from JPMorgan over the weekend, JPMorgan will step in and make sure that your business can keep operating and that your customers will get paid and that the financial system does not collapse, but you won’t get paid much.
But in the crypto world, the famous precedents are pretty much Sam Bankman-Fried
bailing out crypto lenders this summer. FTXbailed out BlockFi Inc., getting an option to buy it foras little as $15 millionor as much as $240 million; it had been valued at $3 billion in 2021. Alameda helped out Voyager Digital with a $75 million loan, and FTX ultimately agreed tobuy its assets out of bankruptcy, cashing out customers but paying something like $51 million for its actual business; its equity market capitalization was more than $1 billion in April.This summer large crypto firms were on sale at pennies on the dollar if you had some ready cash and a tolerance for risk; Bankman-Fried did… Now, it seems, his large crypto firm was on sale at pennies on the dollar, and Zhao has the cash.
Yeah I agree. My perceptions have changed a bit since writing this post, at least in technical AI Safety there seems to be a pretty good on-ramp of education → junior roles → hopefully senior roles??? Haven’t gotten there yet ¯_(ツ)_/¯
Sounds like a good plan! I think there are other challenges with alignment that this wouldn’t solve (e.g. inner misalignment), but this would help. If you haven’t seen it, you might be interested in https://arxiv.org/abs/2008.02275
I strongly agree. SBF and Future Fund don’t seem to have any track record in party politics. Given the massive reputational risks to EA that might not be easily fixable, I think their political advocacy should be closely scrutinized and possibly slowed to build up more capacity before engaging.
My perception is informed by the Flynn campaign, which seemed to have important failures. Local political leaders criticized the campaign for failing engage with local media and elected officials. They spent more an $800,000 on an attack ad calling the eventual winner of the race a “lobbyist for a corporation accused of driving up drug prices”, but many EAs now believe that claim was “very misleading”. They received plenty of negative local media coverage. They did manage to secure a $1M donation from a PAC aligned with Nancy Pelosi, and building relationships with mainstream Democrats seems to be part of the strategy in supporting Foushee. This could be a useful strategy, but also risks associating EA with corrupt big money party politics.
I’ve signed up for fundraising emails from the EA Donor Network organized following the Flynn campaign. They’ve recommended two candidates so far, Victoria Gu and Seth Magaziner in Rhode Island, both with single paragraph explanations of why the candidates are worth supporting. When I emailed back looking for more information about why we should donate to these candidates, I received a response to my first email but not a second. To be clear, I’m not really looking for an explanation via private emails—I’d like EA political work to adopt the same standards of transparency and rigorous analysis that have powered EA success in other domains.
This is really cool, a whole topic within AI safety that I haven’t seen much focus on. I’ll plan to read some of these papers, but if any of your participants is interested in writing up summaries or key insights from the curriculum I’d definitely be interested in reading them.
Agreed on that as well. The Carlsmith report is the only quantitative model of AI risk I’m aware of and it was the right call to do this analysis on it. I think we do have reasonably large error bars on its parameters (though perhaps smaller than an order of magnitude) meaning your insight is important.
Why aren’t there more models? My guess is that it’s just very difficult, with lots of overlapping and entangled scenarios that are hard to tease apart. How would you go about constructing an overall x-risk from the list of disjunctive risks? You can’t assume they’re independent events, and generating conditional probabilities for each seems challenging and not necessarily helpful.
Ajeya Cotra’s BioAnchors report is another quantitative model of that drives lots of beliefs on AI timelines. Stephanie Lin won the EA Critique Contest with one critique, but I’d be curious if you’d have other concerns with it.
The application form is showing up as private for me. Very cool idea though, the success of Eleuther and Stability suggests that this is a viable model. Excited to see it unfold and hopefully contribute!
What’s your thought process on this? It’s a tough question, but any sense of the rough likelihoods?