Project lead of LessWrong 2.0, often helping the EA Forum with various issues with the forum. If something is broken on the site, it’s a good chance it’s my fault (Sorry!).
Habryka
Karma: 18,811
I am the author of the linked post that DPiepgrass was commenting on: https://www.lesswrong.com/posts/HCAyiuZe9wz8tG6EF/my-tentative-best-guess-on-how-eas-and-rationalists
This link’s hypothesis is about people just trying to fit in―but SBF seemed not to try to fit in to his peer group! He engaged in a series of reckless and fraudulent behaviors that none of his peers seemed to want.
(Author of the post) My model is that Sam had some initial tendencies for reckless behavior and bullet-biting, and those were then greatly exacerbated via evaporative cooling dynamics at FTX.
It sounds like SBF drove away everyone who couldn’t stand his methods until only people who tolerated him were left. That’s a pretty different way of making an organization go insane.
Relatedly, this kind of evaporative cooling is exactly the dynamic I was trying to point to in my post. Quotes:
People who don’t want to live up to the demanding standard leave, which causes evaporative cooling and this raises the standards for the people who remain. Frequently this also causes the group to lose critical mass.
[...]
My current best model of what happened at an individual psychological level was many people being attracted to FTX/Alameda because of the potential resources, then many rounds of evaporative cooling as anyone who was not extremely hardcore according to the group standard was kicked out, with there being a constant sense of insecurity for everyone involved that came from the frequent purges of people who seemed to not be on board with the group standard.
Do you have links to people being very worried about gray goo stuff?
(Also, the post you link to makes this clear, but this was a prediction from when Eliezer was a teenager, or just turned 20, which does not make for a particularly good comparison, IMO)
Also, I don’t think “completely uninvestigated” is a correct characterization—they were investigated enough to be presented to a grand jury, which indicted SBF for campaign-finance violations. Federal prosecutors do not generally indict without a pretty good investigation first, especially in high-profile cases. I think we have a pretty decent idea of what he did (see pp. 18-22 of the prosecution’s sentencing memo). Moreover, Salame and Singh—who don’t have extradition-related issues—pled guilty to campaign-finance violations.
Oh, that is interesting and an update for me. I had interpreted the relevant section of the memo as more of a “we didn’t really investigate it, but here is what we have”, but you know this stuff much better than I do.
I… think I will continue describing this as “weird” though it makes sense that as a lawyer it’s not that weird to you.
It feels very off to have a bunch of crimes uninvestigated because someone fled the country and then was extradited, and I am pretty confused why the Bahamas cooperated with Sam here (my guess is that it’s a case of political corruption, though I can also imagine other reasons). It’s not like the Bahamas had anything obvious to gain from Sam not being convicted of the campaign finance violations.
This is great!
Note that this list is not comprehensive. In-particular it doesn’t go into detail on any of the campaign finance violations and other policy-stuff that Sam was involved in, which I think never ended up investigated due to a kind of weird agreement with the Bahamas authorities. But during the trial we did hear some pretty clear evidence of at least campaign finance violations (and my guess is there would be a bunch more if one kept digging, as well as stuff that wouldn’t necessarily be crimes but stuff I think most people here would still consider highly unethical).
It seems to me that a case study of how exactly FTX occurred, and where things failed, would be among one of the best things to use to figure out what thing to do instead.
Currently the majority of people who have an interest in this are blocked by not really knowing what worked and didn’t work in the FTX case, and so probably will have trouble arguing compellingly for any alternative, and also lack some of the most crucial data. My guess is you might have the relevant information from informal conversations, but most don’t.
I do think also just directly looking for an alternative seems good. I am not saying that doing an FTX investigation is literally the very best thing to do in the world, it just seems better than what I see EA leadership spending their time on instead. If you had the choice between “figure out a mechanism detecting and propagating information about future adversarial behavior” and “do an FTX investigation”, I would feel pretty great about both, and honestly don’t really know which one I would prefer. As far as I can tell neither of these things is seeing much effort invested into it.
My current sense is that there is no motivation to find an alternative because people mistakenly think it works fine enough and so there is no need to try to find something better (and also in the absence of an investigation and clear arguments about why the rumor thing doesn’t work, people probably think they can’t really be blamed if the strategy fails again)
I think investing in FTX was genuinely a good idea, if you were a profit maximizer, even if you strongly suspected the fraud. As Jason says, as an investor losing money due to fraud isn’t any worse than losing money because a company fails to otherwise be profitable, so even assigning 20%-30% probability to fraud for a high-risk investment like FTX where you are expecting >2x returns in a short number of years will not make a huge difference to your bottomline.
In many ways you should expect being the kind of person who is willing to commit fraud to be positively associated with returns, because doing illegal and fradulent things means that the people who run the organization take on massive risk where you are not exposed to the downside, but you are exposed to the upside. It’s not worth it to literally invest in fraud, but it is worth it to invest into the kind of company where the CEO is willing to go to prison, since you don’t really have any risk of going to prison, but you get the upside of the legal risk they take on (think of Uber blatantly violating laws until they established a new market, which probably exposed leadership to substantial legal risk, but investors just got to reap the profits).
I’m not the person quoted, but I agree with this part, and some of the reasons for why I expect the results of an investigation like this to be boring aren’t based on any private or confidential information, so perhaps worth sharing.
One key reason: I think rumor mills are not very effective fraud detection mechanisms.
Huh, the same reason you cite for why you are not interested in doing an investigation is one of the key reasons why I want an investigation.
It seems to me that current EA leadership is basically planning to continue a “our primary defense against bad actors is the rumor mill” strategy. Having an analysis of how that strategy did not work, and in some sense can’t work for things like this seems like it’s one of the things that would have the most potential to give rise to something better here.
Becca, Nicole and Max all stand out as people who I think burned out trying to make things go better around FTX stuff.
Also Claire leaving her position worsened my expectations of how much Open Phil will do things that seem bad. Alexander also seems substantially worse than Holden on this dimension. I think Holden was on the way out anyways, but my sense was Claire found the FTX-adjacent work very stressful and that played a role in her leaving (I don’t thinks she agrees with me on many of these issues, but I nevertheless trusted her decision-making more than others in the space).
Yeah, I think this would only make sense if you would somehow end up majorly shaping the algorithms and structure of Twitter. I don’t think just being a shareholder really does much here.
Thanks, that’s very helpful, though if you think it’s mostly because the PR cost has already been paid, then that does provide little solace under my worldview. Let’s assume the PR costs were still ongoing, do you think it would have then flipped the decision?
FWIW, buying Twitter still seems plausibly like a good idea to me. It sure seems to be the single place that is most shaping public opinion on a large number of topics I care a lot about (like AI x-risk attitudes), and making that go better seems worth a lot.
I don’t particularly feel like my knowledge here is confidential, it would just take a bunch of inferential distance to cross. I do have some confidential information, but it doesn’t feel that load-bearing to me.
This dialogue has a bit of a flavor of the kind of thing I am worried about: https://www.lesswrong.com/posts/vFqa8DZCuhyrbSnyx/integrity-in-ai-governance-and-advocacy?revision=1.0.0
I think this captures some of what I mean, though my model is also that the “Maximally naive” view is not very stable, in that if you are being “maximally naive” you do often end up just lying to people (because the predictable benefits from lying to people outweigh the predictable costs in that moment).
I do think a combination of being “maximally naive” combined with strong norms against deception and in favor of honesty can work, though in-general people want good reasons for following norms, and arguing for honesty requires some non-naive reasoning.
My understanding is that UK law and state law whistleblower protections are extremely weak and only cover knowledge of literal and usually substantial crimes (including in California). I don’t think any legally-mandated whistleblower protections make much of a difference for the kind of thing that EAs are likely to encounter.
I checked the state of the law in the FTX case, and unless someone knew specifically of clear fraud going on, they would have not been protected, which seems like it makes them mostly useless for things we care about. They also wouldn’t cover e.g. capabilities companies being reckless or violating commitments they made, unless they break some clear law, and even then protections are pretty limited. So I can’t really think of any case, except the most extreme, in which at least the US state protections come into play.
I was not aware of any CEA or 80k whistleblower systems. If they have some, that seems good! Is there any place that has more details on them? (you also didn’t mention them in the article you linked, which I had read recently, so I wasn’t aware of them)
Also, for the record, organizational whisteblower protections seem not that important to me. I e.g. care more about having norms against libel suits and other litigious behavior, though the norms for that seem mostly gone, so I expect substantially less whistleblowing of that type in the future. I mostly covered them because I was comprehensively covering the list of things people submitted to the Coordination Forum.
Publishing it probably wouldn’t make sense at any other time, so Zach may have been stuck between being rushed into publishing it too early or not responding to the public-interest event at all.
Yeah, agree, that makes sense. I do think it was the wrong call, but I can understand the perceived urgency.
Also, it seems unlikely he booked six weeks off right after SBF’s sentencing for that reason.
Totally, to be clear, I think it’s totally fine for Zack to take time off, and wasn’t intending to comment on that at all. I was just responding to the (what I perceived to be a separate thread) of wanting to hold off on engaging until he had formed considered opinions.
Yeah, I think just buying Twitter to steer the narrative seems quite bad. But like, I have spent a large fraction of my career trying to think of mechanism design for discussion and social media platforms and so my relation to Twitter is I think a pretty healthy “I think I see lots of ways in which you could make this platform much more sanity-promoting” in a way that isn’t about just spreading my memes and ideologies.
Will has somewhat less of that background, and I think would have less justified confidence in his ability to actually make the platform better from a general sanity perspective, though still seems pretty plausible to me he saw or sees genuine ways to make the platform better for humanity.