Formerly a software engineer at Google, now I’m doing independent AI alignment research.
Because of my focus on AI alignment, I tend to post more on LessWrong and AI Alignment Forum than I do here.
I’m always happy to connect with other researchers or people interested in AI alignment and effective altruism. Feel free to send me a private message!
I’ve also been very upset since the FTX scandal began, and I love this community too. I think you’re right that EA will lose some people. But I am not so worried the community will collapse (although it’s possible that ending the global EA community could be a good thing). People’s memories are short, and all things pass. In one year, I would be willing to bet you there will still be lots of (and still not enough!) good people working on and donating to important, tractable, and neglected causes. There will still be an EA Forum with lively debates happening, and that arguments about FTX will by that point make up a small fraction of the content. There will be still new people discovering EA and getting inspired by the potential to increase their positive impact in the world.
To be sure, I do think we should be worried* about the future of EA right now. But more in the sense of worried about whether EA can remain true to its core values and ideals going forward than about whether it can survive in some form.
--
*Note that when I say “we should be worried”, I actually mean “we should be putting careful attention toward” rather than “we should be consumed by anxiety about”. Be kind to yourself, and if you’re feeling more of the latter, now may be a good time to double down on self-care.
Can you clarify which “public hearings” were demanded? Not sure if you’re talking about how quickly the bankruptcy process has been moving at FTX, or how the reactions from people on EA Forum since the news about FTX started.
I followed this link, but I don’t understand what it has to do with regulatory capture. The linked thread seems to be about nepotistic hiring and conflicts of interest at/around OpenAI.
I have thoughts on other points you made but just wanted to comment on this one bit for the moment:
I think it is very strange that SBF had so many inconsistencies between his claimed moral positions and his behavior and nobody noticed it.
Habryka noticed. His full comment is at that link but here are some key excerpts (emphasis mine):
Like, to be clear, I think the vast majority of EAs had little they could have or should have done here. But I think that I, and a bunch of people in the EA leadership, had the ability to actually do something about this.
I sent emails in which I warned people of SBF. I had had messages written but that I never sent that seem to me like if I had sent them they would have actually caused people to realize a bunch of inconsistencies in Sam’s story. I had sat down my whole team, swore them to secrecy, and told them various pretty clearly illegal things that I heard Sam had done [sadly all uncomfirmed, asking for confidentiality and only in rumors] that convinced me that we should avoid doing business with him as much as possible (this was when we were considering whether to do a bunch of community building in the Bahamas). Like, in my worldview, I did do my due diligence, and FTX completely failed my due diligence, and I just failed to somehow propagate that knowledge.
...
I do also think it is pretty unlikely we could have prevented FTX exploding, though I do think we could have likely prevented FTX being super embedded in the EA Community, having a ton of people rely on its funding, and having Sam be held up in tons of places as a paragon of the EA community. Like, I think we fucked up pretty hard by saying for a few years that we think Sam did great by our values, when I think it was already pretty clear by that point that he quite likely wasn’t.
Thanks for the quality summary. I finally opened this post after a couple days of ignoring it in my doomscrolling, because I thought there would be nothing new in it vs. other recent posts on FTX. But I found this critique actually gave me some new things to think about.
I think it’s a good question, but I’m unsure whether a public discussion calling out names is the right way to go. And I think it might be net negative.
On the one hand, publicly calling people out could raise red flags about potential bad actors in the EA community early and in a transparent way. On the other hand, it could lead to witch hunts, false positives, etc. You can also imagine that whatever bad actors are in the community would start retaliating and listing good actors here who are their enemies, so it could cause a lot of confusion and infighting. (Think about the way Donald Trump uses public discourse.)
I think private discussions about this with others you trust are probably a good idea, to help guide personal decisions about who else to trust, work for, donate to etc. And this issue might make more decentralization actionable. For example, to mitigate bad actor risk, we might just have a background expectation that there is always X% chance that any individual in the EA community—even someone who is currently beloved—actually turns out to be a bad actor. And so no individual in the community should be responsible for more than Y% of the money, power, reputation etc. of the community to give EA more resilience against the inevitability of bad actors popping up here and there.
There may be other systemic ways to manage the risk too without making publicly outing suspected bad actors a regular thing. I’m also still open to the idea that public bad actor callouts might be a good idea, but I think it’s a really delicate thing and I’d like to see a convincing argument/plan for how to make the discussion be productive before I would support it.
Commending Habryka for willing to share about these things. It takes courage and I think reflections/discussions like this could be really valuable (perhaps essential) to the EA community having the kind of reckoning about FTX that we need.
Maybe, but not so clear. He may have cared about consequentialist utilitarianism but just faked the deontology part.
Thanks, I appreciate this and bet some others will too.
There was a post Linch did earlier this year about their experience as a junior grantmaker in EA. It had an interesting part about conflicts of interest:
4. Conflicts of Interests are unavoidable if you want to maximize impact as a grantmaker in EA.
a. In tension with the above point, the EA community is just really small, and the subcommunities of fields within it (AI safety, or forecasting, or local community building) are even smaller.
b. Having a CoI often correlates strongly with having enough local knowledge to make an informed decision, so (e.g) if the grantmakers with the most CoIs in a committee always recuse themselves, a) you make worse decisions, and b) the other grantmakers have to burn more time to be more caught up to speed with what you know.
c. I started off with a policy of recusing myself from even small CoIs. But these days, I mostly accord with (what I think is) the equilibrium: a) definite recusal for romantic relationships, b) very likely recusal for employment or housing relationships, c) probable recusal for close friends, d) disclosure but no self-recusal by default for other relationships.
d. To avoid issues with #3, I’ve also made a conscious effort to do more recusals in reverse: that is, I’ve made a conscious effort to avoid being too friendly with grantees.
It makes sense to me what they were saying, even though it’s troubling, because conflicts of interest are a problem.
It seems natural that more due diligence and organizational safeguards (like avoiding CoIs) will be some of the lessons learned from the FTX disaster. This could be an opportunity to prioritize such things more highly to improve EA’s worst-case preparedness (e.g. corruption) , even at the expense of some impact/efficiency gains from having the best experts everywhere when things are going well.
That’s true but hiding karma is an important thing for reducing bias (e.g. people being inclined to upvote something because it’s already upvoted) and mental health. So it would be a great thing to have has a built-in feature (and perhaps even the default) so that it’s accessible for many more users.
Karma scores are still important when it’s hidden for sorting content by quality/popularity, it’s just more of a behind-the-scenes thing.
(Thanks for all your great work on these forums.)
LessWrong has no such feature built-in that I’m aware of. Though there are external tools people have created to hide karma.
TIL that the code for LessWrong/EA Forum is open source.
The extension for blinding karma and author names has been a game changer for me. Massively improves my forum experience. Strong upvote, it’d be great to have these as native features so that they are much more accessible and others can enjoy the debiasing and mental health benefits.
I tend to prefer blinding karma instead of the author name. But they’re both useful at different times. I think adding both and making independently controllable would be a huge step forward. Then the community can experiment with favorite configurations for different contexts.
Thanks to the OP, I’ve been meaning to post about this for months. And thanks to the forum devs who are doing a ton of work behind the scenes to make everything on these forums possible.
Strongly agree about the vote counts, I’ve been using a browser plugin to hide them for a couple months now. I think it should be a forum option and probably the default.
Stuart Russell is probably the most prominent example.
I think Dan Hendryks is doing good work in this area as well, as well as a bunch of people on the AI alignment team at DeepMind.
But yea, it’d be great if a lot more ML researchers/engineers engaged with the AI x-risk arguments and alignment research.
It’s short for the Alignment Forum: https://www.alignmentforum.org/
The U.S. National Institute of Standards and Technology (“NIST”) is developing an AI Risk Management Framework.
Just a sidenote for anyone interested in this. There is an existing effort from some folks in the AI safety community to influence the development of this framework in a positive direction. See Actionable Guidance for High-Consequence AI Risk Management (Barett et al. 2022).
I see what you mean, but if you value cause prioritization seriously enough, it is really stifling to have literally no place to discuss x-risks in detail. Carefully managed private spaces are the best compromise I’ve seen so far, but if there’s something better then I’d be really glad to learn about it.
Thanks for explaining, but who are you considering to be the “regulator” who is “captured” in this story? I guess you are thinking of either OpenPhil or OpenAI’s board as the “regulator” of OpenAI. I’ve always heard the term “regulatory capture” in the context of companies capturing government regulators, but I guess it makes sense that it could be applied to other kinds of overseers of a company, such as its board or funder.