Formerly a software engineer at Google, now I’m doing independent AI alignment research.
Because of my focus on AI alignment, I tend to post more on LessWrong and AI Alignment Forum than I do here.
I’m always happy to connect with other researchers or people interested in AI alignment and effective altruism. Feel free to send me a private message!
Will the AI alignment Slack continue to run?
Thanks JJ and everyone who has worked on AISS for all your great work!
Peter Singer and Tse Yip Fai were doing some work on animal welfare relating to AI last year: https://link.springer.com/article/10.1007/s43681-022-00187-z It looks like Fai at least is still working in this area. But I’m not sure whether they have considered or initiated outreach to AGI labs, that seems like a great idea.
I place significant weight on the possibility that when labs are in the process of training AGI or near-AGI systems, they will be able to see alignment opportunities that we can’t from a more theoretical or distanced POV. In this sense, I’m sympathetic to Anthropic’s empirical approach to safety. I also think there are a lot of really smart and creative people working at these labs.
Leading labs also employ some people focused on the worst risks. For misalignment risks, I am most worried about deceptive alignment, and Anthropic recently hired one of the people who coined that term. (From this angle, I would feel safer about these risks if Anthropic were in the lead rather than OpenAI. I know less about OpenAI’s current alignment team.)
Let me be clear though: Even if I’m right above and massively catastrophic misalignment risk one of these labs creating AGI is ~20%, I consider that very much an unacceptably high risk. I think even a 1% chance of extinction is unacceptably high. If some other kind of project had a 1% chance of causing human extinction, I don’t think the public would stand for it. Imagine some particle accelerator or biotech project had a 1% chance of causing human extinction. If the public found out, I think they would want the project shut down immediately until it could be pursued safely. And I think they would be justified in that, if there’s a way to coordinate on doing so.
A key part of my model right now relies on who develops the first AGI and on how many AGIs are developed.
If the first AGI is developed by OpenAI, Google DeepMind or Anthropic—all of whom seem relatively cautious (perhaps some more than others) - I put the chance of massively catastrophic misalignment at <20%.
If one of those labs is first and somehow able to prevent other actors from creating AGI after this, then that leaves my overall massively catastrophic misalignment risk at <20%. However, while I think it’s likely one of these labs would be first, I’m highly uncertain about whether they would achieve the pivotal outcome of preventing subsequent AGIs.
So, if some less cautious actor overtakes the leading labs, or if the leading lab who first develops AGI cannot prevent many others from building AGI afterward, I view there’s a much higher likelihood of massively catastrophic misalignment from one of these attempts to build AGI. In this scenario, my overall massively catastrophic misalignment risk is definitely >50%, and perhaps closer to the 75%~90% range.
You’re right—I wasn’t very happy with my word choice calling Google the ‘engine of competition’ in this situation. The engine was already in place and involves the various actors working on AGI and the incentives to do so. But these recent developments with Google doubling down on AI to protect their search/ad revenue are revving up that engine.
It’s somewhat surprising to me the way this is shaking out. I would expect DeepMind and OpenAI’s AGI research to be competing with one another*. But here it looks like Google is the engine of competition, less motivated by any future focused ideas about AGI more just by the fact that their core search/ad business model appears to be threatened by OpenAI’s AGI research.
*And hopefully cooperating with one another too.
I think it’s not quite right that low trust is costlier than high trust. Low trust is costly when things are going well. There’s kind of a slow burn of additional cost.
But high trust is very costly when bad actors, corruption or mistakes arise that a low trust community would have preempted. So the cost is lumpier, cheap in the good times and expensive in the bad.
(I read fairly quickly so may have missed where you clarified this.)
If anyone consults a lawyer about this or starts the process with FTXrepay@ftx.us , it could be very useful to many of us if you followed up here and shared what your experience of the process was like.
I’m a long-time fan of renting over buying. I’ve been happily renting apartments since I started living on my own around ~2006. I’ve never owned a place and don’t have any wishes or plans to. I skimmed the John Halstead post you linked to—a lot of his overall points have been motivations for me as well.
Last time I really looked into this (it’s been a few years), the price-to-rent ratio varied a lot depending on the kind of place you live in. Generally if you lived in a major city, the ratio greatly favored renters. But in some lower-populated / suburban or rural areas, the ratio favored buyers: https://www.mrmoneymustache.com/2015/07/27/rent-vs-buy/ This is just one factor, but one I haven’t found many people are aware of and so I like to bring up.
I do think it’s a very personal question, and buying a home makes sense for some people. But I think it’s generally overrated, and at least in North America I see way more people mistakenly buying property than mistakenly renting. But everyone needs to think through it for themselves. Some things that might make me think someone could be a good fit for ownership are a) if they like spending a lot of time fixing/working on their home, b) they want to be a landlord, c) they really don’t like using parks/public areas and want a lot of private space, and/or d) they don’t mind a long commute.
I’m surprised you were putting such high odds on it being a mistake at this point (even before the arrest). From my understanding (all public info), FTX’s terms of service agreed that they would not touch customer funds. But then FTX loaned those funds to Alameda, who made risky bets with them.
IANAL but this seems to me like pretty clear case of fraud from FTX. I didn’t think any of those aspects of the story were really disputed, but I have not been following the story as closely in the past week or so.
Will all the results of the survey be shared publicly on EA Forum? I couldn’t find mention about this in the couple announcements I’ve seen for this survey.
It looks like at least some of the 2020 survey results were shared publicly. [1, 2, 3] But I can’t find 2021 survey results. (Maybe there was no 2021 EA Survey?)
Thanks for the link and highlights!
Sam claims that he donated to Republicans: “I donated to both parties. I donated about the same amount to both parties (...) That was not generally known (...) All my Republican donations were dark (...) and the reason was not for regulatory reasons—it’s just that reporters freak the fuck out if you donate to Republicans [inaudible] they’re all liberal, and I didn’t want to have that fight”. If true, this seems to fit the notion that Sam didn’t just donate to look good (i.e. he donated at least partly because of his personal altruistic beliefs)
What do you mean that this donation strategy would be from Sam’s “personal altruistic beliefs”? Donating equally to both political parties has been a common strategy among major corporations for a long time. It’s a way for them to push their own agenda in government. It’s generally an amoral self-interested strategy, not an altruistic one.
I am a big fan of gratitude practice. I try to write a little in a gratitude journal most nights, which has helped my overall state of mind since I started doing it. I would recommend anybody to try it, including people involved in EA. And I’m glad you suggested it, as a little gratitude during a crisis like this can be especially helpful.
I have some reservations about posting things I’m grateful for publicly on this forum though. Gratitude can be a bit vulnerable, and this forum has more eyes on it than usual lately. Posting to a community about why you’re thankful for that community could also be misinterpreted as being obsequious or virtue signalling. I think most of the benefits of gratitude practice can be enjoyed privately or with someone you trust, but if other people felt inclined to share their gratitude here, I would probably enjoy reading it and not be judgmental. And I may change my mind later and post some of that here as well :)
I would probably more excited about this thread if the forum had a feature to post comments anonymously. I don’t see any downside to an anonymous public gratitude thread, but I’m probably too lazy to create an anonymous account just for that purpose.
Ultimately this was a failure of the EA ideas more so than the EA community. SBF used EA ideas as a justification for his actions. Very few EAs would condone his amoral stance w.r.t. business ethics, but business ethics isn’t really a central part of EA ideas. Ultimately, I think the main failure was EAs failing to adequately condemn naive utilitarianism.
So I disagree with this because:
It’s unclear whether it’s right to attribute SBF’s choices to a failure of EA ideas. Following SBF’s interview with Kelsey Piper and based on other things I’ve been reading, I don’t think we can be sure at this point whether SBF was generally more motivated by naive utilitarianism or by seeking to expand his own power and influence. And it’s unclear which of those headspaces led him to the decision to defraud FTX customers.
It’s plausible there actually were serious ways that the EA community failed with respect to SBF. According to a couple accounts, at least several people in the community had reason to believe SBF was dishonest and sketchy. Some of them spoke up about it and others didn’t. The accounts say that these concerns were shared with more central leaders in EA who didn’t take a lot of action based on that information (e.g. they could have stopped promoting Sam as a shining example of an EA after learning of reports that he was dishonest, even if they continued to accept funding from him). [1]
If this story is true (don’t know for sure yet), then that would likely point to community failures in the sense that EA had a fairly centralized network of community/funding that was vulnerable, and it failed to distance itself from a known or suspected bad actor. This is pretty close to the OP’s point about the EA community being high-trust and so far not developing sufficient mechanisms to verify that trust as it has scaled.
--
[1]: I do want to clarify that in addition to this story still not being unconfirmed, I’m mostly not trying to place a ton of blame or hostility on EA leaders who may have made mistakes. Leadership is hard, the situation sounds hard and I think EA leaders have done a lot of good things outside of this situation. What we find out may reduce how much responsibility I think the EA movement should put with those people, but overall I’m much more interested in looking at systemic problems/solutions than fixating on the blame of individuals.
Can you say a bit more about what you think EA has lost that makes it valuable?
Thanks for clarifying. That helps me understand your concern about the unilateralist’s curse with funders acting independently. But i don’t understand why the OP proposal of evaluating/encouraging funding diversification for important cause areas would exacerbate it. Presumably those funders could make risky bets regardless of this evaluation. Is it because you think it would bring a lot more funders into these areas or give them more permission to fund projects that they are currently ignoring?
Was it this post by chance? https://forum.effectivealtruism.org/posts/AbohvyvtF6P7cXBgy/brainstorming-ways-to-make-ea-safer-and-more-inclusive This one seems to be on a very similar topic. But it has a different name so it’s probably not the same one but possibly Richard revised the title at some point.
Thanks for explaining, but who are you considering to be the “regulator” who is “captured” in this story? I guess you are thinking of either OpenPhil or OpenAI’s board as the “regulator” of OpenAI. I’ve always heard the term “regulatory capture” in the context of companies capturing government regulators, but I guess it makes sense that it could be applied to other kinds of overseers of a company, such as its board or funder.
I’ve also been very upset since the FTX scandal began, and I love this community too. I think you’re right that EA will lose some people. But I am not so worried the community will collapse (although it’s possible that ending the global EA community could be a good thing). People’s memories are short, and all things pass. In one year, I would be willing to bet you there will still be lots of (and still not enough!) good people working on and donating to important, tractable, and neglected causes. There will still be an EA Forum with lively debates happening, and that arguments about FTX will by that point make up a small fraction of the content. There will be still new people discovering EA and getting inspired by the potential to increase their positive impact in the world.
To be sure, I do think we should be worried* about the future of EA right now. But more in the sense of worried about whether EA can remain true to its core values and ideals going forward than about whether it can survive in some form.
--
*Note that when I say “we should be worried”, I actually mean “we should be putting careful attention toward” rather than “we should be consumed by anxiety about”. Be kind to yourself, and if you’re feeling more of the latter, now may be a good time to double down on self-care.
Open Phil claims that campaigns to make more Americans go vegan and vegetarian haven’t been very successful. But does this analysis account for immigration?
If people who already live in the US are shifting their diets, but new immigrants skew omnivore, a simple analysis could easily miss the former shift because immigration is fairly large in the US.
Source of Open Phil claim at https://www.openphilanthropy.org/research/how-can-we-reduce-demand-for-meat/ :