Kinda pro-pluralist, kinda anti-Bay EA.
I have come here to extend the principle of charity to bad criticisms of EA and kick ass. And I’m all out of charity.
(my opinions are fully my own, and do not represent the views of any close associates or the company I work for)
Sorry Oli, but what is up with this (and your following) comment?
From what I’ve read from you[1] seem to value what you call “integrity” almost as a deontological good above all others. And this has gained you many admirers. But to my mind high integrity actors don’t make the claims you’ve made in both of these comments without bringing examples or evidence. Maybe you’re reacting to Sean’s use of ‘garden variety incompetence’ which you think is unfair to Bostrom’s attempts to tow the fine line between independence and managing university politics but still, I feel you could have done better here.
To make my case:
When you talk about “other organizations… become a hollow shell of political correctness and vapid ideas” you have to be referring to CSER & Leverhulme here right, like it’s the only context that makes sense.
If not, I feel like that’s very misleadingly phrased.
But if it is, then calling those organisations ‘hollow shells’ of ‘vapid ideas’ is like really rude, and if you’re going to go there at least have the proof to back it up?
Now that just might be you having very different politics from CSER & Leverhulme people. But then you say “he [Bostrom] didn’t compromise on the integrity of the institution he was building”, which again I read as you directly contrasting against CSER & Leverhulme—or even Sean in person.
Is this true? Surely organisation can have different politics or even have worse ideas without compromising on integrity?
If they did compromise on integrity, feels like you should share what those are.
If it is directed at Sean personally, that feels very nasty. Making assertions about someone’s integrity without solid proof isn’t just speculation, it’s harmful to the person and also poor ‘epistemic hygiene’ for the community at large.
You say “the track record here speaks quite badly to Sean’s allocation of responsibility by my lights”. But I don’t know what ‘track record’ your speaking about here. Is it at FHI? CSER & Leverhulme? Sean himself?
Finally, this trio of claims in your second comment really rubbed me[2] the wrong way. You say that you think:
“CSER and Leverhulme, which I think are institutions that have overall caused more harm than good and I wish didn’t exist”
This is a huge claim imo. More harm than good? So much so that you wish it didn’t exist? With literally no evidence apart from it being your opinion???
“Sean thought were obvious choices were things that would have ultimately had long-term bad consequences”
I assume that this is about relationship management with the university perhaps? But I don’t know what to make of it because you don’t say what these ‘obvioous choices are’, or why you think they’re so likely to have bad consequences
“I also wouldn’t be surprised if Sean’s takes were ultimately responsible for a good chunk of associated pressure and attacks on people’s intellectual integrity”
This might be the worst one. Why are Sean’s takes responsible? What were the attacks on people’s integrity? Was this something Sean did on purpose?
I don’t know what history you’re referring to here, and the language used is accusatory and hostile. It feels really bad form to write it without clarifying what you’re referring to for people (like me) who don’t know what context you’re talking about.
Maybe from your perspective you feel like you’re just floating questions here and sharing your personal perspective, but given the content of what you’ve said I think it would have been better if you had either brought more examples or been less hostile.
Many people find the Forum anxiety inducing because of the high amount of criticism. So, in the spirit of Giving Season, I’m going to give some positive feedback and shout-outs for the Forum in 2023 (from my PoV). So, without further ado, I present the 🎄✨🏆 totally-not-serious-worth-no-internet-points-JWS-2023-Forum-Awards: 🏆✨🎄[1]
Best Forum Post I read this year:
10 years of Earning to Give by @AGB: A clear, grounded, and moving look at what it actually means to ‘Earn to Give’. In particular, the ‘Why engage?’ section really resonated with me.
Honourable Mentions:
Why I’m still going out to bat for the EA movement post FTX by @Gemma Paterson: This post was a reflection on EAs tough year which was similar to my own journey, but was also able to make a case for why EA is worth fighting for.
Third Wave Effective Altruism by @Ben_West: A short post, but one that came at exactly the right time. I think this is a really important topic for EA to be thinking about right now and look forward to more posts inspired by it next year.
The Seeker’s Game – Vignettes from the Bay by @Yulia: I really enjoyed both the human perspective of this post, and the style with which it was written.
Best Sequence I read this year:
The CURVE Sequence: Causes and Uncertainty: Rethinking Value in Expectation from Rethink Priorities: This sequence was utterly brilliant. All its posts are well-written and led to good discussion from the community. I particularly want to shout out Arvo’s piece that stress-tested what EV from x-risk looks like under specific assumptions.
EA Forum Posters of the Year:
@titotal - I really value titotal’s contributions to the Forum. Even though I sometimes disagree I always learn a lot by thinking about where and why I do so, and I hope they stick around so I can continue to learn from them.
@Holly_Elmore - Similarly to Titotal, while I don’t always agree with Holly she just makes really solid points that always positively add to the discussion. Often she’s able to clearly find the crux of a debate and ask the key questions.
@Ivy Mazzola - Whenever I’ve seen Ivy post on the Forum, even on heated topics, she always approaches them with integrity and grace that I really wanted to give recognition to.
@David Mathers - When David discusses ideas in the Forum he consistently does so in a good faith and truth-seeking way that really shows engagement on the Forum at its best.
Non-Forum EA Poasters of the Year:
David Thorstad (on his blog) - I’ve really appreciated David’s blog this year. I don’t always agree, and you can find me in the comments pushing back sometimes, I’ve found his content high quality overall, especially on Existential Risk and the Time of Perils.
Agustín (on Twitter) - Agus has been really fighting the good fighting in the Twitter trenches recently, on all sorts of topics (though sadly on the terrible state of AI discourse). Somehow he manages to translate complicated EA ideas clearly, convincingly, and remain calm in the face of less-than-charitable interlocutors.
Nathan Young (on Twitter) - Nathan’s commitment to honest exploration of his beliefs, dialogue with those who disagree, and finding mechanisms of belief resolution has been constantly inspiring.
Congrats to the winners! Feel free to give them more plaudits in the comments or just shout out your own Forum favourites from this year! I wish everyone who’s reading this every success in the year to come and I look forward to reading more good content on the Forum in 2024!
Note that all of the award categories and choices are arbitrary, this is an intensely vibes-based post
After listening, here are my thoughts on the podcast (times refer roughly to youtube timestamps):[1]
Recap[2]
Neither Sam nor Will think less of EA principles because of the FTX collapse. Sam says so in the intro (4:40), and towards the end mentions how unhappy he was to see people “dancing on the grave of EA” (59:17) and likens the simple rejection of EA because of SBF to the rejection of vegetarianism because of Hitler’s diet (57:10). Will also agrees that FTX hasn’t made him reconsider EA principles (56:24), and says that the FTX debacle happened in spite of or not because of EA (54:52).[3]
On whether the beliefs of EA are responsible, Will says that longtermism is not the reason, since one could justify maximalist and extreme behaviours for more near-termist concerns as well (1:07:32). He also thinks that EA communicators consistently said that the ends do not justify the means (54:58).
Will is concerned about, in attempting to get into the mind of SBF and explain his behaviour, there’s a danger of looking to exculpatory of him and wants to make it clear that the collapse of FTX caused huge damage (14:15) and it’s clear in the following section that’s he’s read a lot of the evidence submitted to the case, including Twitter messages from people who lost dearly-needed savings in FTX.
However, some of the podcast does seem to come across that way sometimes. He recounts that his impression of SBF since 2012 was that of “a very thoughtful, incredibly morally motivated person” (37:29), and reflections make him seem quite credulous of the explanations he was given (40:23).[4] He notices that his reflections are sounding defensive of SBF about halfway through the podcast (42:01).
The Sam Harris subreddit has had an overwhelmingly negative reaction to the podcast. Many people there are basically Sam and Will of either continuing to be duped, or running PR for SBF. Of course this is a self-selected-overly-online community, but I think it underscores the anger and incredulity over what happened here, and how very careful and tactful messaging needs to be over this.
Sam offers two major theories as to why SBF did what he did (11:52):
It was a conscious fraud and long-con from the beginning, where SBF used EA as a cover to gain influence/power/good-will
Due to SBF’s beliefs about ethics and approach to risk/probability, he was placing a series of bets that were eventually likely to fail and eventually blew up in his face.
Will actually rejects the above (18:17), and doesn’t think SBF’s behaviour can be framed as a calculated decisions because it simply doesn’t make sense (27:31), and that the ‘long-con’ explanation can’t explain why FTX wanted regulation and press attention, and even why it was associated with EA (24:10). Instead, he believes the main culprit was SBF’s Hubris (28:11). He also notes SBF’s unusual risk tolerance (23:04) and suggests the cause was not a risk calculation gone wrong but an overall attitude to risk (29:47).
Will seemingly derives a lot of his explanations from the work of Eugene Soltes, a Harvard Business School professor, especially his book Why They Do It, which he references throughout the podcast.
While Will says he rejects both of Sam’s original propositions (theory 1 and 2), sometimes he shades into backing theory 2, such as when he says that SBF really did believe in the ideas of EA (49:46)
While Sam is more agnostic about what exactly was driving SBF, both make mention of how seriously SBF took ideas throughout. Will notes that SBF found the idea of earn-to-give compelling way before FTX existed (8:16), Sam references SBF’s interview with Tyler Cowen where he said he’d accept the St Petersburg paradox forever as a red flag (21:22). He raises SBF’s neuro-atypical background as another potential red flag (35:31) but Will didn’t seem to buy it and the conversation moves on from that pretty quickly.
The discussion ends with a discussion about the effects of the whole saga on EA and what changes there have been. Will says that there has been huge harm to EA and that many now think ill of it (54:23)
Will thinks that EA should have been, and should be more decentralised (58:24). He also had legal guidance (or instruction?) telling him not to share information or his perspective during the internal investigation (58:20). The combination of these meant that he wasn’t able to provide guidance to the EA community that he wanted to during the crisis(58:47).
Will often refers to GWWC and the 9000+ pledgers as normal people living normal lives (1:03:23), and that EAs are “pretty normal people” that are willing to put their money and time where their mouth is (1:05:31)
Will mentions the FTX collapse was incredibly hard on him, that it was the hardest year and a half of his life, and that he nearly lost motivation for the EA project altogether (1:16:44).
He thinks that the strongest critique of longtermism he’s come across since WWOTF was published isn’t the pascal’s mugging type stuff, but that AI risks might be much more of a near-term threat than a long-term one (1:10:44), that AI x-risk concerns have been increasingly vindicated in recent years (1:11:38) and that intelligence explosion arguments of the IJ Good variety are holding up and deserve consideration (1:14:57)
Personal Thoughts
So there’s not an actual deep-dive into what happened with SBF and FTX, and how much Will or figures in EA actually knew. Perhaps the podcast was trying to cover too much ground in 80 minutes, or perhaps Sam didn’t want to come off as too hostile of a host? I feel like both a talking about the whole thing at an oddly abstract level, and not referencing the evidence that’s come out in court.
While I also agree with both that EA principles are still good, and that most EAs are doing good in the world, there’s clearly a connection between EA—or at least a bastardised, naïvely maximalist view of it—and SBF. The prosecution and the judge seemed to take the view that SBF was a high risk of doing the same or a similar thing again in the future, and that he has not shown remorse. This makes sense if Sam was acting in the way he did because he thought he was doing the right thing, and the fact that it was an attitude rather than a ‘rational calculation’ doesn’t make it less driven by ideas.
So I think that’s where I’ve ended up on this (I’m not an expert on financials, or what precise laws FTX broke, and how their attempted scheme operated or how often they were brazenly lying for. Feels like those with an outside view are pretty damn negative on SBF). I think trying to fix the name ‘EA’ or ‘not EA’ to what SBF and the FTX team believed is pretty unhelpful. I think Ellison and SBF had a very naïve, maximalist view of the world and their actions. They believed they had special ability and knowledge to act in the world, and to break existing rules and norms in order to make the world better if they saw it, even if this incurred high risks, if their expectation was that it would work out in EV terms. An additional error here, and perhaps where the ‘Hubris’ theory does play in, is that there was no error mechanism to correct these beliefs. Even after the whole collapse, and a 25-year sentence, it still seems to me that SBF thinks he made the ‘right’ call and got unlucky.
My takeaway is that this cluster of beliefs[5] is dangerous and the EA community should develop an immune system to reject these ideas. Ryan Carey refers to this as ‘risky beneficentrism’, and I think part of ‘Third-Wave’ EA should be about rejecting this cluster of ideas, making this publicly known, and disassociating EA from the individuals, leaders, or organisations who still hold on to it in the aftermath of this entire debacle.
For clarity Sam refers to Sam Harris, and SBF refers to Sam Bankman-Fried
Not necessarily in order, I’ve tried to group similar points together
I think this makes some sense if you view EA as a set of ideas/principles, less so if you view EA as a set of people and organisations
During this section especially I kinda wanted to shout at my podcast when Will asked rhetorically “was he lying to me that whole time?” the answer is yes Will, it seems like they were. The code snippets from Nishad Singh and Gary Wang that the prosecution shared are pretty damning, for example.
See the following link in the text to Ryan Carey’s post. But I think the main dangerous ideas to my mind are:
1) Naïve consequentialism
2) The ability and desire to rapidly change the world
3) A rejection of existing norms and common-sense morality
4) No uncertainty about whether the values above or the empirical consequences of the actions
5) Most importantly, no feedback or error correction mechanism for any of the above.
This is a critically important point to hold in mind if the reason for the move seems to be due to safety concerns as opposed to personal malpractice/deceiving the board[1]
I don’t know what the hell happened. I guess further clarifications on the decision-making process and corporate landscape will be known tomorrow or, more likely, early next working week
I’ve voiced concerns before that EA is unaware that it can be drawn into ‘one-way fights’ sometimes, and this feels like another such moment. The Silicon Valley tech-twitter scene[2] has exploded over this, and so far EA is not coming out well in their eyes from what I can see. I think the days of “e/acc” being a meme movement are rapidly drawing to a close, and EA might find itself in a hostile atmosphere in what used to be one of the most EA-friendly places in the world.
Again, early speculations, but be careful out there Bay-Area EAs. Keep your wits about you.
edit: As always, disagree/downvoters, would be good to hear why you disagree, as I’m not sure what I’ve written below merits either a disagree and especially not a downvote.
Thanks for sharing your thoughts Rebecca.
I do find myself wishing that some of these discussions from the core/leadership of EA[1] were less vague. I noticed this with Habrkya’s reaction to the recent EA column in the Washington Post—where he mentions ‘people he’s talked to at CEA’. Would be good to know who those people at CEA are.
I accept some people are told things informally, and in confidence etc., but it would seem to be useful to have as much as is possible/reasonable in the public domain, especially since these discussions/decisions seem to have such a large impact on the rest of the community in terms of reputational impact, organisational structure and hiring, grantmaking priorities and decisions etc.
For example, I again respect you said that your full thoughts would be ‘highly costly’ to share, but it’d be enlightening to know which members of the EV board you disagreed with so much that you felt you had to resign. If you can’t share that, knowing why you can’t share that. Or if not that, knowing what the concrete issues were. If you allege that there were “extensive and significant mistakes made which have not been addressed” and that these mistakes “make me very concerned about the amount of harm EA might do in the future” then I really want to know what these mistakes were concretely and who made/is making them. I think the vagueness is another sign that EA’s healing process post-FTX still has a way to go.[2]
Above all though, I hope you’re doing well, and would be happy to have an individual conversation if you think that would be useful, or if you aren’t willing to share things on the Forum.
I mean it is literally accurate, the two EAs previously on the board are gone.[1] It’s part of the story, though not all of it. Honestly @Fermi–Dirac Distribution if I were you I’d consider changing it back, but obviously your call.
These individuals in question are also on the boards of important EA organisations too. And that feels clearly germane to the Forum. Their board performance on this issue surely has implications for their stewardship of these organisations and by proxy their influence on EA as a whole
We don’t have the full picture of what happened. But some things are becoming clearer. And it’s very difficult to separate ‘waiting for information’ from ‘pleading no-contest in the court of public opinion’ which is what the board seem to have done
In principal, given this story who knows if that’ll last
Just want to register some disagreement here about the name change, to others in this thread and Will (not just you Gemma!). In rough order of decreasing importance:
I really don’t like the name MoreGood. It’s a direct callback to LessWrong. I don’t want to have to endorse LW to endorse EAF, or EA more generally, or the causes we care about, and this name change would signal that. Yes, there’s some shared intellectual history, but I don’t think LW-rationalism is inherent to or necessary for EA.
For people new to/interested in EA, they’ll probably search for “EA” or “Effective Altruism”. They wouldn’t know the rebrand or name change unless there was a way to preserve it for SEO.
I think EA Forum is fine, and it is the major place for EA discussion online at the moment. I don’t think it’s that unrepresentative of EA?
Any other online forum will also be skewed towards those online or ‘extremely online’. I think EA Twitter is much worse for this than the Forum.
In the spirit of do-ocracy, there’s no reason that other people can’t set up an alternative forum with a different focus/set of norms, though it will probably suffer from the network effects that make it difficult to challenge social media incumbents.
I do accept it was just a small draft suggestion though.
Is it necessary to relitigate Wytham Abbey in this thread? The OP suggests that Claire’s position is unusual because:
While it is common for funders to serve on boards, it is not necessarily best practice.
Not because Wytham Abbey was a poor purchase. I understand that the community is very split on this, and the take you’ve shared is a good representation of the ‘pro’ side, but it seems to me that you’ve used this to re-open the WA discussion here rather than discuss whether EVF should consider changing the composition/structure of its boards, and what the merits of that would be, which I think is the charitable object-level discussion that the OP wants to have here imo.
The HLI discussion on the Forum recently felt off to me, bad vibes all around. It seems very heated, not a lot of scout mindset, and reading the various back-and-forth chains I felt like I was ‘getting Eulered’ as Scott once described.
I’m not an expert on evaluating charities, but I followed a lot of links to previous discussions and found this discussion involving one of the people running an RCT on Strongminds (which a lot of people are waiting for the final results of) who was highly sceptical of SM efficacy. But the person offering counterarguments in the thread seems to be just as valid to me? My current position, for what it’s worth,[1] is:
the initial Strongminds results of 10x cash transfer should raise a sceptical response. most things aren’t that effective
it’s worth there being exploration of what the SWB approach would recommend as the top charities (think of this as trying other bandits in a multi-armed bandit charity evaluation problem)
it’s very difficult to do good social science, and the RCT won’t give us dispositive evidence about the effectiveness of Strongminds (especially at scale), but it may help us update. In general we should be mindful of how far we can make rigorous empirical claims in the social sciences
HLI has used language too loosely in the past and overclaimed/been overconfident, which Michael has apologised for, though perhaps some critics would like a stronger signal of neutrality (this links to the ‘epistemic probation’ comments)
GiveWell’s own ‘best guess’ analysis seems to be that Strongminds is 2.3x that of GiveDirectly.[2] I’m generally a big fan of the GiveDirectly approach for reasons of autonomy—even if Strongminds got reduced in efficacy to around ~1x GD, it’d still be a good intervention? I’m much more concerned with what this number is than the tone of HLIs or Michael’s claims tbh (though not at the expense of epistemic rigour).
The world is rife with actively wasted or even negative action, spending, and charity. The integrity of EA research, and holding charity evaluators to account is important to both the EA mission and EAs identity. But HLI seems to have been singled out for very harsh criticism,[3] but so much of the world is worse.
I’m also quite unsettled by a lot of what I call ‘drive-by downvoting’. While writing a comment is a lot more effort than clicking to vote on a comment/post, I think the signal is a lot higher, and would help those involved in debates reach consensus better. Some people with high-karma accounts seem to be making some very strong votes on that thread, and very few are making their reasoning clear (though I salute those who are in either direction).
So I’m very unsure how to feel. It’s an important issue, but I’m not sure the Forum has shown itself in a good light in this instance.
And I stress this isn’t much in this area, I generally defer to evaluators
On the table at the top of the link, go to the column ‘GiveWell best guess’ and the row ‘Cost-effectiveness, relative to cash’
Again, I don’t think I have the ability to adjudicate here, which is part of why I’m so confused.
I wasn’t really impressed or persuaded by it to be honest, actually the more I read it the worse it got. I agree with you that he’d make a very interesting 80k podcast guest if he and Rob got the chance to go at it hammer and tongs, but I don’t think the current team are up for adversarial conversations (even well-mannered ones) that it would probably turn out to be.
Poverty is No Pond is right to point out the vast complexity of understanding the potential impacts that aid interventions might have, but I think this empirical and moral cluelessness applies to all actions all the time, and he doesn’t seem to apply them to his friend Aaron. There’s a funny section in the linked piece where he excoriates GiveWell for their hedging language—yet in Poverty is No Pond he laments that more of those in the aid space aren’t more humbly agnostic. Even if we are clueless about how the world works, we must still act in one way or another. He doesn’t seem to give any evidence for his suggestions about from epistemic nihilism and personal/lived experience.
I don’t think pointing out that bednets can be used for fishing is a very strong argument against them, and I don’t necessarily trust Wenar’s assessment of the evidence here when compared to AMF, GiveWill, and others in the space. Just as he uses the “It’s difficult to get a man to understand something, said Upton Sinclair, when his salary depends on his not understanding it.” line to criticse Ord, I can use it to criticise him here, and much of his view on the value of quantification and empirical research.
A lot of the piece is written in a very hostile, and honestly child-like manner to me. The parts about MacAskill are blazingly hostile, to the point that it comes off as much more hit-piecy than truth-seeking. Like the whole bit about ‘altruism’ just seems to underline how little Wenar really understands about EA. I don’t think the SBF is insightful or adds much to what’s been reported on elsewhere.
By the bit it got to longtermism it started to look increasingly ahistorical, pure conjecture, and vitriol, and I lost interest in reading beyond a quick skim. Maybe I’ve been harsh, but I think that it probably deserves to be treated harshly.
Excellent work. I’ve run out of good words to say about this Sequence. Hats off to the RP team.
I sympathise with your NB at the beginning, but to be honest in the absence of specific examples or wider data it’s hard for me to ground this criticism or test its validity. Ironically, it’s almost as if this post is too fundamental instead of action-guiding for me.
Doesn’t mean you’re wrong per se, but this post is almost more of a hypothesis than an argument.
Again, a fan of you and your approach David, but I think you underestimate just how hostile/toxic Émile has been toward all of EA. I think it’s very fair to substitute one for the other, and it’s the kind of thing we do all the time in real, social settings. In a way, you seem to be emulating a hardcore ‘decoupling’ mindset here.
Like, at risk of being inflammatory, an intuition pump from your perspective might be:
It is possible that many complaints about Trump are true and also that Trump raises important concerns. I would not like to see personal criticism of Trump become a substitute for engagement with criticism by Trump.
I think many EAs view ‘engagement with criticism by Torres’ in the same way that you’d see ‘engagement with criticism by Trump’, that the critic is just so toxic/bad-faith that nothing good can come of engagement.
I want to register that my perspective on medium-term[1] AI existential risk (shortened to AIXR from now on) has changed quite a lot this year. Currently, I’d describe it as moving from ‘Deep Uncertainty’ to ‘risk is low in absolute terms, but high enough to be concerned about’. I guess atm I’d think that my estimates are moving closer toward the Superforecasters in the recent XPT report (though I’d still say I’m still Deeply Uncertain on this issue, to the extent that I don’t think the probability calculus is that meaningful to apply)
Some points around this change:
I’m not sure it’s meaningful to cleanly distinguish AIXR from other anthropogenic x-risks, especially since negative consequences of AI may plausibly increase other x-risks (e.g. from Nuclear War, biosecurity, Climate Change etc.)
I think in practice, the most likely risks from AI would come from deployment of powerful systems that have catastrophic consequences that are then rolled back. I’m think of Bing ‘Sydney’ here as the canonical empirical case.[2] I just don’t believe we’re going to get no warning shots.
Similary, most negative projections of AI don’t take into account negative social reaction and systematic human response to these events. You either assume we’ll get no warning shots and then get exterminated (‘sharp left turn’) or think that humanity is doomed to not co-operate (‘Moloch’). I think that the evidence suggests instead that societies and governments will react against increasing AI capability if they view it negatively, rather than simply stand by and watch it happen, which is what many AIXR seem to assume or at least imply to me.
I think the AIXR community underestimated the importance of AI Governance and engaging with politics. Instead of politicians/the public ‘not getting it’, they infact seem to have ‘got it’. The strategy of only letting people vetted to be smart enough think about AIXR seems to have been flawed, the same kind of thinking that led to thinking a ‘pivotal act’ strategy was viable.
In fact, I’ve been pleasantly surprised by how welcoming both politicians and the public have been to the Overton Window being opened. Turns out the median voter in a liberal democracy doesn’t like ‘let large co-orporations create powerful models with little understanding of their consequences without significant oversight’. I think people’s expectations of positive co-ordination on AI issues should have gone up this year, and conversely it should make your AIXR estimate go down unless you think relative alignment progress has declined even more.
While there have been an awful lot of terrible arguments against AIXR raised this year, some have been new and valuable to me. Some examples:
As titotal has written, I expect ‘early AGI’ will be a ‘buggy mess’. More specifically, I doubt it could one-shot solve ‘take over the world’ unless you assume god-like capability by assumption, instead of ‘more capable than humans in some important ways, but not invincible’.
Progress via ‘stack moar layers lol’ will plausibly slow down, or at least run into some severe issues.[3] The current hype-wave seems to be just following this as far as it will go with compute and data, rather than exploring alternative architectures that could be much more efficient.
I’m still unimpressed by how Frontier systems perform on tests such as The ARC Challenge which test hypothesis creativity and testing in a few-shot scenario (and on hidden data) without ‘ground truth’, as opposed to training on trillions-upon-trillions of training examples with masks and true labels.
Related to the above, I view creativity and explanation as critical to science and the progress of science, so I’m not sure what ‘automation of science’ would actually look like. It makes me very sceptical of claims like Davidson’s with larger numbers like ’1000x capability’ (edit, it’s actually 1000x in a year! Is that a one-time increase or perpetual explosive growth? It just seems way too strong a claim).[4]
Progress in AI Safety has empirically been correlated, at least weakly, with increases in capability. I didn’t agree with everything in Nora’s recent post, but I think she’s right about assessing a fully thoeretical approach to progress (such as MIRI’s strategy, as far as I can tell).
In summary, it doesn’t seem like the field of AIXR comes out particularly strongly to me on the I-T-N Framework relative to other existential and catastrophic risks. It may be ‘more important’, but seems similarly intractable in the style that these global problems generally are, and definitely not as neglected any more (in terms of funding, where does it stand relative to nuclear or bio for example? would be interesting to know).
However, I still think x-risk and catastrophic risk from AI this century is unjustifiably high. I just don’t think it’s plausible to have levels of pDoom ~>90% under current evidence unless you have private insight.
I think the main existential/catastrophic issues around AI in the foreseeable future revolve around political institutions and great power conflict, rather than humans being wiped out agentically (either deceptively by a malicious power-seeker or unintentionally by an idiot-savant)
Anyway, these thoughts aren’t fully worked out, I’m still exploring what I think on this issue, but just wanted to register where my current thoughts are at in case it helps others in the community.
Say on a ~50 year time scale, or to the end of the century
Clarification: I’m not saying ‘Sydney’ had catastrophic consequences, but that a future system could be released in a similar way due to internal corporate pressures, and that system could then act negatively in the real world in a way its developers did not expect.
Btw, xuan’s twitter is one of the best I know of to get insights into the state of AI. Sometimes I agree, sometimes I don’t, but her takes are always legit
What does it even mean to say 1,000x capability, especially in terms of science? Is there a number here that Davidson is tracking? When would he view it as falsified?
Thanks for writing this up Richard. There was a plan a few months ago on the forum to divide and conquer the book, and maybe post up a joint review, but it fell by the wayside. That’s partly due to real-life being higher priority, but I also got ground down by the hostile nature of the book—and slogging through it to reproduce the arguments just wasn’t going to happen. I welcome attempts to steelman parts of this book[1], or associated works, as I do think it is important for EA to understand and to engage with our critics.
I think there are some good bits in there that are worth understand where critical/leftist challenges to Effective Altruism come from. Srinivasan’s foreword is a good outline of a ‘low decoupling’ challenge to the analytic philosophy approach that’s core to a lot of EA thinking, and I found Simone de Lima’s Chapter 2 the highlight of what I read—I think that the Chapter title does it an injustice. I’d encourage people to read David Thorstad’s review of that chapter for a charitable/steelman reading of it. In many places, especially the end of Chapter 17, the contributors are clear that they view Capitalism as the primary source of evil—and that this is a self-evident moral fact, and it’s a pretty clear crux to understand their point-of-view.
But yeah, a lot of it is just… really, really bad. I’m going to single out Crary’s Chapter 16 “Against Effective Altruism” here. I believe it’s mostly a reproduction of a previous essay (perhaps with small changes) that are already in the public domain.[2] Some points that are especially egregious to me:
Crary presents the ‘Institutional’ and ‘Philosophical’ critiques, before combining them into her ‘Composite’ critique that is meant to carry the weight of both. But there’s very little evidence cited for it at all—for the Composite Critique the main argument is that it “finds extensive support in philosophical reflection about the social sciences”, such as the Frankfurt school, which… will do very little to persuade many EAs. If the choice is between EA or the Frankfurt school, then that’s a very easy choice.
Much like the rest of the book, there’s an assumption that Capitalism is wrong, and that any deviations from removing it are of very little moral value. Instead of working with animal industries, we should be working for complete abolition. Instead of donating to Give Directly, we should be undoing the Bretton Woods organisations. This is, as you say, just a testimonial of their anti-capitalist perspective, and reads more as a rallying cry for fellow critics of EA to join the fight against it, rather than a consideration of whether EA is right at all.
In particular, Crary keeps calling EA morally corrupt throughout her piece. Perhaps I have too naïve a view of the academy—but this seems beyond the pale. Any academics who are reading this, how often have mentors/reviewers/interlocutors called your work “a textbook case of moral corruption”? What do you think would happen if they did? In her reference to this, she quotes the original source of the idea as saying that this isn’t villifying any individuals and—well, my response is “yeah right”. You should know you’re playing with fire when you use that kind of language, and cowardly hiding a get-out-of-jail free card in your citations about how your terminology doesn’t mean what it usually means doesn’t cut it. Calling what someone believes as ‘moral corruption’ is a serious and harmful attack.
Her quip at the end, that accepting her philosophical critiques and changing the way EA works would allow it “finally a step toward doing a bit of good” is I think both historically wrong and frankly insulting to the good work that EA individuals and organisations have done.
I honestly think that Crary’s article might be one of worst anti-EA articles I’ve read and boy is that saying something, because there are a lot. As I said in the intro maybe someone can make a stronger case for it, it definitely riled me up, but it may honestly just be as bad an article as it seems.
There is one final point worth considering though. There’s a meme in some rationalist-spaces that EAs are “quokkas”—which I don’t like[3], but it may have a kernel of truth if we’re so “turn the other cheek, don’t fight noise with noise” etc. I want to suggest that EA should take a stronger response to this kind of critique in the future. I think the current line is to just not respond to this kind of thing? But I think we’re big enough as a movement where our criticism is also noticeable, and will on its own persuade critics. I think it was completely avoidable for someone as prominent as Timnit Gebru to call us all fascists. Instead, we should be pushing back on prominent criticisms that are clearly wrong and beyond the pale—and do so clearly but forcefully. In short, in cases like this, we may have too many Darwins and not enough Huxleys.
Update:
FLI have released a full statement on their website here, and there is an FAQ post on that statement where discussion has mostly moved to on the Forum. I will respond to these updates there, and otherwise leave this post as-is (for now).
However, it looks like an ‘ignorance-based’ defence is the correct interpretation of what happened here. I don’t regret this post—I still think it was important, and got valuable information out there. I also think that emotional responses should not be seen as ‘wrong’. Nevertheless, I do have some updating to do, and I thank all commenters in the thread below.
I have also made some retractions, with explanations in the footnotes
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Epistemic Status: Unclear, but without much reason to dispute the factual case presented by Expo. As I wrote this comment, an ignorance-based defence seemed less and less convincing, and consequently my anger rose. I apologise if this means the post is of a lower tone than the forum is used to. I will also happily correct or retract this post partially or fully if better evidence is provided.
[Clarity Edit: FLI refers to the Future of Life Institute (FLI) not the Future for Humanity Institute (FHI) which has caused some confusion below. Max Tegmark is President of the former, Nick Bostrom is Director of the latter. The two dramas today are not related, other than longtermist organisations needing better acronyms]
Some other things noted for further detail from the article:
I have no reason to dispute the claim of Nya Dagbladet to be a ‘pro-nazi’ site[1] (clarifying edit/correction in footnote) - users with a bigger knowledge of the Swedish media ecosphere might be able to clarify, but this claim seems to be trustworthy.
Expo claims that a promise from FLI for a grant is in a written application to the County Administration of Dalarna in Sweden with a signature by Tergmark, despite Tegmark initially denying the claim—they now have a picture of Tegmark sending a letter of intent, which unless forged, looks 100% genuine.
A response from Dentons (a US law-firm representing FLI)
seems to confirm the decision to fundNya Dagbladethad been made, but there is now no promise from FLI and not likely to be[2]. They have not responded to Expo since the initial email exchange. It is unclear why FLI decided to make the grant initially, and later change their minds.Tegmark’s brother, Per, seems to be affiliated with the Populist, anti-vax right in Sweden (note,this is only after a very cursory Google search). The reasons this seems to be relevant is that Per has been a contributing writer for Nya Dagbladet in recent years.
My understanding:
In August/September, FLI promised funding in a letter of intent for $100k to Nya Dagbladet so they could set up their foundation.
In October, the promise of funds seems to have been secure enough for the Dalarna County Administrative Board to register the foundation.
In December, Expo (presumably investigating the new Nya Dagbladet Foundation) follow the money back to FLI and question Tegmark
Tegmark initially denies having done so, but Expo have the signed letter. Eventually Dentons, a law firm, respond for FLI and deny that any funds had been released to Nya Dagbladet, and that this decision had been made before Expo got in contact with Tegmark/FLI. They have not responded since.
I cannot speak to any legal questions here, or liability that FLI might face, though I’m not sure why there would be.
There are, however, massive reputational issues at stake. The EA movement is under intense scrutiny right now, and this seems to be another case of
a major actora well-known actor in our movement doing something that has massively poor consequences for the public perception of EA unless they can explain why. Critically, Nya Dagbladet while small seems to beopenly[3] far-right, supporting anti-vaccination sentiment and holocaust denial. I am struggling to charitably interpret how funding them would improve the future of humanity, or do the most good for the world right now.I think it would be prudent for someone from FLI to explain what happened here.
If not, people both inside and outside the EA movement, be they supporters or critics, may
correctly[4] be led to infer thata major organisation ina well-known organisation aligned with EA promised funds to an openly politically far-right organisation, knowing what they stand for. That is not what EA should stand for, and to the extent that it does, I would want no part in it.Many seem to be taking the ‘pro-nazi’ as a crux. That was the characterisation Expo gave, and I went with their framing as default. Depending on your definition of ‘pro-nazi’ this might be false, Nya Dagbladet don’t seem to openly support the persecution of Jews or a white ethnostate openly that I could see—but it’d be very difficult for any publication to do so openly. In the Expo article, there’s a sidebar with two of the most damning pieces of content.
I would at least characterise them as far-right/populist reactionary/ethno-nationalist, which even if not as morally horrifying as openly ‘pro-nazi’, is something which I believe to be strongly antithetical to what EA is and should stand for. But I think I will elaborate on my thoughts on EA/politics in a future post, rather than here. In any case, I think the issue is why this grant was considered in the first place given the political affiliation of the recipients, rather than whether those political affiliations are less far-right than the Expo article implies.
[Edit: I think that this claim is false. No grant was ever confirmed, and the FAQ states that the ‘letter of intent’ was a specific request by Nya Dagbladet and not part of FLI’s usual grant-making process]
[Edit: I retract the use of ‘openly’ here, they seem to openly be populist right, but don’t make their far-rights leanings immediately obvious]
[Edit: I retract the use of ‘correctly’ here—I meant it to refer to a counterfactual case where the worst possible case was true, but I think it is probably more confusing than useful]