Many people find the Forum anxiety inducing because of the high amount of criticism. So, in the spirit of Giving Season, I’m going to give some positive feedback and shout-outs for the Forum in 2023 (from my PoV). So, without further ado, I present the 🎄✨🏆 totally-not-serious-worth-no-internet-points-JWS-2023-Forum-Awards: 🏆✨🎄[1]
Best Forum Post I read this year:
10 years of Earning to Give by @AGB: A clear, grounded, and moving look at what it actually means to ‘Earn to Give’. In particular, the ‘Why engage?’ section really resonated with me.
Third Wave Effective Altruism by @Ben_West: A short post, but one that came at exactly the right time. I think this is a really important topic for EA to be thinking about right now and look forward to more posts inspired by it next year.
@titotal - I really value titotal’s contributions to the Forum. Even though I sometimes disagree I always learn a lot by thinking about where and why I do so, and I hope they stick around so I can continue to learn from them.
@Holly_Elmore - Similarly to Titotal, while I don’t always agree with Holly she just makes really solid points that always positively add to the discussion. Often she’s able to clearly find the crux of a debate and ask the key questions.
@Ivy Mazzola - Whenever I’ve seen Ivy post on the Forum, even on heated topics, she always approaches them with integrity and grace that I really wanted to give recognition to.
@David Mathers - When David discusses ideas in the Forum he consistently does so in a good faith and truth-seeking way that really shows engagement on the Forum at its best.
Non-Forum EA Poasters of the Year:
David Thorstad (on his blog) - I’ve really appreciated David’s blog this year. I don’t always agree, and you can find me in the comments pushing back sometimes, I’ve found his content high quality overall, especially on Existential Risk and the Time of Perils.
Agustín (on Twitter) - Agus has been really fighting the good fighting in the Twitter trenches recently, on all sorts of topics (though sadly on the terrible state of AI discourse). Somehow he manages to translate complicated EA ideas clearly, convincingly, and remain calm in the face of less-than-charitable interlocutors.
Nathan Young (on Twitter) - Nathan’s commitment to honest exploration of his beliefs, dialogue with those who disagree, and finding mechanisms of belief resolution has been constantly inspiring.
Congrats to the winners! Feel free to give them more plaudits in the comments or just shout out your own Forum favourites from this year! I wish everyone who’s reading this every success in the year to come and I look forward to reading more good content on the Forum in 2024!
Thanks for saying nice things about me! For the record, I also think David Thorstad’s contributions are very valuable (whether or not his views are ultimately correct).
The HLI discussion on the Forum recently felt off to me, bad vibes all around. It seems very heated, not a lot of scout mindset, and reading the various back-and-forth chains I felt like I was ‘getting Eulered’ as Scott once described.
I’m not an expert on evaluating charities, but I followed a lot of links to previous discussions and found this discussion involving one of the people running an RCT on Strongminds (which a lot of people are waiting for the final results of) who was highly sceptical of SM efficacy. But the person offering counterarguments in the thread seems to be just as valid to me? My current position, for what it’s worth,[1] is:
the initial Strongminds results of 10x cash transfer should raise a sceptical response. most things aren’t that effective
it’s worth there being exploration of what the SWB approach would recommend as the top charities (think of this as trying other bandits in a multi-armed bandit charity evaluation problem)
it’s very difficult to do good social science, and the RCT won’t give us dispositive evidence about the effectiveness of Strongminds (especially at scale), but it may help us update. In general we should be mindful of how far we can make rigorous empirical claims in the social sciences
HLI has used language too loosely in the past and overclaimed/been overconfident, which Michael has apologised for, though perhaps some critics would like a stronger signal of neutrality (this links to the ‘epistemic probation’ comments)
GiveWell’s own ‘best guess’ analysis seems to be that Strongminds is 2.3x that of GiveDirectly.[2] I’m generally a big fan of the GiveDirectly approach for reasons of autonomy—even if Strongminds got reduced in efficacy to around ~1x GD, it’d still be a good intervention? I’m much more concerned with what this number is than the tone of HLIs or Michael’s claims tbh (though not at the expense of epistemic rigour).
The world is rife with actively wasted or even negative action, spending, and charity. The integrity of EA research, and holding charity evaluators to account is important to both the EA mission and EAs identity. But HLI seems to have been singled out for very harsh criticism,[3] but so much of the world is worse.
I’m also quite unsettled by a lot of what I call ‘drive-by downvoting’. While writing a comment is a lot more effort than clicking to vote on a comment/post, I think the signal is a lot higher, and would help those involved in debates reach consensus better. Some people with high-karma accounts seem to be making some very strong votes on that thread, and very few are making their reasoning clear (though I salute those who are in either direction).
So I’m very unsure how to feel. It’s an important issue, but I’m not sure the Forum has shown itself in a good light in this instance.
Some people with high-karma accounts seem to be making some very strong votes on that thread, and very few are making their reasoning clear (though I salute those who are in either direction).
I think this is a significant datum in favor of being able to see the strong up/up/down/strong down spread for each post/comment. If it appeared that much of the karma activity was the result of a handful of people strongvoting each comment in a directional activity, that would influence how I read the karma count as evidence in trying to discern the community’s viewpoint. More importantly, it would probably inform HLI’s takeaways—in its shoes, I would treat evidence of a broad consensus of support for certain negative statements much, much more seriously than evidence of carpet-bomb voting by a small group on those statements.
Indeed our new reacts system separates them. But our new reacts system also doesn’t have strong votes. A problem with displaying the number of types of votes when strong votes are involved is that it much more easily allows for deanonymization if there are only a few people in the thread.
That makes sense. On the karma side, I think some of my discomfort comes from the underlying operationalization of post/comment karma as merely additive of individual karma weights.
True opinion of the value of the bulk of posts/comments probably lies on a bell curve, so I would expect most posts/comments to have significantly more upvotes than strong upvotes if voters are “honestly” conveying preferences and those preferences are fairly representative of the user base. Where the karma is coming predominately from strongvotes, the odds that the displayed total reflects the opinion of a smallish minority that feels passionately is much higher. That can be problematic if it gives the impression of community consensus where no such consensus exists.
If it were up to me, I would probably favor a rule along the lines of: a post/comment can’t get more than X% of its net positive karma from strongvotes, to ensure that a high karma count reflects some degree of breadth of community support rather than voting by a small handful of people with powerful strongvotes. Downvotes are a bit trickier, because the strong downvote hammer is an effective way of quickly pushing down norm-breaking and otherwise problematic content, and I think putting posts into deep negative territory is generally used for that purpose.
EA is just a few months out from a massive scandal caused in part by socially enforced artificial consensus (FTX), but judging by this post nothing has been learned and the “shut up and just be nice to everyone else on the team” culture is back again, even when truth gets sacrificed on the process. No thinks HLI is stealing billions of dollars of course, but the charge that they keep quasi-deliberately stacking the deck in StrongMinds’ favour is far from outrageous and should be discussed honestly and straightforwardly.
JWS’ quick take has often been in negative agreevote territory and is +3 at this writing. Meanwhile, the comments of the lead HLI critic suggesting potential bad faith have seen consistent patterns of high upvote / agreevote. I don’t see much evidence of “shut up and just be nice to everyone else on the team” culture here.
I don’t think the Forum’s reaction to the HLI post has been “shut up and just be nice to everyone else on the team”, as Jason’s response suggested.
I don’t think mine suggests that either! In fact, my first bullet point has a similar sceptical prior to what you express in this comment[1] I also literally say “holding charity evaluators to account is important to both the EA mission and EAs identity”, and point that I don’t want to sacrifice epistemic rigour. In fact, one of my main points is that people—even those disagreeing with HLI, are shutting up too much! I think disagreement without explanation is bad, and I salute the thorough critics on that post who have made their reasoning for putting HLI in ‘epistemic probation’ clear.
I don’t suggest ‘sacrificing the truth’. My position is that the truth on StrongMind’s efficacy is hard to get a strong signal on, and therefore HLI should have been more modest early on their history, instead of framing it as the most effective way to donate.
As for the question of whether HLI were “quasi-deliberately stacking the deck”, well I was quite open that I think I am confused on where the truth is, and find it difficult to adjudicate what the correct takeway should be.
I don’t think we really disagree that much, and I definitely agree that the HLI discussion should proceed transparently and EA has a lot to learn from the last year, including FTX. I think if you maybe re-read my Quick Take, I’m not taking the position you think I am.
I want to register that my perspective on medium-term[1] AI existential risk (shortened to AIXR from now on) has changed quite a lot this year. Currently, I’d describe it as moving from ‘Deep Uncertainty’ to ‘risk is low in absolute terms, but high enough to be concerned about’. I guess atm I’d think that my estimates are moving closer toward the Superforecasters in the recent XPT report (though I’d still say I’m still Deeply Uncertain on this issue, to the extent that I don’t think the probability calculus is that meaningful to apply)
Some points around this change:
I’m not sure it’s meaningful to cleanly distinguish AIXR from other anthropogenic x-risks, especially since negative consequences of AI may plausibly increase other x-risks (e.g. from Nuclear War, biosecurity, Climate Change etc.)
I think in practice, the most likely risks from AI would come from deployment of powerful systems that have catastrophic consequences that are then rolled back. I’m think of Bing ‘Sydney’ here as the canonical empirical case.[2] I just don’t believe we’re going to get no warning shots.
Similary, most negative projections of AI don’t take into account negative social reaction and systematic human response to these events. You either assume we’ll get no warning shots and then get exterminated (‘sharp left turn’) or think that humanity is doomed to not co-operate (‘Moloch’). I think that the evidence suggests instead that societies and governments will react against increasing AI capability if they view it negatively, rather than simply stand by and watch it happen, which is what many AIXR seem to assume or at least imply to me.
I think the AIXR community underestimated the importance of AI Governance and engaging with politics. Instead of politicians/the public ‘not getting it’, they infact seem to have ‘got it’. The strategy of only letting people vetted to be smart enough think about AIXR seems to have been flawed, the same kind of thinking that led to thinking a ‘pivotal act’ strategy was viable.
In fact, I’ve been pleasantly surprised by how welcoming both politicians and the public have been to the Overton Window being opened. Turns out the median voter in a liberal democracy doesn’t like ‘let large co-orporations create powerful models with little understanding of their consequences without significant oversight’. I think people’s expectations of positive co-ordination on AI issues should have gone up this year, and conversely it should make your AIXR estimate go down unless you think relative alignment progress has declined even more.
While there have been an awful lot of terrible arguments against AIXR raised this year, some have been new and valuable to me. Some examples:
As titotal has written, I expect ‘early AGI’ will be a ‘buggy mess’. More specifically, I doubt it could one-shot solve ‘take over the world’ unless you assume god-like capability by assumption, instead of ‘more capable than humans in some important ways, but not invincible’.
Progress via ‘stack moar layers lol’ will plausibly slow down, or at least run into some severe issues.[3] The current hype-wave seems to be just following this as far as it will go with compute and data, rather than exploring alternative architectures that could be much more efficient.
I’m still unimpressed by how Frontier systems perform on tests such as The ARC Challenge which test hypothesis creativity and testing in a few-shot scenario (and on hidden data) without ‘ground truth’, as opposed to training on trillions-upon-trillions of training examples with masks and true labels.
Related to the above, I view creativity and explanation as critical to science and the progress of science, so I’m not sure what ‘automation of science’ would actually look like. It makes me very sceptical of claims like Davidson’s with larger numbers like ’1000x capability’ (edit, it’s actually 1000x in a year! Is that a one-time increase or perpetual explosive growth? It just seems way too strong a claim).[4]
Progress in AI Safety has empirically been correlated, at least weakly, with increases in capability. I didn’t agree with everything in Nora’s recent post, but I think she’s right about assessing a fully thoeretical approach to progress (such as MIRI’s strategy, as far as I can tell).
In summary, it doesn’t seem like the field of AIXR comes out particularly strongly to me on the I-T-N Framework relative to other existential and catastrophic risks. It may be ‘more important’, but seems similarly intractable in the style that these global problems generally are, and definitely not as neglected any more (in terms of funding, where does it stand relative to nuclear or bio for example? would be interesting to know).
However, I still think x-risk and catastrophic risk from AI this century is unjustifiably high. I just don’t think it’s plausible to have levels of pDoom ~>90% under current evidence unless you have private insight.
I think the main existential/catastrophic issues around AI in the foreseeable future revolve around political institutions and great power conflict, rather than humans being wiped out agentically (either deceptively by a malicious power-seeker or unintentionally by an idiot-savant)
Anyway, these thoughts aren’t fully worked out, I’m still exploring what I think on this issue, but just wanted to register where my current thoughts are at in case it helps others in the community.
Clarification: I’m not saying ‘Sydney’ had catastrophic consequences, but that a future system could be released in a similar way due to internal corporate pressures, and that system could then act negatively in the real world in a way its developers did not expect.
Btw, xuan’s twitter is one of the best I know of to get insights into the state of AI. Sometimes I agree, sometimes I don’t, but her takes are always legit
What does it even mean to say 1,000x capability, especially in terms of science? Is there a number here that Davidson is tracking? When would he view it as falsified?
Thanks :) I might do an actual post at the end of the year? In the meantime I just wanted to get my ideas out there as I find it incredibly difficult to actually finish any of the many Forum drafts I have 😭
Honestly I just wrote a list of potential x-risks to make a similar reference class. It wasn’t mean to be a specific claim, just examples for the quick take!
I guess climate change might be less of an existential risk in an of itself (per Halstead), but there might be interplays between them that increase their combined risk (I think Ord talks about this in the precipice). I’m also sympathetic to Luke Kemp’s view that we should really just care about overall x-risk, regardless of cause area, as extinction by any means would be as bad for humanities potential.[1]
I think it’s plausible to consider x-risk from AI higher than Climate Change over the rest of this century, but my position at the moment is that this would be more like 5% v 1% or 1% v 0.01% than 90% v 0.001%, but as I said I’m not sure trying to put precise probability estimates is that useful.
Definitely accept the general point that it’d be good to be more specific with this language in a front-page post though.
My point is that even though AI emits some amount of carbon gases, I’m struggling to find a scenario where it’s a major issue for global warming as AI can help provide solutions here as well.
(Oh, my point wasn’t that climate change couldn’t be an x-risk, though it has been disputed, more that I don’t see the pathway for AI to exacerbate climate change).
It makes me very sceptical of claims like Davidson’s with larger numbers like ’1000x capability’ (edit, it’s actually 1000x in a year! Is that a one-time increase or perpetual explosive growth? It just seems way too strong a claim).
I was wondering why he said that, since I’ve read his report before and that didn’t come up at all. I suppose a few scattered recollections I have are
Tom would probably suggest you play around with the takeoffspeeds playground to gain a better intuition (I couldn’t find anything 1,000x-in-a-year-related at all though)
Capabilities takeoff speed ≠ impact takeoff speed (Tom: “overall I expect impact takeoff speed to be slower than capabilities takeoff, with the important exception that AI’s impact might mostly happen pretty suddenly after we have superhuman AI”)
This is an off-the-cuff quick take that captures my current mood. It may not have a long half-life, and I hope I am wrong
Right now I am scared
Reading the tea-leaves, Altman and Brockman may be back at OpenAI, the company charter changed, and the board—including Toner and MacAulay—removed from the company
The mood in the Valley, and in general intellectual circles, seems to have snapped against EA[1]
This could be as bad for EA’s reputation as FTX
At a time when important political decisions about the future of AI are being made, and potential coalitions are being formed
And this time it’d be second-impact syndrome
I am scared EA in its current form may not handle the backlash that may come
I am scared that we have not done enough reform in the last year from the first disaster to prepare ourselves
I am scared because I think EA is a force for making the world better. It has allowed me to do a small bit to improve the world. Through it, I’ve met amazing and inspiring people who work tirelessly and honestly to actually make the world a better place. Through them, I’ve heard of countless more actually doing what they think is right and giving what they can to make the world we find ourselves is more full of kindness, happiness, and flourishing
I am scared because this could be lost, I don’t know what to do to prevent it
As with Nonlinear and FTX, I think that for the vast majority of people, there’s little upside to following this in real-time.
It’s very distracting, we have very little information, things are changing fast, and it’s not very action-relevant for most of us.
I’m also very optimistic that the people “who work tirelessly and honestly to actually make the world a better place” will keep working on it after this, whatever happens to “EA”, and there will still be ways to meet them and collaborate.
It’s hard to see how the backlash could actually destroy GiveWell or stop Moskowitz and Tuna giving away their money through Open Phil/something that resembles Open Phil. That’s a lot of EA right there.
It’s hard yes, but I think the risk vectors are (note—these are different scenarios, not things that follow in chronological order, though they could):
Open Philanthropy gets under increasing scrutiny due to its political influence
OP gets viewed as a fully politicised propaganda operation from EA, and people stop associating with it, accepting its money, or call for legal or political investigations into it etc
Givewell etc dissassociate themselves from EA due to EA having a strong negative social reaction from potential collaborators or donors
OP/Givewell dissociate from and stop funding the EA community for similar reasons as the above, and the EA community does not survive
Basically I think that ideas are more important than funding. And if society/those in positions of power put the ideas of EA in the bin, money isn’t going to fix that
This is all speculative, but I can’t help the feeling that regardless of how the OpenAI crisis resolves a lot of people now consider EA to be their enemy :(
My general thoughts on this can be stated as: I’m mostly of the opinion that EA will survive this, bar something massively wrong like the board members willfully lying or massive fraud from EAs, primarily because most of the criticism is directed to the AI safety wing, and EA is more than AI safety, after all.
Nevertheless, I do think that this could be true for the AI safety wing, and they may have just hit a key limit to their power. In particular, depending on how this goes, I could foresee a reduction in AI safety power and influence, and IMO this was completely avoidable.
I think a lot will depend on the board justification. If Ilya can say “we’re pushing capabilities down a path that is imminently highly dangerous, potentially existentially, and Sam couldn’t be trusted to manage this safely” with proof that might work—but then why not say that?[1]
If it’s just “we decided to go in a different direction”, then firing him and demoting Brockman with little to no notice, and without informing their largest business partner and funder, its bizarre that they took such a drastic step in the way they did
I was actually writing up my AI-risk sceptical thoughts and what EA might want to take from that, but I think I might leave that to one side for now until I can approach it with a more even mindset
Putting aside that I feel both you and I are sceptical that a new capability jump has emerged, or that scaling LLMs is actually a route to existential doom
If Ilya can say “we’re pushing capabilities down a path that is imminently highly dangerous, potentially existentially, and Sam couldn’t be trusted to manage this safely” with proof that might work—but then why not say that?
I suspect this is due to the fact that quite frankly, the concerns they had about Sam Altman being unsafe on AI basically had no evidence except speculation from the EA/LW forum, which is not enough evidence at all in the corporate world/legal world, and to be quite frank, the EA/LW standard of evidence on AI risk being a big deal enough to investigate is very low, sometimes non-existent, and that simply does not work once you have to deal with companies/the legal system.
More generally, EA/LW is shockingly loose, sometimes non-existent in its standards of evidence for AI risk, which doesn’t play well with the corporate/legal system.
This is admittedly a less charitable take than say, Lukas Gloor’s take.
This is admittedly a less charitable take than say, Lukas Gloor’s take.
Haha, I was just going to say that I’d be very surprised if the people on the OpenAI board didn’t have access to a lot more info than the people on the EA forum or Lesswrong, who are speculating about the culture and leadership at AI labs from the sidelines.
TBH, if you put a randomly selected EA from a movement of 1,000s of people in charge of the OpenAI board, I would be very concerned that a non-trivial fraction of them probably would make decisions the way you describe. That’s something that EA opinion leaders could maybe think about and address.
But I don’t think most people who hold influential positions within EA (or EA-minded people who hold influential positions in the world at large, for that matter) are likely to be that superficial in their analysis of things. (In particular, I’m strongly disagreeing with the idea that it’s likely that the board “basically had no evidence except speculation from the EA/LW forum”. I think one thing EA is unusually good at – or maybe I should say “some/many parts of EA are unusually good at” – is hiring people for important roles who think for themselves and have generally good takes about things and acknowledge the possibility of being wrong about stuff. [Not to say that there isn’t any groupthink among EAs. Also, “unusually good” isn’t necessarily that high of a bar.])
I don’t know for sure what they did or didn’t consider, so this is just me going off of my general sense for people similar to Helen or Tasha. (I don’t know much about Tasha. I’ve briefly met Helen but either didn’t speak to her or only did small talk. I read some texts by her and probably listened to a talk or two.)
While I generally agree that they almost certainly have more information on what happened, which is why I’m not really certain on this theory, my main reason here is that for the most part, AI safety as a cause basically managed to get away with incredibly weak standards of evidence for a long time, until the deep learning era in 2019-, especially with all the evolution analogies, and even now it still tends to have very low standards (though I do believe it’s slowly improving right now). This probably influenced a lot of EA safetyists like Ilya, who almost certainly imbibed the norms of the AI safety field, and one of them is that there is a very low standard of evidence needed to claim big things, and that’s going to conflict with corporate/legal standards of evidence.
But I don’t think most people who hold influential positions within EA (or EA-minded people who hold influential positions in the world at large, for that matter) are likely to be that superficial in their analysis of things. (In particular, I’m strongly disagreeing with the idea that it’s likely that the board “basically had no evidence except speculation from the EA/LW forum”. I think one thing EA is unusually good at – or maybe I should say “some/many parts of EA are unusually good at” – is hiring people for important roles who think for themselves and have generally good takes about things and acknowledge the possibility of being wrong about stuff. [Not to say that there isn’t any groupthink among EAs. Also, “unusually good” isn’t necessarily that high of a bar.])
I agree with this weakly, in the sense that being high up in EA is at least a slight update towards them actually thinking through things and being able to make actual cases. My disagreement here is that this effect is probably not strong enough to wash away the cultural effects of operating in a cause area where they don’t need to meet any standard of evidence except long-winded blog posts and getting rewarded, for many reasons.
Also, the board second-guessed it’s decision, which would be evidence for the theory that they couldn’t make a case that actually abided to the standard of evidence for a corporate/legal setting.
If it was any other cause like say GiveWell or some other causes in EA, I would trust them much more that they do have good reason. But AI safety has been so reliant on very low-non-existent standards of evidence or epistemics that they probably couldn’t explain themselves in a way that would abide by the strictness of a corporate/legal standard of evidence.
Edit: The firing wasn’t because of safety related concerns.
I am unsure how I feel about takes like this. On one hand, I want EAs and the EA community to be a supportive bunch. So, expressing how you are feeling and receiving productive/helpful/etc. comments is great. The SBF fiasco was mentally strenuous for many, so it is understandable why anything seemingly negative for EA elicits some of the same emotions, especially if you deeply care about this band of people genuinely aiming to do the most good they can.
On the other hand, I think such takes could also contribute to something I would call a “negative memetic spiral.” In this particular case, several speculative projections are expressed together, and despite the qualifying statement at the beginning, I can’t help but feel that several or all of these things will manifest IRL. And when you kind of start believing in such forecasts, you might start saying similar things or expressing similar sentiments. In the worst case, the negative sentiment chain grows rapidly.
It is possible that nothing consequential happens. People’s mood during moments of panic are highly volatile, so five years in, maybe no one even cares about this episode. But in the present, it becomes a thing against the movement/community. (I think a particular individual may have picked up one such comment from the Forum and posted it online to appease to their audience and elevate negative sentiments around EA?).
Taking a step back, gathering more information, and thinking independently, I was able to reason myself out of many of your projections. We are two days in and there is still an acute lack of clarity about what happened. Emmett Shear, the interim CEO of OpenAI, stated that the board’s decision wasn’t over some safety vs. product disagreement. Several safety-aligned people at OpenAI signed the letter demanding that the board should resign, and they seem to be equally disappointed over recent events; this is more evidence that the safety vs. product disagreement likely didn’t lead to Altman’s ousting. There is also somewhat of a shift back to the “center,” at least on Twitter, as there are quite a few reasonable, level-headed takes on what happened and also on EA. I don’t know about the mood in the Bay though, since I don’t live there.
I am unsure if I am expressing my point well, but this is my off-the-cuff take on your off-the-cuff take.
Thanks for your response Akash. I appreciate your thoughts, and I don’t mind that they’re off-the-cuff :)
I agree with some of what you say, and part of what I think is your underlying point , but in some others I’m a bit less clear. I’ve tried to think about two points where I’m not clear, but please do point if I’ve got something egregiously wrong!
1) You seem to be saying that sharing negative thoughts and projections can lead others to do so, and this can then impact other people’s actions in a negative way. It could also be used by anti-EA people against us.[1]
I guess I can kind of see some of this, but I guess I’d view the cure as being worse than the disease sometimes. I think sharing how we’re thinking and feeling is overall a good thing that could help us understand each other more, and I don’t think self-censorship is the right call here. Writing this out I think maybe I disagree with you about whether negative memetic spirals are actually a thing causally instead of descriptively. I think people may be just as likely apriori to have ‘positive memetic spirals’ or ‘regressions to the vibe mean’ or whatever
2) I’m not sure what ‘I was able to reason myself out of many of your projections’ means, apart from ‘hold fire on making projections’. But currently:
We’ve still heard absolutely nothing from the board
Emmett has potentially indicated he will resign if the board doesn’t explain their reasoning or show evidence to back up their decision[2]
This leads me to doubt his earlier claim that the board told him what the firing was about
There still seems to be a concerted effort by some parties to bring Altman back
I don’t see a shift to the centre on Twitter about what happened or about EA. I think the initial flames have cooled a bit, but my intuition is that a shift has happened, even if the reasons for that shift haven’t been fully borne out by the facts
I still think that it’s a very precarious situation that could have some bad downside consequences for EA. Maybe I could have highlighted my uncertainty more though, I wasn’t trying to say exactly what was going to happen.
‘Hold fire on making projections’ is the correct read, and I agree with everything else you mention in point 2.
About point 1 — I think sharing negative thoughts is absolutely a-ok and important. I take issue with airing bold projections when basic facts of the matter aren’t even clear. I thought you were stating something akin to “xyz are going to happen,” but re-reading your initial post, I believe I misjudged.
Congrats to the CEA Events Team for their hard work and for organising such a good event! 👏
The vibe was really positive! Anecdotally I had heard that the last EAG SF was gloom central, but this event felt much more cheery. I’m not entirely sure why, but it might have had something to do with the open venue, the good weather, or there being more places to touch grass in London compared to the Bay.
I left the conference intellectually energised (though physically exhausted). I’m ready to start drafting some more Forum Post ideas that I will vastly overestimate my ability to finish and publish 😌
AI was (unsurprisingly) the talk of the town. But I found that quite a few people,[2] myself included, were actually more optimistic on AI because of the speed of the social response to AI progress and how pro-safety it seems to be, along with low polarisation along partisan lines.
Related to the above, I came away with the impression that AI Governance may be as if not more important than Technical Alignment in the next 6-12 months. The window for signficiant political opportunity is open now but may not stay open forever, so the AI Governance Space is probably where the most impactful opportunties might be at the moment.
My main negative takeaway was that there seemed to be so little reflection on the difficult last ~6 months for the EA movement. There was 1 session on the FTX, but none at all on the other problems we’ve faced as a community such as Sexual Abuse and Harassment, Trust in EA Leadership, Community Epistemic Health, and whether EA Institutions ought to be reformed. In the opening talk, the only reference was that the ship of EA feels like it’s been ‘going through a storm’, and the ideas presented weren’t really accompanied by a route to embed them in the movement. To me it felt like another missed opportunity after Toby Ord’s speech, and I don’t feel like we as a community have fully grasped or reckoned with the consequences of the last 6 months. I think this was also a common sentiment among people I talked to[3] regardless of whether they agreed with me on proposed solutions.
Shrimp Welfare is now post-meme and part of the orthodoxy, and the Shrimp Welfare Project is unironically an S(hrimp)-Tier Charity. 🦐🥰
EA Conference Food continues to be good to me. I don’t know why it seems to get consistently low ratings in the feedback surveys 🤷 I’d happily tile the universe with those red velvet desserts.
[edit: a day after posting, I think this perhaps reads more combative that I intended? It was meant to be more ‘crisis of faith, looking for reassurance if it exists’ than ‘dunk on those crazy longtermists’. I’ll leave the quick take as-is, but maybe clarification of my intentions might be useful to others]
Warning! Hot Take! 🔥🔥🔥 (Also v rambly and not rigorous)
A creeping thought has entered my head recently that I haven’t been able to get rid of…
Is most/all longtermist spending unjustified?
The EA move toward AI Safety and Longtermism is often based on EV calculations that show the long term future is overwhelmingly valuable, and thus is the intervention that is most cost-effective.
However, more in-depth looks at the EV of x-risk prevention (1, 2) cast significant doubt on those EV calculations, which might make longtermist interventions much less cost-effective than the most effective “neartermist” ones.
But my doubts get worse...
GiveWell estimates around $5k to save a life. So I went looking for some longtermist calculations, and I really couldn’t fund any robust ones![1] Can anyone point me in some robust calculations for longtermist funds/organisations where they go ‘yep, under our assumptions and data, our interventions are at least competitive with top Global Health charities’?
Because it seems to me like that hasn’t been done. But if we’re being EA, people with high and intractable p(doom) from AI shouldn’t work on AI, they should probably EtG for Global Health instead (if we’re going to be full maximise EV about everything). Like, if we’re taking EA seriously, shouldn’t MIRI shut down all AI operations and become a Global Health org? Wouldn’t that be a strong +EV move given their pessimistic assessments of reducing xRisk and their knowledge of +EV global health interventions?
But it gets worse...
Suppose that we go, ‘ok, let’s take EV estimates seriously but not literally’. In which case fine, but that undermines the whole ‘longtermist interventions overwhelmingly dominate EV’ move that many people feel has happened to EA causes over recent years. I don’t think that move grounds longtermism, instead I think it grounds a more pluralistic approach to all of EA that values a large variety of approaches to doing good, but acknowledges the uncertainty and limits on each, and even actively prevents the movement from being focused on any one area. That still seems incompatible with the monetary and social rise of longtermism and AI Safety.
tl;dr: Have any longtermist orgs done robust and defensible EV caculations for their interventions? If not, isn’t that… overwhelmingly sus?
I found some posts arguing that under set of assumptions X then longtermist interventions would be more effective, often by reducing the overall xrisk rate by 1% or something like that. But very little if no work had been done to actually show that was a likely result of the interventions
which might make longtermist interventions much less cost-effective than the most effective “neartermist” ones.
Why do you think this?
For some very rough maths (appologies in advance for any errors), even Thorstad’s paper (with a 2 century long time of perils, a 0.1% post-peril risk rate, no economic/population growth, no moral progress, people live for 100 years) suggests that reducing p(doom) by 2% is worth as much as saving 16x8billion lives—i.e. each microdoom is worth 6.4million lives. I think we can buy microdooms more cheaply than $5,000*6.4million = $32billion each.
(I can’t actually find those calculations in Thorstad’s paper, could you point them out to me? afaik he mostly looks at the value of fractional reduction in x-risk, while microdooms are an absolute reduction if I understand correctly? happy to be corrected or shown in the right direction!)
My concerns here are twofold:
1 - epistemological: Let’s say those numbers are correct from the Thorstad paper, that a microdoom has to cost ⇐ $32bn to be GiveWell cost-effective. The question is, how would we know this. In his recent post Paul Cristiano thinks that RSPs could lead to a ’10x reduction’ in AI risk. How does he know this? Is this just a risk reduction this century? This decade? Is it a permanent reduction?
It’s one thing to argue that under set of conditions X work on x-risk reduction is cost-effective as you’ve done here. But I’m more interested in the question of whether conditions X hold, because that’s where the rubber hits the road. If those conditions don’t hold, then that’s why longtermism might not ground x-risk work.[1]
There’s also the question of persistence. I think the Thorstad model either assumes the persistence of x-risk reduction, or the persistence of a low-risk post-peril rate. But how could we possibly know that such considerations hold? It seems to me that cluelessness just obliterates grounding strong longtermist interventions here.
2 - practical: Let’s say that longtermist organisations/funds have actually answered criticisms in 1, and under their own lights have made these calculations. I want to see those calculations! How have they calculated the post-risk peril rate? Where is the support for their views on population growth? Do they assume that we colonise the galaxy by default or not? How do they get feedback that overcomes cluelessness objections?
And critically, I want to see why they’ve argued or assumed these things. afaict I can’t find these calculations.
I’m not being snarky here, I really want to be wrong on this and be shown robust calculations on longtermist work, but I haven’t found any. I think the risk is that longtermist EA looks like any other charity/movement that funds ‘what they kinda they think is good’ rather than what we actually know does good, which was meant to be the whole point of EA in the first place.
It could still be grounded for other reasons though, for example if x-risk reduction is highly correlated with GCR reduction, and the latter is of high value
(I can’t actually find those calculations in Thorstad’s paper, could you point them out to me? afaik he mostly looks at the value of fractional reduction in x-risk, while microdooms are an absolute reduction if I understand correctly? happy to be corrected or shown in the right direction!)
He assumes 20% risk and a 10% relative risk reduction, which I translate into 2% absolute risk of doom, and then see the table on p12.
Isn’t the move here something like, “If doom soon, then all pre-doom value nets to zero”?
Which tbh I’m not sure is wrong. If I expect doom tomorrow, all efforts today should be to reduce it; one night’s sleep not being bitten by mosquitoes doesn’t matter. Stretching this outward in time, doesn’t change the calculus much for a while, maybe about a lifetime or a few lifetimes or so. And a huge chunk of xrisk is concentrated in this century.
The x-risk models actually support the opposite conclusion though. They are generally focused on the balance of two values, v and r—where v the value of a time period and r is the risk of extinction in that period.
If r is sufficiently high, then it operates as a de facto discount rate on the future, which means that the most effective way to increase good is to increase the present v rather than reduce r. For an analogy, if a patient has an incredibly high risk of succumbing to terminal cancer, the way to increase wellbeing may be to give them morphine and palliative care rather than perscribe them risky treatments that may or may not work (and might only be temporary)
Now one could argue against this by saying ‘do not get go gentle into that good night, in the face of destruction we should still do our best’. I have sympathy with that view, but it’s not grounded in the general ‘follow the EV’ framework of EA, and it would have consequences beyond supporting longtermism
I think (at least) somebody at Open Philanthropy needs to start thinking about reacting to an increasing move towards portraying it, either sincerely or strategically, as a shadowy cabal-like entity influencing the world in an ‘evil/sinister’ way, similar to how many right-wingers across the world believe that George Soros is contributing to the decline of Western Civilization through his political philanthropy.
Last time there was an explicitly hostile media campaign against EA the reaction was not to do anything, and the result is that Émile P. Torres has a large media presence,[1] launched the term TESCREAL to some success, and EA-critical thoughts became a lot more public and harsh in certain left-ish academic circles. In many think pieces responding to WWOTF or FTX or SBF, they get extensively cited as a primary EA-critic, for example.
I think the ‘ignore it’ strategy was a mistake and I’m afraid the same mistake might happen again, with potentially worse consequences.
Last time there was an explicitly hostile media campaign against EA the reaction was not to do anything, and the result is that Émile P. Torres has a large media presence,[1] launched the term TESCREAL to some success, and EA-critical thoughts became a lot more public and harsh in certain left-ish academic circles.
You say this as if there were ways to respond which would have prevented this. I’m not sure these exist, and in general I think “ignore it” is a really really solid heuristic in an era where conflict drives clicks.
I think responding in a way that is calm, boring, and factual will help. It’s not going to get Émile to publicly recant anything. The goal is just for people who find Émile’s stuff to see that there’s another side to the story. They aren’t going to publicly say “yo Émile I think there might be another side to the story”. But fewer of them will signal boost their writings on the theory that “EAs have nothing to say in their own defense, therefore they are guilty”. Also, I think people often interpret silence as a contemptuous response, and that can be enraging in itself.
Maybe it would be useful to discuss concrete examples of engagement and think about what’s been helpful/harmful.
Offhand, I would guess that the holiday fundraiser that Émile and Nathan Young ran (for GiveDirectly I think it was?) was positive.
I think this post was probably positive (I read it around a year ago, my recollections are a bit vague). But I guess that post itself could be an argument that even attempting to engage with Émile in good faith is potentially dangerous.
Perhaps the right strategy is something like: assume good faith, except with specific critics who have a known history of bad faith. And consider that your comparative advantage may lie elsewhere, unless others would describe you as unusually good at being charitable.
Offhand, I would guess that the holiday fundraiser that Émile and Nathan Young ran (for GiveDirectly I think it was?) was positive.
What makes you think this? I would guess it was pretty negative, by legitimizing Torres, and most of the donations funging heavily against other EA causes.
I would guess any legitimization of Émile by Nathan was symmetrical with a legitimization of Nathan by Émile. However I didn’t get the sense that either was legitimizing the other, so much as both were legitimizing GiveDirectly. It seems valuable to legitimize GiveDirectly, especially among the “left-ish academic circles” reading Émile who might otherwise believe that Émile is against all EA causes/organizations. (And among “left-ish academics” who might otherwise believe that Nathan scorns “near-termist” causes.)
There’s a lot of cause prioritization disagreement within EA, but it doesn’t usually get vicious, in part because EAs have “skin in the game” with regard to using their time & money in order to make the world a better place. One hypothesis is that if we can get Émile’s audience to feel some genuine curiosity about how to make their holiday giving effective, they’ll wonder why some people are longtermists. I think it’s absolutely fine to disagree with longtermism, but I also think that longtermists are generally thoughtful and well-intentioned, and it’s worth understanding why they give to the causes they do.
most of the donations funging heavily against other EA causes
Do you have specific reasons to believe this? It’s a possibility, but I could just as easily see most donations coming from non-EAs, or EAs who consider GiveDirectly a top pick anyways. Even if EA donors didn’t consider GiveDirectly a top pick on its own, they might have considered “GiveDirectly plus better relations with Émile with no extra cost” to be a top pick, and I feel hesitant to judge this more harshly than I would judge any other EA cause prioritization.
BTW, a mental model here is:
...it is striking by how often these shifts in opinion appear, upon closer inspection, to be triggered by Torres experiencing a feeling of rejection, such as being denied a job, not being invited to a podcast, or having a book collaboration terminated. Torres’s subsequent “realization” that these people and communities, once held in such high esteem, were in fact profoundly evil or dangerous routinely comes after those personal setbacks, as a post hoc rationalization.
If Émile is motivated to attack EA because they feel rejected by it, it’s conceivable to me that their motivation for aggression would decrease if a super kind and understanding therapist-type person listened to them really well privately and helped them feel heard & understood. The fundraiser thing makes me think this could work if done well, although the Helen Pluckrose thing from Mark’s post makes me think it’s risky. But if it’s private, especially from a person who’s not particularly well-known, I assume it wouldn’t run the specific risk of legitimization.
1 - Choosing opponents is sometimes not up to you. As an analogy, being in a physical fight only takes one party to throw punches. When debates start to have significant consequences socially and politically, it’s worth considering that letting hostile ideas spread unchallenged may work out badly in the future.
2 - I’m not sure it’s clear that “the silent majority can often already see their mistakes” in this case. I don’t think this is a minor view on EA. I think a lot of people are sympathetic to Torres’ point of view, and a signficiant part of that is (in my opinion) because there wasn’t a lot of pushback when they started making these claims in major outlets.
On my first comment, I agree that I don’t think much could have been much done to stop Émile turning against EA,[1] but I absolutely don’t think it was inevitable that they would have had such a wide impact. They made the Bulletin of Atomic Scientists! They’re partnered with Timnit, who has large influence and sympathy in AI space! People who could have been potential allies in a coalition basically think our movement is evil.[2] They get sympathetically cited in academic criticisms of EA.
Was some pushback going to happen? Yes, but I don’t think inevitably at this scale. I do think more could have been done to actually push back on their claims that went over the line in terms of hostility and accuracy, and I think that could have led to a better climate at this critical juncture for AI discussions and policy where we need to build coalitions with communities who don’t fully agree with us.
My concern is that this new wave of criticism and attack on OpenPhil might not simply fade away but could instead cement an anti-EA narrative that could put the movement and the causes we care about at risk. So looking at what happened in the Torres case, and looking at the negative depictions of Open Philanthropy recently, ‘ignore it’ doesn’t look like a good strategy
I don’t think the hostility between the near-term harm and the AI x-riskers would have been prevented by more attack rebutting Emille Torres.
The real problem is that the near-term AI harm people perceive Ai x-riskers as ignoring their concerns and actively making the near term harms worse.
Unfortunately, I think this sentiment is at least partly accurate. When Timnit got pushed out of google for pointing out near-term harms of AI, there was almost no support from the x-risk crowd (I can’t find any big name EA’s on this list, for example). This probably contributed to her current anti-EA stance.
As for real world harms, well, we can just say that openAI was started by an x-risker, and has kickstarted an AI race, causing a myriad of real world harms such as scams, art plagiarism, data theft, etc.
The way to actually fix this would be actual solidarity and bridge building on dealing with the short-term harms of AI.
I don’t want to fully re-litigate this history, as I’m more concerned about the future of OpenPhilanthropy being blindsided by a political attack (it might be low probability, but you’d think OpenPhil would be open to being concerned about low-chance high-impact threats to it)
I don’t think the hostility between the near-term harm and the AI x-riskers would have been prevented by more attack rebutting Emille Torres.
Agreed. It predated Emile’s public anti-EA turn for sure. But it was never inevitable. Indeed, supporting Timnit during her firing from Google may have been a super low-cost way to show solidarity. It might have meant that Emile and Timnit wouldn’t have become allies who have strong ideological influence over a large part of AI research space.
The way to actually fix this would be actual solidarity and bridge building on dealing with the short-term harms of AI.
There seems to have been some momentum towards more collaboration after the AI Safety Summit though. I hope the Bletchley Declaration can be an inflection point for more of this.
‘The way to actually fix this would be actual solidarity and bridge building on dealing with the short-term harms of AI.’
What would this look like? I feel like, if all you do is say nice things, that is a good idea usually, but it won’t move the dial that much (and also is potentially lying, depending on context and your own opinions; we can’t just assume all concerns about short-term harm, let alone proposed solutions, are well thought out). But if you’re advocating spending actual EA money and labour on this, surely you’d first need to make a case that stuff “dealing with the short term harms of AI” is not just good (plausible), but also better than spending the money on other EA stuff. I feel like a hidden crux here might be that you, personally, don’t believe in AI X-risk*, so you think it’s an improvement if AI-related money is spent on short term stuff, whether or not that is better than spending it on animal welfare or global health and development, or for that matter anti-racist/feminist/socialist stuff not to do with AI. But obviously, people who do buy that AI X-risk is comparable/better as a cause area than standard near-term EA stuff or biorisk, can’t take that line.
*I am also fairly skeptical it is a good use of EA money and effort for what it’s worth, though I’ve ended up working on it anyway.
This seems a little zero-sum, which is not how successful social movements tend to operate. I’ll freely confess that I am on the “near term risk” team, but that doesn’t mean the two groups can’t work together.
A simplified example: Say 30% of a council are concerned abut near term harms, and 30% are concerned about x-risk, and each wants policies passed that address their own concerns. If the two spend all their time shitting on each other for having misplaced priorities, neither of them will get what they want. But if they work together, they have a majority, and can pass a combined bill that addresses both near-term harm and AI x-risk, benefiting both.
Unfortunately, the best time to do this bridge building and alliance making was several years ago, and the distrust is already rather entrenched. But I genuinely think that working to mend those bridges will make both groups better off in the long run.
You haven’t actually addressed the main question of the previous comment: What would this bridge building look like? Your council example does not match the current reality very well.
It feels like you also sidestep other stuff in the comment, and it is unclear what your position is. Should we spend EA money (or other resources) on “short-term harms”? If yes, is the main reason because funding the marginal AI ethics research is better than the marginal bed-net and the marginal AI xrisk research? Or would the main reason for spending money on “short-term harms” that we buy sympathy with the group of people concerned about “short-term harms”, so we can later pass regulations together with them to reduce both “short-term harm” and AI x-risk.
I just read most of the article. It was not that satisfying in this context. Most of it is arguments that we should work together (which I dont disagree with).
And I imagine it will be quite hard to convince most AI xrisk people of “whether AI is closer to a stupid ‘stochastic parrot’ or on the ‘verge-of-superintelligence’ doesn’t really matter; ”. If we were to adopt Gideon’s desired framing, it looks like we would need to make sacrifices in epistemics. Related:
The relevant question isn’t “are the important harms to be prioritised the existential harms or the non-existential ones?”, “will AI be agents or not?’, nor ’will AI be stochastic parrots or superintelligence?” Rather, the relevant question is whether we think that power-accumulation and concentration in and through AI systems, at different scales of capability, is extremely risky.
Some of Gideon’s suggestion such as protest or compute governance are already being pursued. Not sure if that counts as bridge building though, because these might be good ideas anyways.
For the record, I’m very willing to be corrected/amend my Quick Take (and my beliefs on this is in general) if “ignore it” isn’t an accurate summary of what was done. Perhaps there was internal action taken with academic spaces/EA organisations that I’m not aware of? I still think the net effect of EA actions in any case was closer to “ignore it”, but the literal strong claim may be incorrect.
Edit: I actually think these considertions should go for many of the comments in this sub-thread, not just my own. There’s a lot to disagree about, but I don’t think any comment in this chain is worthy of downvotes? (Especially strong ones)
A meta-thought on this take given the discussion its generated.
Currently this is at net 10 upvotes from 20 total votes at time of writing—but is ahead 8 to 6 on agree/disagree votes. Based on Forum voting norms, I don’t think this is particularly deserving of downvotes given the suggested criteria? Especially strong ones? Disagreevotes—go ahead, be my guest! Comments pointing out where I’ve gone wrong—I actively encourage you to do so!
I put this in a Quick Take, not a top-level post so it’s not as visible on the Forum front page. (and the point of a QT is for ’exploratory, draft-stage, rough thoughts like this).I led off with saying “I think”—I’m just voicing my concerns about the atmosphere surrounding OpenPhil and its perception. It’s written in good faith, albeit with a concerned tone. I don’t think it violates what the EA Forum should be about.[1]
I know these kind of comments are annoying but still, I wanted to point out that this vote distribution feels a bit unfair, or at least unexplained to me. Sure, silent downvoting is a signal, but it’s a crude and noisy signal and I don’t really have much to update off here.
If you downvoted but don’t want to get involved in a public discussion about it, feel free to send me a DM with feedback instead. We don’t have to get into a discussion about the merits (if you don’t want to!), I’m just confused at the vote distribution.
The harsh crticism of EA has only been a good thing, forcing us to have higher standards and rigour. We don’t want an echochamber.
I would see it as a thoroughly good thing if Open Philanthropy were to combat the protrayal of itself as a shadowy cabal (like in the recent politico piece) for example by:
Having more democratic buy-in with the public
e.g. Having a bigger public presence in media, relying on a more diverse pool of funding than (i.e. less billionarie funding)
Engaged in less political lobbying
More transparent about the network of organisations around them
e.g. from the Politico article: ”… said Open Philanthropy’s use of Horizon … suggests an attempt to mask the program’s ties to Open Philanthropy, the effective altruism movement or leading AI firms”
I am not convinced that “having a bigger public presence in media” is a reliable way to get democratic buy-in. (There is also some “damned if you, damned if you don’t” dynamic going on here—if OP was constantly engaging in media interactions, they’d probably be accused of “unduly influencing the discourse/the media landscape”)
Could you describe what a more democratic OP would look like?
You mention “less billionaire funding”—OP was built on the idea of giving away Dustin’s and Cari’s money in the most effective way. OP is not fundraising, it is grantmaking! So how could it, as you put it, “rely on a more diverse pool of funding”?
(also: https://forum.effectivealtruism.org/posts/zuqpqqFoue5LyutTv/the-ea-community-does-not-own-its-donors-money)
I also suspect we would see the same dynamic as above: If OP did actively try to secure additional money in the forms of government grants, they’d be maligned for absorbing public resources in spite of their own wealth.
I think a blanket condemnation of political lobbying or the suggestion to “do less” of it is not helpful. Advocating for better policies (in animal welfare, GHD, pandemic preparedness etc.) is in my view one of the most impactful things you can do. I fear we are throwing the baby out with the bathwater here.
Suing people nearly always makes you look like the assholes I think.
As for Torres, it is fine for people to push back against specific false things they say. But fundamentally, even once you get past the misrepresentations, there is a bunch of stuff that they highlight that various prominent EAs really do believe and say that genuinely does seem outrageous or scary to most people, and no amount of pushback is likely to persuade most of those people otherwise.
In some cases, I think that outrage fairly clearly isn’t really justified once you think things through very carefully: i.e. for example the quote from Nick Beckstead about saving lives being all-things-equal higher value in rich countries, because of flow-through effects which Torres always says makes Beckstead a white supremacist. But in other cases well, it’s hardly news that utilitarianism has a bunch of implications that strongly contradict moral commonsense, or that EAs are sympathetic to utilitarianism. And ‘oh, but I don’t endorse [outrageous sounding view], I merely think there is like a 60% chance it is true, and you should be careful about moral uncertainty’ does not sound very reassuring to a normal outside person.
For example, take Will on double-or-nothing gambles (https://conversationswithtyler.com/episodes/william-macaskill/) where you do something that has a 49% chance of destroying everyone, and a 51% chance of doubling the number of humans in existence (now and in the future). It’s a little hard to make out exactly what Will’s overall position on this, but he does say it is hard to justify not taking those gambles:
‘Then, in this case, it’s not an example of very low probabilities, very large amounts of value. Then your view would have to argue that, “Well, the future, as it is, is like close to the upper bound of value,” in order to make sense of the idea that you shouldn’t flip 50⁄50. I think, actually, that position would be pretty hard to defend, is my guess. My thought is that, probably, within a situation where any view you say ends up having pretty bad, implausible consequences’
And he does seem to say there are some gambles of this kind he might take:
’Also, just briefly on the 51/49: Because of the pluralism that I talked about — although, again, it’s meta pluralism — of putting weight on many different model views, I would at least need the probabilities to be quite a bit wider in order to take the gamble...’
Or to give another example, the Bostrom and Shulman paper on digital minds talks about how if digital minds really have better lives than us, than classical (total) utilitarianism says they should take all our resources and let us starve. Bostrom and Shulman are against that in the paper. But I think it is fair to say they take utilitarianism seriously as a moral theory. And lots of people are going to think taking seriously the idea that this could be right is already corrupt, and vaguely Hitler-ish/reminiscent of white settler expansionism against Native Americans.
In my view, EAs should be more clearly committed to rejecting (total*) utilitarianism in these sorts of cases than they actually are. Though I understand that moral philosophers correctly think the arguments for utilitarianism, or views which have similar implications to utilitarianism in these contexts, are disturbingly strong.
*In both of the cases described, person-affecting versions of classical utilitarianism which deny creating happy people is good doesn’t have the scary consequences.
First, I want to thank you for engaging David. I get the sense we’ve disagreed a lot on some recent topics on the Forum, so I do want to say I appreciate you explaining your point of view to me on them, even if I do struggle to understand. Your comment above covers a lot of ground, so if you want to switch to a higher-bandwidth way of discussing them, I would be happy to. I apologise in advance if my reply below comes across as overly hostile or in bad-faith—it’s not my intention, but I do admit I’ve somewhat lost my cool on this topic of late. But in my defence, sometimes that’s the appropriate response. As I tried to summarise in my earlier comment, continuing to co-operate when the other player is defecting is a bad approach.
As for your comment/reply though, I’m not entirely sure what to make of it. To try to clarify, I was trying to understand why the Twitter discourse between people focused on AI xRisk and the FAact Community[1] has been so toxic over the last week, almost entirely (as far as I can see) from the latter to the former. Instead, I feel like you’ve steered the conversation away to a discussion about the implications of naïve utilitariansim. I also feel we may disagree on how much Torres has legitimate criticisms and how of their work is simply wilful ‘misrepresentation’ (I wonder if you’ve changed your mind on Torres since last year?). There are definitely connections there, but I don’t think it’s quite the same conversation, and I think it somewhat telling that you responded to suggestions 3 & 4, and not 1 & 2, which I think are far less controversial (fwiw I agree that legal action should only be used once all other courses of actions have failed).
To clarify what I’m trying to get at here with some more examples, which I hope will be reasonably unobjectionable even if incorrect:
Yesterday Timnit again insinuated that William MacAskill was a eugenicist. You can read that tweet and I don’t think she means this in a ‘belongs to a historical tradition’ way I think she means it in a ‘this is what he believes’ way. I haven’t seen anyone from the FAact Community call this out. In fact, Margaret Mitchell’s response to Jess Whittlestone’s attempt to offer an olive branch was met with confusion that there’s any extreme behaviour amongst the AI Ethics field.
People working in AI Safety and/or associated with EA should therefore expect to be called eugencists, and the more Timnit’s perspective gains prominence that more they will have to deal with the consequences of this.
Noah Giansiracua’s thread that I linked in the last tweet is highly conspiratiorial, spreads reckless misinformation, and is often just wrong. But not only has he doubled down despite pushback,[2] but he today tried to bridge the Safety/Ethics divide today seemingly unware that trashing the other side in a 26 tweet screed is massively damaging to this goal.
This suggests that while AI Safety efforts to build bridges may have some success, there may a strong and connected group of scholars who will either not countenance it at all, or be happy to stick the knife in once the opportunity appears. If I were an AI Safety academic, I wouldn’t trust Noah.
In general, my hope is that work is going on behind the scenes and off Twitter to build bridges between the two camps. But a lot of names on the FAact side that seem to be more toxic are quite prominent, and given the culture of silence/bullying involved there (again, see the Rumman Chowdhury tweet in the original comment, with further evidence here) I’m not sure I feel as hopeful it will happen as I did in recent weeks.
The more I look into it, the more I see the hostility as asymmetric. I’d be very open to counter-evidence on this point, but I don’t see AI Safety people treating the other camp with such naked hostility, and definitely not from the more influential members of the movement, as far as I can tell. (And almost certainly not any more than usual over the past week or so? As I said, a lot of this seems to have kicked off post CAIS Letter).
My call to not ‘be passive’ was one in which I expect hostility to the field of AI Safety to continue, perhaps grow, and be amplified by influential figures in the AI space. I maintian the general EA media strategy of ignoring critics, and if engaging them only doing so with the utmost politeness, has been a net negative strategy, and will continue to be so if continued—with perhaps very bad consequences.
Anyway, I’d like to thank you for sharing your perspective, and I do hope my perceptions have been skewed to be too pessimistic. To others reading, I’d really appreciate hearing your thoughts on these topics, and points of view or explanations that might change my mind
I guess I thought my points about total utilitarianism were relevant, because ‘we can make people like us more by pushing back more against misrepresentation’ is only true insofar as the real views we have will not offend people. I’m also just generically anxious about people in EA believing things that feel scary to me. (As I say, I’m not actually against people correcting misrepresentations obviously.)
I don’t really have much sense of how reasonable critics are or aren’t being, beyond the claim that sometimes they touch on genuinely scary things about total utilitarianism, and that it’s a bit of a problem that the main group arguing for AI safety contains a lot of prominent people with views that (theoretically) imply that we should be prepared to take big chances of AI catastrophe rather than pass up small chances of lots of v. happy digital people.
On Torres specifically: I don’t really follow them in detail (these topics make me anxious), but I didn’t intend to be claiming that they are a fair or measured critic, just that they have decent technical understanding of the philosophical issues involved and sometimes puts their finger on real weaknesses. That is compatible with them also saying a lot of stuff that’s just false. I think motivated reasoning is a more likely explanation for why they says false things than conscious lying, but that’s just because that’s my prior about most people. (Edit: Actually, I’m a little less sure of that, after being reminded of the sockpuppetry allegations by quinn below. If those are true, that is deliberate dishonesty.)
Regarding Gebru calling Will a eugenicist. Well, I really doubt you could “sue” over that, or demonstrate to the people most concerned about this that he doesn’t count as one by any reasonable definition. Some people use “eugenicist” for any preference that a non-disabled person comes into existence rather than a different disabled person. And Will does have that preference. In What We Owe the Future, he takes it as obvious that if you have a medical condition that means if you conceive right now, your child will have awful painful migraines, then you should wait a few weeks to conceive so that you have a different child who doesn’t have migraines. I think plenty ordinary people would be fine with that and puzzled by Gebru-like reactions, but it probably does meet some literal definitions that have been given for “eugenics”. Just suggesting he is a “eugenicist” without further clarification is nonetheless misleading and unfair in my view, but that’s not quite what libel is. Certainly I have met philosophers with strong disability rights views who regard Will’s kind of reaction to the migraine case as bigoted. (Not endorsing that view myself.)
None of this is some kind of overall endorsement of how the ‘AI ethics’ crowd on Twitter talk overall, or about EAs specifically. I haven’t been much exposed to it, and when I have been, I generally haven’t liked it.
Does anyone work at, or know somebody who works at Cohere?
Last November their CEO Aidan Gomez published an anti-effective-altruism internal letter/memo (Bloomberg reporting here, apparently confirmed as true though no further comment)
I got the vibe from Twitter/X that Aidan didn’t like EA, but making an internal statement about it to your company seems really odd to me? Like why do your engineers and project managers need to know about your anti-EA opinions to build their products? Maybe it came after the AI Safety Summit?
Does anyone in the AI Safety Space of EA have any knowledge or context to help out here?
Ideally I’d like to read the memo and see what it’s definition of EA is and what its arguments are.
Given Cohere are a leading, though not frontier-level, AI company feels like it’s bad to have a fairly prominent CEO who hates EA so much he releases internal memos about it to the company.
I don’t think it’s odd at all. As the Bloomberg article notes, this was in response to the martyrdom of Saint Altman, when everyone thought the evil Effective Altruists were stealing/smashing OA for no reason and destroying humanity’s future (as well as the fortune of many of the individuals involved, to a degree few of them bothered to disclose) and/or turning it over to the Chinese commies. An internal memo decrying ‘Effective Altruism’ was far from the most extreme response at the time; but I doubt Gomez would write it today, if only because so much news has come out since then and it no longer looks like such a simple morality play. (For example, a lot of SV people were shocked to learn he had been fired from YC by Paul Graham et al for the same reason. That was a very closely held secret.)
Ah good point that it was in the aftermath of the OpenAI board weekend, but it still seems like a very extreme/odd reaction to me (though I have to note the benefit of hindsight as well as my own personal biases). I still think it’d be interesting to see what Aidan actually said, and/or why he’s formed such a negative opinion of EA, but I think your right than the simplest explanation here is:
Sam gets fired → I think evil EA is behind it → I need to show how not-EA I am → release anti-EA internal memo
I’d be a bit surprised if you could find people on this forum who (still) work at Cohere. Hard to see a stronger signal to interview elsewhere than your CEO explaining in a public memo why they hate you.
but making an internal statement about it to your company seems really odd to me? Like why do your engineers and project managers need to know about your anti-EA opinions to build their products?
I agree it’s odd in the sense that most companies don’t do it. I see it as a attempt to enforce a certain kind of culture (promoting conformity, discouragement of dissent, “just build now” at the expense of ethics, etc) that I don’t much care for. But the CEO also made it abundantly clear he doesn’t like people who think like me either, so ¯\_(ツ)_/¯.
I’ve generally been quite optimistic that the increased awareness AI xRisk has got recently can lead to some actual progress in reducing the risks and harms from AI. However, I’ve become increasingly sad at the ongoing rivalry between the AI ‘Safety’ and ‘Ethics’ camps[1] 😔 Since the CAIS Letter was released, there seems to have been an increasing level of hostility on Twitter between the two camps, though my impression is that the holistility is mainly one-directional.[2]
I dearly hope that a coalition of some form can be built here, even if it is an uneasy one, but I fear that it might not be possible. It unfortunately seems like a textbook case of mistake vs conflict theory approaches at work? I’d love someone to change my mind, and say that Twitter amplifies the loudest voices,[3] and that in the background people are making attempts to build bridges. But I fear that instead that the centre cannot hold, and that there will be not just simmering resentment but open hostility between the two camps.
If that happens, then I don’t think those involved in AI Safety work can afford to remain passive in response to sustained attack. I think that this has already damaged the prospects of the movement,[4] and future consequences could be even worse. If the other player in your game is constantly defecting, it’s probably time to start defecting back.
Can someone please persuade me that my pessimism is unfounded?
I’m open to be corrected here, but I feel like those sceptical of the AI xRisk/AI Safety communities have upped the ante in terms of the amount of criticism and its vitriol—though I am open to the explanation that I’ve been looking out for it more
The EA community’s silence regarding Torres has led to the acronym ‘TESCREAL’ gaining increasing prominence amongst academic circles—and it is not a neutral one, and gives them more prominence and a larger platform.
I can’t say I have a strategy David. I’ve just been quite upset and riled up by the discourse over the last week just as I had gained some optimism :( I’m afraid that by trying to turn the other cheek to hostility, those working to mitigate AI xRisk end up ceding the court of public opinion to those hostile to it.
I think some suggestions would be:
Standing up to, and callying out, bullying in these discussions can cause a preference cascade of pushback to it—see here—but someone needs to stand up for people to realise that dominant voices are not representative of a field, and silence may obscure areas for collaboration and mutual coalitions to form.
Being aware of what critiques of EA/AI xRisk get traction in adjacent communities. Some of it might be malicious, but a lot of it seems to be a default attitude of scepticism merged with misunderstandings. While not everyone would change their mind, I think people reaching ‘across the aisle’ might correct the record in many people’s minds. Even if not for the person making the claims, perhaps for those watching and reading online.
Publicly pushing back on Torres. I don’t know what went down when they were more involved in the EA movement that caused their opinion to flip 180 degrees, but I think the main ‘strategy’ has been to ignore their work and not respond to their criticism. The result: their ideas gaining prominence in the AI Ethics field, publications in notable outlets, despite acting consistently in bad faith. To their credit, they are voraciously productive in their output and I don’t expect to it slow down. Continuing with a failed strategy doesn’t sound like the right call here.
In cases of the most severe hostility, potential considering legal or institutional action? In this example, can you really just get away with calling someone a eugenicist when it’s so obviously false? But there have been cases where people have successfully sued for defamation for statements made on Twitter. That’s an extreme option though, but not worth ignoring entirely.
I would recommend trying to figure out how much loud people matter. Like it’s unclear if anyone is both susceptible to sneer/dunk culture and potentially useful someday. Kindness and rigor come with pretty massive selection effects, i.e., people who want the world to be better and are willing to apply scrutiny to their goals will pretty naturally discredit hostile pundits and just as naturally get funneled toward more sophisticated framings or literatures.
I don’t claim this attitude would work for all the scicomm and public opinion strategy sectors of the movement or classes of levers, but it works well to help me stay busy and focused and epistemically virtuous.
Some of my posts/comments can be quite long—I like responding with as much context as possible on the Forum, but as some of the original content itself is quite long, that means my responses can be quite long! I don’t think that’s necessarily a problem in itself, but the problem then comes with receiving disagree votes without comments elaborating them.
<I want to say, this isn’t complaining about disagreement. I like disagreement[2], it means I get to test my ideas and arguments>
However, it does pose an issue with updating my thoughts. A long post that has positive upvotes, negative disagree votes, and no (or few) comments means it’s hard for me to know where my opinion differs from other EA Forum users, and how far and in what direction I ought to update in. The best examples from my own history:
My takeaways from Cremer’s Vox piece. Do the disagree votes apply to Cremer’s piece, my (relatively) positive response to it, or the specific points I highlighted, or all of the above?
Post shorter takes: This means the agree/disagree signal will be more clearly linked to my context, but it means I won’t be able to add all the content that I do now, which I think adds value.
Post fewer takes: Ceteris paribus this might not be expected work, but the argument would be that with more time between fewer contributions, the quality of them would go up so they’d usually make one clear point for readers to agree/disagree with
Post more takes: Splitting a large comment into lots of sub-comments would do the same thing as ‘post shorter takes’, and keep context. The cost is a karma-farming effect, and splitting a thread into multiple parts, making it harder to follow.
Explicitly ask for comments: I feel like this wouldn’t work? It feels a bit whiny and has no enforcement mechanism?
Pre-commit to up-voting responses: Relies on others to trust me, might not be credible for those who strongly disagree with me.
status: very rambly. This is an idea I want to explore in an upcoming post about longtermism, would be grateful to anyone’s thoughts. For more detailed context, see https://plato.stanford.edu/entries/time/ for debate on the nature of time in philosophy
Does rejecting longtermism require rejecting the B-Theory of Time (i.e. eternalism, the view that the past, present, and future have the same ontological status)? Saying that future people don’t exist (and therefore can’t be harmed, can’t lose out by not existing, don’t have the same moral rights as present ‘existing’ people) also implies that the future they live in (i.e. the city, country, planet, solar system, galaxy etc) doesn’t exist. This seems to fly in the face of our best understandings of physics (i.e. Relativity and Quantum Physics don’t have a place for a special ‘present’ afaik, in fact Relativity suggests that there is no such thing as a universal present)[1] I’m not sure people grapple with that contradiction.
Furthermore, if you want to claim the special ‘present’ moment does exist, and that our moral obligations only hold in that frame of time, doesn’t that mean that our obligations to the past don’t matter? Many have criticised Bostrom’s use of ‘mere ripples’ to describe historical atrocities,[2] but doesn’t believing in a unique present moment imply that the past doesn’t exist, and hence those atrocities don’t exist either and that we have no obligations that refer to past events or people? One could get around this by assuming a ‘growing block’ theory of time, where the past exists, as does the present, but the future doesn’t. But if you believe in a special, unique present then asserting the past exists but the future doesn’t seems no more valid than the other way around?
I actually think that has legs as more than a niche philosophical point, but would be very interested in hearing others’ thoughts.
FWIW, someone could reject longtermism for reasons other than specific person-affecting views or even pure time preferences.
Even without a universal present, there’s still your present (and past), and you can do ethics relative to that. Maybe this doesn’t seem impartial enough, and could lead to agents with the same otherwise impartial ethical views and same descriptive views disagreeing about what to do, and those are undesirable?
OTOH, causality is still directed and we can still partially order events that way or via agreement across reference frames. The descendants of humanity, say, 1000 years from your present (well, humanity here in our part of the multiverse, say) are still all after your actions now, probably no matter what (physically valid) reference frame you consider, maybe barring time travel. This is because humans’ reference frames are all very similar to one another, as differences in velocity, acceleration, force and gravity are generally very small.
So, one approach could be to rank or estimate the value of your available options on each reference frame and weigh across them, or look for agreement or look for Pareto improvements. Right now for you, the different reference frames should agree, but they could come apart for you or other agents in the future if/when we or our descendants start colonizing space, traveling at substantial fractions of the speed of light.
Also, people who won’t come to exist don’t exist under the B-theory, so they can’t experience harm. Maybe they’re harmed by not existing, but they won’t be around to experience that. Future people could have interests, but if we only recognize interests for people that actually exist under the B-theory, then extinction wouldn’t be bad for those who never come to exist as a result, because they don’t exist under the B-theory.
Great response Michael! Made me realise I’m conflating a lot of things (so bear with me)
By longtermism, I really mean MacAskill’s:
future people matter
there could be a lot of them
our actions now could affect their future (in some morally significant way)
And in that sense, I really think only bullet 1 is the moral claim. The other two are empirical, about what our forecasts are, and what our morally relevant actions are. I get the sense that those who reject longtermism want to reject it on moral grounds, not empirical ones, so they must reject bullet-point 1. The main ways of doing so are, as you mention, person-affecting views or a pure-rate of time preference, both of which I am sceptical are without their difficult bullets to bite.[1]
The argument I want to propose is this:
The moral theories we regard as correct ought to cohere with our current best understandings of physics[2]
The Special Theory of Relativity (STR) is part of our current best understanding of physics
STR implies that there is no universal present moment (i.e. without an observer’s explicit frame of reference)
Some set of person-affecting views [P] assume that there is a universal present moment (i.e. that we can clearly separate some people as not in the present and therefore not worthy of moral concern, and others which do have this property)
From 3 & 4, STR and P are contradictory
From 1 & 5, we ought to reject all person-affecting views that are in P
And I would argue that a common naïve negative reactions to longtermism (along the liens of) “potential future people don’t matter, and it is evil to do anything for them at the expense of present people since they don’t exist” is in P, and therefore ought to be rejected. In fact, the only ways to really get out of this seem to be either that someone’s chosen person-affecting views are not in P, or that 1 is false. The former is open to question of course, and the latter seems highly suspect.
The point about certain people never existing under B-Theory is an interesting counter. Is it possible to have B-Theory without strong determinism (the kind that undermines moral responsibility)? I guess I’d reframe it as under B-Theory, future people exist, and so whoever can affect their wellbeing causally (whether from their reference frame of past, present, or future) ought to do so if they can. You could still believe that is empirically very difficult to do so, but it makes sense to me that outside some special sphere of moral obligation (say my family, children, or those who I have made promises too), the only difference between someone far away in time and close is the extent to which I can causally help them, and not their position in time—just the same way we might say that the reason to be sceptical of charity in different locations in the present is their effectiveness at aid and not their physical location in space.
That causal effectiveness (corresponding to opening bullet point 3) seems to me the most promising response to longtermism, not that those future people exist or not, or have moral value (bullet point 1), but it seems that’s what most objections lean on.
I don’t think having to bite bullets is a knockdown refutation, but if your view has them I want to see evidence you’ve included them in your meal, especially if criticise others for not biting their own bullets
Has anyone else listened to the latest episode of Clearer Thinking? Spencer interviews Richard Lang about Douglas Harding’s “Headless Way”, and if you squint enough it’s related to the classic philosophical problems of consciousness, but it did remind me a bit of Scott A’s classic story “Universal Love, Said The Cactus Person” which made me laugh. (N.B. Spencer is a lot more gracious and inquisitive than the protagonist!)
But yeah if you find the conversation interesting and/or like practising mindfulness meditation, Richard has a series of guided meditations on the Waking Up App, so go and check those out.
In this comment I was going to quote the following from R. M. Hare:
“Think of one world into whose fabric values are objectively built; and think of another in which those values have been annihilated. And remember that in both worlds the people in them go on being concerned about the same things—there is no difference in the ‘subjective’ concern which people have for things, only in their ‘objective’ value. Now I ask, What is the difference between the states of affairs in these two worlds? Can any other answer be given except ‘None whatever’?”
I remember this being quoted in Mackie’s Ethics during my undergraduate degree, and it’s always stuck with me as a powerful argument against moral non-naturalism and a close approximation of my thoughts on moral philosophy and meta-ethics.
But after some Google-Fu I couldn’t actually track down the original quote. Most people think it comes from the Essay Nothing Matters in Hare’s Applications of Moral Philosophy. While this definitely seems to be in the same spirit of the quote, the online scanned pdf version of Nothing Matters that I found doesn’t contain this quote at all. I don’t have access to any of the academic institutions to check other versions of the paper or book.
Maybe I just missed the quote by skimreading too quickly? Are there multiple versions of the article? Is it possible that this is a case of citogenesis? Perhaps Mackie misquoted what R.M. Hare said, or perhaps misattributed it and and it actually came from somewhere else? Maybe it was Mackie’s quote all along?
Help me EA Forum, you’re my only hope! I’m placing a £50 bounty to a charity of your choice for anyone who can find the original source of this quote, in R. M. Hare’s work or otherwise, as long as I can verify it (e.g. a screenshot of the quote if it’s from a journal/book I don’t have access to).
Many people find the Forum anxiety inducing because of the high amount of criticism. So, in the spirit of Giving Season, I’m going to give some positive feedback and shout-outs for the Forum in 2023 (from my PoV). So, without further ado, I present the 🎄✨🏆 totally-not-serious-worth-no-internet-points-JWS-2023-Forum-Awards: 🏆✨🎄[1]
Best Forum Post I read this year:
10 years of Earning to Give by @AGB: A clear, grounded, and moving look at what it actually means to ‘Earn to Give’. In particular, the ‘Why engage?’ section really resonated with me.
Honourable Mentions:
Why I’m still going out to bat for the EA movement post FTX by @Gemma Paterson: This post was a reflection on EAs tough year which was similar to my own journey, but was also able to make a case for why EA is worth fighting for.
Third Wave Effective Altruism by @Ben_West: A short post, but one that came at exactly the right time. I think this is a really important topic for EA to be thinking about right now and look forward to more posts inspired by it next year.
The Seeker’s Game – Vignettes from the Bay by @Yulia: I really enjoyed both the human perspective of this post, and the style with which it was written.
Best Sequence I read this year:
The CURVE Sequence: Causes and Uncertainty: Rethinking Value in Expectation from Rethink Priorities: This sequence was utterly brilliant. All its posts are well-written and led to good discussion from the community. I particularly want to shout out Arvo’s piece that stress-tested what EV from x-risk looks like under specific assumptions.
EA Forum Posters of the Year:
@titotal - I really value titotal’s contributions to the Forum. Even though I sometimes disagree I always learn a lot by thinking about where and why I do so, and I hope they stick around so I can continue to learn from them.
@Holly_Elmore - Similarly to Titotal, while I don’t always agree with Holly she just makes really solid points that always positively add to the discussion. Often she’s able to clearly find the crux of a debate and ask the key questions.
@Ivy Mazzola - Whenever I’ve seen Ivy post on the Forum, even on heated topics, she always approaches them with integrity and grace that I really wanted to give recognition to.
@David Mathers - When David discusses ideas in the Forum he consistently does so in a good faith and truth-seeking way that really shows engagement on the Forum at its best.
Non-Forum EA Poasters of the Year:
David Thorstad (on his blog) - I’ve really appreciated David’s blog this year. I don’t always agree, and you can find me in the comments pushing back sometimes, I’ve found his content high quality overall, especially on Existential Risk and the Time of Perils.
Agustín (on Twitter) - Agus has been really fighting the good fighting in the Twitter trenches recently, on all sorts of topics (though sadly on the terrible state of AI discourse). Somehow he manages to translate complicated EA ideas clearly, convincingly, and remain calm in the face of less-than-charitable interlocutors.
Nathan Young (on Twitter) - Nathan’s commitment to honest exploration of his beliefs, dialogue with those who disagree, and finding mechanisms of belief resolution has been constantly inspiring.
Congrats to the winners! Feel free to give them more plaudits in the comments or just shout out your own Forum favourites from this year! I wish everyone who’s reading this every success in the year to come and I look forward to reading more good content on the Forum in 2024!
Note that all of the award categories and choices are arbitrary, this is an intensely vibes-based post
Thanks for saying nice things about me! For the record, I also think David Thorstad’s contributions are very valuable (whether or not his views are ultimately correct).
This is a lovely idea. Bravo!
The HLI discussion on the Forum recently felt off to me, bad vibes all around. It seems very heated, not a lot of scout mindset, and reading the various back-and-forth chains I felt like I was ‘getting Eulered’ as Scott once described.
I’m not an expert on evaluating charities, but I followed a lot of links to previous discussions and found this discussion involving one of the people running an RCT on Strongminds (which a lot of people are waiting for the final results of) who was highly sceptical of SM efficacy. But the person offering counterarguments in the thread seems to be just as valid to me? My current position, for what it’s worth,[1] is:
the initial Strongminds results of 10x cash transfer should raise a sceptical response. most things aren’t that effective
it’s worth there being exploration of what the SWB approach would recommend as the top charities (think of this as trying other bandits in a multi-armed bandit charity evaluation problem)
it’s very difficult to do good social science, and the RCT won’t give us dispositive evidence about the effectiveness of Strongminds (especially at scale), but it may help us update. In general we should be mindful of how far we can make rigorous empirical claims in the social sciences
HLI has used language too loosely in the past and overclaimed/been overconfident, which Michael has apologised for, though perhaps some critics would like a stronger signal of neutrality (this links to the ‘epistemic probation’ comments)
GiveWell’s own ‘best guess’ analysis seems to be that Strongminds is 2.3x that of GiveDirectly.[2] I’m generally a big fan of the GiveDirectly approach for reasons of autonomy—even if Strongminds got reduced in efficacy to around ~1x GD, it’d still be a good intervention? I’m much more concerned with what this number is than the tone of HLIs or Michael’s claims tbh (though not at the expense of epistemic rigour).
The world is rife with actively wasted or even negative action, spending, and charity. The integrity of EA research, and holding charity evaluators to account is important to both the EA mission and EAs identity. But HLI seems to have been singled out for very harsh criticism,[3] but so much of the world is worse.
I’m also quite unsettled by a lot of what I call ‘drive-by downvoting’. While writing a comment is a lot more effort than clicking to vote on a comment/post, I think the signal is a lot higher, and would help those involved in debates reach consensus better. Some people with high-karma accounts seem to be making some very strong votes on that thread, and very few are making their reasoning clear (though I salute those who are in either direction).
So I’m very unsure how to feel. It’s an important issue, but I’m not sure the Forum has shown itself in a good light in this instance.
And I stress this isn’t much in this area, I generally defer to evaluators
On the table at the top of the link, go to the column ‘GiveWell best guess’ and the row ‘Cost-effectiveness, relative to cash’
Again, I don’t think I have the ability to adjudicate here, which is part of why I’m so confused.
I think this is a significant datum in favor of being able to see the strong up/up/down/strong down spread for each post/comment. If it appeared that much of the karma activity was the result of a handful of people strongvoting each comment in a directional activity, that would influence how I read the karma count as evidence in trying to discern the community’s viewpoint. More importantly, it would probably inform HLI’s takeaways—in its shoes, I would treat evidence of a broad consensus of support for certain negative statements much, much more seriously than evidence of carpet-bomb voting by a small group on those statements.
Indeed our new reacts system separates them. But our new reacts system also doesn’t have strong votes. A problem with displaying the number of types of votes when strong votes are involved is that it much more easily allows for deanonymization if there are only a few people in the thread.
That makes sense. On the karma side, I think some of my discomfort comes from the underlying operationalization of post/comment karma as merely additive of individual karma weights.
True opinion of the value of the bulk of posts/comments probably lies on a bell curve, so I would expect most posts/comments to have significantly more upvotes than strong upvotes if voters are “honestly” conveying preferences and those preferences are fairly representative of the user base. Where the karma is coming predominately from strongvotes, the odds that the displayed total reflects the opinion of a smallish minority that feels passionately is much higher. That can be problematic if it gives the impression of community consensus where no such consensus exists.
If it were up to me, I would probably favor a rule along the lines of: a post/comment can’t get more than X% of its net positive karma from strongvotes, to ensure that a high karma count reflects some degree of breadth of community support rather than voting by a small handful of people with powerful strongvotes. Downvotes are a bit trickier, because the strong downvote hammer is an effective way of quickly pushing down norm-breaking and otherwise problematic content, and I think putting posts into deep negative territory is generally used for that purpose.
Looks like this feature is being rolled out on new posts. Or at least one post: https://forum.effectivealtruism.org/posts/gEmkxFuMck8SHC55w/introducing-the-effective-altruism-addiction-recovery-group
EA is just a few months out from a massive scandal caused in part by socially enforced artificial consensus (FTX), but judging by this post nothing has been learned and the “shut up and just be nice to everyone else on the team” culture is back again, even when truth gets sacrificed on the process. No thinks HLI is stealing billions of dollars of course, but the charge that they keep quasi-deliberately stacking the deck in StrongMinds’ favour is far from outrageous and should be discussed honestly and straightforwardly.
JWS’ quick take has often been in negative agreevote territory and is +3 at this writing. Meanwhile, the comments of the lead HLI critic suggesting potential bad faith have seen consistent patterns of high upvote / agreevote. I don’t see much evidence of “shut up and just be nice to everyone else on the team” culture here.
Hey Sol, some thoughts on this comment:
I don’t think the Forum’s reaction to the HLI post has been “shut up and just be nice to everyone else on the team”, as Jason’s response suggested.
I don’t think mine suggests that either! In fact, my first bullet point has a similar sceptical prior to what you express in this comment[1] I also literally say “holding charity evaluators to account is important to both the EA mission and EAs identity”, and point that I don’t want to sacrifice epistemic rigour. In fact, one of my main points is that people—even those disagreeing with HLI, are shutting up too much! I think disagreement without explanation is bad, and I salute the thorough critics on that post who have made their reasoning for putting HLI in ‘epistemic probation’ clear.
I don’t suggest ‘sacrificing the truth’. My position is that the truth on StrongMind’s efficacy is hard to get a strong signal on, and therefore HLI should have been more modest early on their history, instead of framing it as the most effective way to donate.
As for the question of whether HLI were “quasi-deliberately stacking the deck”, well I was quite open that I think I am confused on where the truth is, and find it difficult to adjudicate what the correct takeway should be.
I don’t think we really disagree that much, and I definitely agree that the HLI discussion should proceed transparently and EA has a lot to learn from the last year, including FTX. I think if you maybe re-read my Quick Take, I’m not taking the position you think I am.
That’s my interpretation of course, please correct me if I’ve misunderstood
I want to register that my perspective on medium-term[1] AI existential risk (shortened to AIXR from now on) has changed quite a lot this year. Currently, I’d describe it as moving from ‘Deep Uncertainty’ to ‘risk is low in absolute terms, but high enough to be concerned about’. I guess atm I’d think that my estimates are moving closer toward the Superforecasters in the recent XPT report (though I’d still say I’m still Deeply Uncertain on this issue, to the extent that I don’t think the probability calculus is that meaningful to apply)
Some points around this change:
I’m not sure it’s meaningful to cleanly distinguish AIXR from other anthropogenic x-risks, especially since negative consequences of AI may plausibly increase other x-risks (e.g. from Nuclear War, biosecurity, Climate Change etc.)
I think in practice, the most likely risks from AI would come from deployment of powerful systems that have catastrophic consequences that are then rolled back. I’m think of Bing ‘Sydney’ here as the canonical empirical case.[2] I just don’t believe we’re going to get no warning shots.
Similary, most negative projections of AI don’t take into account negative social reaction and systematic human response to these events. You either assume we’ll get no warning shots and then get exterminated (‘sharp left turn’) or think that humanity is doomed to not co-operate (‘Moloch’). I think that the evidence suggests instead that societies and governments will react against increasing AI capability if they view it negatively, rather than simply stand by and watch it happen, which is what many AIXR seem to assume or at least imply to me.
I think the AIXR community underestimated the importance of AI Governance and engaging with politics. Instead of politicians/the public ‘not getting it’, they infact seem to have ‘got it’. The strategy of only letting people vetted to be smart enough think about AIXR seems to have been flawed, the same kind of thinking that led to thinking a ‘pivotal act’ strategy was viable.
In fact, I’ve been pleasantly surprised by how welcoming both politicians and the public have been to the Overton Window being opened. Turns out the median voter in a liberal democracy doesn’t like ‘let large co-orporations create powerful models with little understanding of their consequences without significant oversight’. I think people’s expectations of positive co-ordination on AI issues should have gone up this year, and conversely it should make your AIXR estimate go down unless you think relative alignment progress has declined even more.
While there have been an awful lot of terrible arguments against AIXR raised this year, some have been new and valuable to me. Some examples:
As titotal has written, I expect ‘early AGI’ will be a ‘buggy mess’. More specifically, I doubt it could one-shot solve ‘take over the world’ unless you assume god-like capability by assumption, instead of ‘more capable than humans in some important ways, but not invincible’.
Progress via ‘stack moar layers lol’ will plausibly slow down, or at least run into some severe issues.[3] The current hype-wave seems to be just following this as far as it will go with compute and data, rather than exploring alternative architectures that could be much more efficient.
I’m still unimpressed by how Frontier systems perform on tests such as The ARC Challenge which test hypothesis creativity and testing in a few-shot scenario (and on hidden data) without ‘ground truth’, as opposed to training on trillions-upon-trillions of training examples with masks and true labels.
Related to the above, I view creativity and explanation as critical to science and the progress of science, so I’m not sure what ‘automation of science’ would actually look like. It makes me very sceptical of claims like Davidson’s with larger numbers like ’1000x capability’ (edit, it’s actually 1000x in a year! Is that a one-time increase or perpetual explosive growth? It just seems way too strong a claim).[4]
Progress in AI Safety has empirically been correlated, at least weakly, with increases in capability. I didn’t agree with everything in Nora’s recent post, but I think she’s right about assessing a fully thoeretical approach to progress (such as MIRI’s strategy, as far as I can tell).
In summary, it doesn’t seem like the field of AIXR comes out particularly strongly to me on the I-T-N Framework relative to other existential and catastrophic risks. It may be ‘more important’, but seems similarly intractable in the style that these global problems generally are, and definitely not as neglected any more (in terms of funding, where does it stand relative to nuclear or bio for example? would be interesting to know).
However, I still think x-risk and catastrophic risk from AI this century is unjustifiably high. I just don’t think it’s plausible to have levels of pDoom ~>90% under current evidence unless you have private insight.
I think the main existential/catastrophic issues around AI in the foreseeable future revolve around political institutions and great power conflict, rather than humans being wiped out agentically (either deceptively by a malicious power-seeker or unintentionally by an idiot-savant)
Anyway, these thoughts aren’t fully worked out, I’m still exploring what I think on this issue, but just wanted to register where my current thoughts are at in case it helps others in the community.
Say on a ~50 year time scale, or to the end of the century
Clarification: I’m not saying ‘Sydney’ had catastrophic consequences, but that a future system could be released in a similar way due to internal corporate pressures, and that system could then act negatively in the real world in a way its developers did not expect.
Btw, xuan’s twitter is one of the best I know of to get insights into the state of AI. Sometimes I agree, sometimes I don’t, but her takes are always legit
What does it even mean to say 1,000x capability, especially in terms of science? Is there a number here that Davidson is tracking? When would he view it as falsified?
This seems like a very sensible and down-to-earth analysis to me, and I’m a bit sad I can’t seem to bookmark it.
Thanks :) I might do an actual post at the end of the year? In the meantime I just wanted to get my ideas out there as I find it incredibly difficult to actually finish any of the many Forum drafts I have 😭
Do the post :)
I agree this feels plenty enough to be a post for me, but we all have different thresholds I guess!
“AI may plausibly increase other x-risks (e.g. from Nuclear War, biosecurity, Climate Change etc.)”
I’m extremely surprised to see climate change listed here. Could you explain?
Honestly I just wrote a list of potential x-risks to make a similar reference class. It wasn’t mean to be a specific claim, just examples for the quick take!
I guess climate change might be less of an existential risk in an of itself (per Halstead), but there might be interplays between them that increase their combined risk (I think Ord talks about this in the precipice). I’m also sympathetic to Luke Kemp’s view that we should really just care about overall x-risk, regardless of cause area, as extinction by any means would be as bad for humanities potential.[1]
I think it’s plausible to consider x-risk from AI higher than Climate Change over the rest of this century, but my position at the moment is that this would be more like 5% v 1% or 1% v 0.01% than 90% v 0.001%, but as I said I’m not sure trying to put precise probability estimates is that useful.
Definitely accept the general point that it’d be good to be more specific with this language in a front-page post though.
Though not necessarily present, some extinctions may well be a lot worse than others there
My point is that even though AI emits some amount of carbon gases, I’m struggling to find a scenario where it’s a major issue for global warming as AI can help provide solutions here as well.
(Oh, my point wasn’t that climate change couldn’t be an x-risk, though it has been disputed, more that I don’t see the pathway for AI to exacerbate climate change).
I would take the proposal to be AI->growth->climate change or other negative growth side effects
I was wondering why he said that, since I’ve read his report before and that didn’t come up at all. I suppose a few scattered recollections I have are
Tom would probably suggest you play around with the takeoffspeeds playground to gain a better intuition (I couldn’t find anything 1,000x-in-a-year-related at all though)
Capabilities takeoff speed ≠ impact takeoff speed (Tom: “overall I expect impact takeoff speed to be slower than capabilities takeoff, with the important exception that AI’s impact might mostly happen pretty suddenly after we have superhuman AI”)
This is an off-the-cuff quick take that captures my current mood. It may not have a long half-life, and I hope I am wrong
Right now I am scared
Reading the tea-leaves, Altman and Brockman may be back at OpenAI, the company charter changed, and the board—including Toner and MacAulay—removed from the company
The mood in the Valley, and in general intellectual circles, seems to have snapped against EA[1]
This could be as bad for EA’s reputation as FTX
At a time when important political decisions about the future of AI are being made, and potential coalitions are being formed
And this time it’d be second-impact syndrome
I am scared EA in its current form may not handle the backlash that may come
I am scared that we have not done enough reform in the last year from the first disaster to prepare ourselves
I am scared because I think EA is a force for making the world better. It has allowed me to do a small bit to improve the world. Through it, I’ve met amazing and inspiring people who work tirelessly and honestly to actually make the world a better place. Through them, I’ve heard of countless more actually doing what they think is right and giving what they can to make the world we find ourselves is more full of kindness, happiness, and flourishing
I am scared because this could be lost, I don’t know what to do to prevent it
I am scared
Not just AI Safety, but EA as a whole. I don’t think those upset and concerned with EA really care about making that distinction
As with Nonlinear and FTX, I think that for the vast majority of people, there’s little upside to following this in real-time.
It’s very distracting, we have very little information, things are changing fast, and it’s not very action-relevant for most of us.
I’m also very optimistic that the people “who work tirelessly and honestly to actually make the world a better place” will keep working on it after this, whatever happens to “EA”, and there will still be ways to meet them and collaborate.
Sending a hug
Thanks for this Lorenzo, I appreciate it <3
It’s hard to see how the backlash could actually destroy GiveWell or stop Moskowitz and Tuna giving away their money through Open Phil/something that resembles Open Phil. That’s a lot of EA right there.
It’s hard yes, but I think the risk vectors are (note—these are different scenarios, not things that follow in chronological order, though they could):
Open Philanthropy gets under increasing scrutiny due to its political influence
OP gets viewed as a fully politicised propaganda operation from EA, and people stop associating with it, accepting its money, or call for legal or political investigations into it etc
Givewell etc dissassociate themselves from EA due to EA having a strong negative social reaction from potential collaborators or donors
OP/Givewell dissociate from and stop funding the EA community for similar reasons as the above, and the EA community does not survive
Basically I think that ideas are more important than funding. And if society/those in positions of power put the ideas of EA in the bin, money isn’t going to fix that
This is all speculative, but I can’t help the feeling that regardless of how the OpenAI crisis resolves a lot of people now consider EA to be their enemy :(
My general thoughts on this can be stated as: I’m mostly of the opinion that EA will survive this, bar something massively wrong like the board members willfully lying or massive fraud from EAs, primarily because most of the criticism is directed to the AI safety wing, and EA is more than AI safety, after all.
Nevertheless, I do think that this could be true for the AI safety wing, and they may have just hit a key limit to their power. In particular, depending on how this goes, I could foresee a reduction in AI safety power and influence, and IMO this was completely avoidable.
I think a lot will depend on the board justification. If Ilya can say “we’re pushing capabilities down a path that is imminently highly dangerous, potentially existentially, and Sam couldn’t be trusted to manage this safely” with proof that might work—but then why not say that?[1]
If it’s just “we decided to go in a different direction”, then firing him and demoting Brockman with little to no notice, and without informing their largest business partner and funder, its bizarre that they took such a drastic step in the way they did
I was actually writing up my AI-risk sceptical thoughts and what EA might want to take from that, but I think I might leave that to one side for now until I can approach it with a more even mindset
Putting aside that I feel both you and I are sceptical that a new capability jump has emerged, or that scaling LLMs is actually a route to existential doom
I suspect this is due to the fact that quite frankly, the concerns they had about Sam Altman being unsafe on AI basically had no evidence except speculation from the EA/LW forum, which is not enough evidence at all in the corporate world/legal world, and to be quite frank, the EA/LW standard of evidence on AI risk being a big deal enough to investigate is very low, sometimes non-existent, and that simply does not work once you have to deal with companies/the legal system.
More generally, EA/LW is shockingly loose, sometimes non-existent in its standards of evidence for AI risk, which doesn’t play well with the corporate/legal system.
This is admittedly a less charitable take than say, Lukas Gloor’s take.
Haha, I was just going to say that I’d be very surprised if the people on the OpenAI board didn’t have access to a lot more info than the people on the EA forum or Lesswrong, who are speculating about the culture and leadership at AI labs from the sidelines.
TBH, if you put a randomly selected EA from a movement of 1,000s of people in charge of the OpenAI board, I would be very concerned that a non-trivial fraction of them probably would make decisions the way you describe. That’s something that EA opinion leaders could maybe think about and address.
But I don’t think most people who hold influential positions within EA (or EA-minded people who hold influential positions in the world at large, for that matter) are likely to be that superficial in their analysis of things. (In particular, I’m strongly disagreeing with the idea that it’s likely that the board “basically had no evidence except speculation from the EA/LW forum”. I think one thing EA is unusually good at – or maybe I should say “some/many parts of EA are unusually good at” – is hiring people for important roles who think for themselves and have generally good takes about things and acknowledge the possibility of being wrong about stuff. [Not to say that there isn’t any groupthink among EAs. Also, “unusually good” isn’t necessarily that high of a bar.])
I don’t know for sure what they did or didn’t consider, so this is just me going off of my general sense for people similar to Helen or Tasha. (I don’t know much about Tasha. I’ve briefly met Helen but either didn’t speak to her or only did small talk. I read some texts by her and probably listened to a talk or two.)
While I generally agree that they almost certainly have more information on what happened, which is why I’m not really certain on this theory, my main reason here is that for the most part, AI safety as a cause basically managed to get away with incredibly weak standards of evidence for a long time, until the deep learning era in 2019-, especially with all the evolution analogies, and even now it still tends to have very low standards (though I do believe it’s slowly improving right now). This probably influenced a lot of EA safetyists like Ilya, who almost certainly imbibed the norms of the AI safety field, and one of them is that there is a very low standard of evidence needed to claim big things, and that’s going to conflict with corporate/legal standards of evidence.
I agree with this weakly, in the sense that being high up in EA is at least a slight update towards them actually thinking through things and being able to make actual cases. My disagreement here is that this effect is probably not strong enough to wash away the cultural effects of operating in a cause area where they don’t need to meet any standard of evidence except long-winded blog posts and getting rewarded, for many reasons.
Also, the board second-guessed it’s decision, which would be evidence for the theory that they couldn’t make a case that actually abided to the standard of evidence for a corporate/legal setting.
If it was any other cause like say GiveWell or some other causes in EA, I would trust them much more that they do have good reason. But AI safety has been so reliant on very low-non-existent standards of evidence or epistemics that they probably couldn’t explain themselves in a way that would abide by the strictness of a corporate/legal standard of evidence.
Edit: The firing wasn’t because of safety related concerns.
Why did you unendorse?
I unendorsed primarily because apparently, the board didn’t fire because of safety concerns, though I’m not sure this is accurate.
I am unsure how I feel about takes like this. On one hand, I want EAs and the EA community to be a supportive bunch. So, expressing how you are feeling and receiving productive/helpful/etc. comments is great. The SBF fiasco was mentally strenuous for many, so it is understandable why anything seemingly negative for EA elicits some of the same emotions, especially if you deeply care about this band of people genuinely aiming to do the most good they can.
On the other hand, I think such takes could also contribute to something I would call a “negative memetic spiral.” In this particular case, several speculative projections are expressed together, and despite the qualifying statement at the beginning, I can’t help but feel that several or all of these things will manifest IRL. And when you kind of start believing in such forecasts, you might start saying similar things or expressing similar sentiments. In the worst case, the negative sentiment chain grows rapidly.
It is possible that nothing consequential happens. People’s mood during moments of panic are highly volatile, so five years in, maybe no one even cares about this episode. But in the present, it becomes a thing against the movement/community. (I think a particular individual may have picked up one such comment from the Forum and posted it online to appease to their audience and elevate negative sentiments around EA?).
Taking a step back, gathering more information, and thinking independently, I was able to reason myself out of many of your projections. We are two days in and there is still an acute lack of clarity about what happened. Emmett Shear, the interim CEO of OpenAI, stated that the board’s decision wasn’t over some safety vs. product disagreement. Several safety-aligned people at OpenAI signed the letter demanding that the board should resign, and they seem to be equally disappointed over recent events; this is more evidence that the safety vs. product disagreement likely didn’t lead to Altman’s ousting. There is also somewhat of a shift back to the “center,” at least on Twitter, as there are quite a few reasonable, level-headed takes on what happened and also on EA. I don’t know about the mood in the Bay though, since I don’t live there.
I am unsure if I am expressing my point well, but this is my off-the-cuff take on your off-the-cuff take.
Thanks for your response Akash. I appreciate your thoughts, and I don’t mind that they’re off-the-cuff :)
I agree with some of what you say, and part of what I think is your underlying point , but in some others I’m a bit less clear. I’ve tried to think about two points where I’m not clear, but please do point if I’ve got something egregiously wrong!
1) You seem to be saying that sharing negative thoughts and projections can lead others to do so, and this can then impact other people’s actions in a negative way. It could also be used by anti-EA people against us.[1]
I guess I can kind of see some of this, but I guess I’d view the cure as being worse than the disease sometimes. I think sharing how we’re thinking and feeling is overall a good thing that could help us understand each other more, and I don’t think self-censorship is the right call here. Writing this out I think maybe I disagree with you about whether negative memetic spirals are actually a thing causally instead of descriptively. I think people may be just as likely apriori to have ‘positive memetic spirals’ or ‘regressions to the vibe mean’ or whatever
2) I’m not sure what ‘I was able to reason myself out of many of your projections’ means, apart from ‘hold fire on making projections’. But currently:
We’ve still heard absolutely nothing from the board
Emmett has potentially indicated he will resign if the board doesn’t explain their reasoning or show evidence to back up their decision[2]
This leads me to doubt his earlier claim that the board told him what the firing was about
There still seems to be a concerted effort by some parties to bring Altman back
I don’t see a shift to the centre on Twitter about what happened or about EA. I think the initial flames have cooled a bit, but my intuition is that a shift has happened, even if the reasons for that shift haven’t been fully borne out by the facts
I still think that it’s a very precarious situation that could have some bad downside consequences for EA. Maybe I could have highlighted my uncertainty more though, I wasn’t trying to say exactly what was going to happen.
I think I know the tweet you’re obliquely referring to, but maybe I’m wrong
https://www.bloomberg.com/news/articles/2023-11-21/altman-openai-board-open-talks-to-negotiate-his-possible-return?fromMostRead=true#xj4y7vzkg
‘Hold fire on making projections’ is the correct read, and I agree with everything else you mention in point 2.
About point 1 — I think sharing negative thoughts is absolutely a-ok and important. I take issue with airing bold projections when basic facts of the matter aren’t even clear. I thought you were stating something akin to “xyz are going to happen,” but re-reading your initial post, I believe I misjudged.
Some personal reflections on EAG London:[1]
Congrats to the CEA Events Team for their hard work and for organising such a good event! 👏
The vibe was really positive! Anecdotally I had heard that the last EAG SF was gloom central, but this event felt much more cheery. I’m not entirely sure why, but it might have had something to do with the open venue, the good weather, or there being more places to touch grass in London compared to the Bay.
I left the conference intellectually energised (though physically exhausted). I’m ready to start drafting some more Forum Post ideas that I will vastly overestimate my ability to finish and publish 😌
AI was (unsurprisingly) the talk of the town. But I found that quite a few people,[2] myself included, were actually more optimistic on AI because of the speed of the social response to AI progress and how pro-safety it seems to be, along with low polarisation along partisan lines.
Related to the above, I came away with the impression that AI Governance may be as if not more important than Technical Alignment in the next 6-12 months. The window for signficiant political opportunity is open now but may not stay open forever, so the AI Governance Space is probably where the most impactful opportunties might be at the moment.
My main negative takeaway was that there seemed to be so little reflection on the difficult last ~6 months for the EA movement. There was 1 session on the FTX, but none at all on the other problems we’ve faced as a community such as Sexual Abuse and Harassment, Trust in EA Leadership, Community Epistemic Health, and whether EA Institutions ought to be reformed. In the opening talk, the only reference was that the ship of EA feels like it’s been ‘going through a storm’, and the ideas presented weren’t really accompanied by a route to embed them in the movement. To me it felt like another missed opportunity after Toby Ord’s speech, and I don’t feel like we as a community have fully grasped or reckoned with the consequences of the last 6 months. I think this was also a common sentiment among people I talked to[3] regardless of whether they agreed with me on proposed solutions.
Shrimp Welfare is now post-meme and part of the orthodoxy, and the Shrimp Welfare Project is unironically an S(hrimp)-Tier Charity. 🦐🥰
EA Conference Food continues to be good to me. I don’t know why it seems to get consistently low ratings in the feedback surveys 🤷 I’d happily tile the universe with those red velvet desserts.
Feel free to add your own thoughts and responses, I’d love to hear them :)
{warning! sampling bias alert!}
{warning! sampling bias alert!}
I assume any event in SF gets a higher proportion of AI doomers than one in London.
[edit: a day after posting, I think this perhaps reads more combative that I intended? It was meant to be more ‘crisis of faith, looking for reassurance if it exists’ than ‘dunk on those crazy longtermists’. I’ll leave the quick take as-is, but maybe clarification of my intentions might be useful to others]
Warning! Hot Take! 🔥🔥🔥 (Also v rambly and not rigorous)
A creeping thought has entered my head recently that I haven’t been able to get rid of…
The EA move toward AI Safety and Longtermism is often based on EV calculations that show the long term future is overwhelmingly valuable, and thus is the intervention that is most cost-effective.
However, more in-depth looks at the EV of x-risk prevention (1, 2) cast significant doubt on those EV calculations, which might make longtermist interventions much less cost-effective than the most effective “neartermist” ones.
But my doubts get worse...
GiveWell estimates around $5k to save a life. So I went looking for some longtermist calculations, and I really couldn’t fund any robust ones![1] Can anyone point me in some robust calculations for longtermist funds/organisations where they go ‘yep, under our assumptions and data, our interventions are at least competitive with top Global Health charities’?
Because it seems to me like that hasn’t been done. But if we’re being EA, people with high and intractable p(doom) from AI shouldn’t work on AI, they should probably EtG for Global Health instead (if we’re going to be full maximise EV about everything). Like, if we’re taking EA seriously, shouldn’t MIRI shut down all AI operations and become a Global Health org? Wouldn’t that be a strong +EV move given their pessimistic assessments of reducing xRisk and their knowledge of +EV global health interventions?
But it gets worse...
Suppose that we go, ‘ok, let’s take EV estimates seriously but not literally’. In which case fine, but that undermines the whole ‘longtermist interventions overwhelmingly dominate EV’ move that many people feel has happened to EA causes over recent years. I don’t think that move grounds longtermism, instead I think it grounds a more pluralistic approach to all of EA that values a large variety of approaches to doing good, but acknowledges the uncertainty and limits on each, and even actively prevents the movement from being focused on any one area. That still seems incompatible with the monetary and social rise of longtermism and AI Safety.
tl;dr: Have any longtermist orgs done robust and defensible EV caculations for their interventions? If not, isn’t that… overwhelmingly sus?
I found some posts arguing that under set of assumptions X then longtermist interventions would be more effective, often by reducing the overall xrisk rate by 1% or something like that. But very little if no work had been done to actually show that was a likely result of the interventions
Why do you think this?
For some very rough maths (appologies in advance for any errors), even Thorstad’s paper (with a 2 century long time of perils, a 0.1% post-peril risk rate, no economic/population growth, no moral progress, people live for 100 years) suggests that reducing p(doom) by 2% is worth as much as saving 16x8billion lives—i.e. each microdoom is worth 6.4million lives. I think we can buy microdooms more cheaply than $5,000*6.4million = $32billion each.
(I can’t actually find those calculations in Thorstad’s paper, could you point them out to me? afaik he mostly looks at the value of fractional reduction in x-risk, while microdooms are an absolute reduction if I understand correctly? happy to be corrected or shown in the right direction!)
My concerns here are twofold:
1 - epistemological: Let’s say those numbers are correct from the Thorstad paper, that a microdoom has to cost ⇐ $32bn to be GiveWell cost-effective. The question is, how would we know this. In his recent post Paul Cristiano thinks that RSPs could lead to a ’10x reduction’ in AI risk. How does he know this? Is this just a risk reduction this century? This decade? Is it a permanent reduction?
It’s one thing to argue that under set of conditions X work on x-risk reduction is cost-effective as you’ve done here. But I’m more interested in the question of whether conditions X hold, because that’s where the rubber hits the road. If those conditions don’t hold, then that’s why longtermism might not ground x-risk work.[1]
There’s also the question of persistence. I think the Thorstad model either assumes the persistence of x-risk reduction, or the persistence of a low-risk post-peril rate. But how could we possibly know that such considerations hold? It seems to me that cluelessness just obliterates grounding strong longtermist interventions here.
2 - practical: Let’s say that longtermist organisations/funds have actually answered criticisms in 1, and under their own lights have made these calculations. I want to see those calculations! How have they calculated the post-risk peril rate? Where is the support for their views on population growth? Do they assume that we colonise the galaxy by default or not? How do they get feedback that overcomes cluelessness objections?
And critically, I want to see why they’ve argued or assumed these things. afaict I can’t find these calculations.
I’m not being snarky here, I really want to be wrong on this and be shown robust calculations on longtermist work, but I haven’t found any. I think the risk is that longtermist EA looks like any other charity/movement that funds ‘what they kinda they think is good’ rather than what we actually know does good, which was meant to be the whole point of EA in the first place.
It could still be grounded for other reasons though, for example if x-risk reduction is highly correlated with GCR reduction, and the latter is of high value
He assumes 20% risk and a 10% relative risk reduction, which I translate into 2% absolute risk of doom, and then see the table on p12.
Isn’t the move here something like, “If doom soon, then all pre-doom value nets to zero”?
Which tbh I’m not sure is wrong. If I expect doom tomorrow, all efforts today should be to reduce it; one night’s sleep not being bitten by mosquitoes doesn’t matter. Stretching this outward in time, doesn’t change the calculus much for a while, maybe about a lifetime or a few lifetimes or so. And a huge chunk of xrisk is concentrated in this century.
The x-risk models actually support the opposite conclusion though. They are generally focused on the balance of two values, v and r—where v the value of a time period and r is the risk of extinction in that period.
If r is sufficiently high, then it operates as a de facto discount rate on the future, which means that the most effective way to increase good is to increase the present v rather than reduce r. For an analogy, if a patient has an incredibly high risk of succumbing to terminal cancer, the way to increase wellbeing may be to give them morphine and palliative care rather than perscribe them risky treatments that may or may not work (and might only be temporary)
Now one could argue against this by saying ‘do not get go gentle into that good night, in the face of destruction we should still do our best’. I have sympathy with that view, but it’s not grounded in the general ‘follow the EV’ framework of EA, and it would have consequences beyond supporting longtermism
I think (at least) somebody at Open Philanthropy needs to start thinking about reacting to an increasing move towards portraying it, either sincerely or strategically, as a shadowy cabal-like entity influencing the world in an ‘evil/sinister’ way, similar to how many right-wingers across the world believe that George Soros is contributing to the decline of Western Civilization through his political philanthropy.
Last time there was an explicitly hostile media campaign against EA the reaction was not to do anything, and the result is that Émile P. Torres has a large media presence,[1] launched the term TESCREAL to some success, and EA-critical thoughts became a lot more public and harsh in certain left-ish academic circles. In many think pieces responding to WWOTF or FTX or SBF, they get extensively cited as a primary EA-critic, for example.
I think the ‘ignore it’ strategy was a mistake and I’m afraid the same mistake might happen again, with potentially worse consequences.
Do people realise that they’ve going to release a documentary sometime soon?
You say this as if there were ways to respond which would have prevented this. I’m not sure these exist, and in general I think “ignore it” is a really really solid heuristic in an era where conflict drives clicks.
I think responding in a way that is calm, boring, and factual will help. It’s not going to get Émile to publicly recant anything. The goal is just for people who find Émile’s stuff to see that there’s another side to the story. They aren’t going to publicly say “yo Émile I think there might be another side to the story”. But fewer of them will signal boost their writings on the theory that “EAs have nothing to say in their own defense, therefore they are guilty”. Also, I think people often interpret silence as a contemptuous response, and that can be enraging in itself.
Maybe it would be useful to discuss concrete examples of engagement and think about what’s been helpful/harmful.
Offhand, I would guess that the holiday fundraiser that Émile and Nathan Young ran (for GiveDirectly I think it was?) was positive.
I think this post was probably positive (I read it around a year ago, my recollections are a bit vague). But I guess that post itself could be an argument that even attempting to engage with Émile in good faith is potentially dangerous.
Perhaps the right strategy is something like: assume good faith, except with specific critics who have a known history of bad faith. And consider that your comparative advantage may lie elsewhere, unless others would describe you as unusually good at being charitable.
What makes you think this? I would guess it was pretty negative, by legitimizing Torres, and most of the donations funging heavily against other EA causes.
I would guess any legitimization of Émile by Nathan was symmetrical with a legitimization of Nathan by Émile. However I didn’t get the sense that either was legitimizing the other, so much as both were legitimizing GiveDirectly. It seems valuable to legitimize GiveDirectly, especially among the “left-ish academic circles” reading Émile who might otherwise believe that Émile is against all EA causes/organizations. (And among “left-ish academics” who might otherwise believe that Nathan scorns “near-termist” causes.)
There’s a lot of cause prioritization disagreement within EA, but it doesn’t usually get vicious, in part because EAs have “skin in the game” with regard to using their time & money in order to make the world a better place. One hypothesis is that if we can get Émile’s audience to feel some genuine curiosity about how to make their holiday giving effective, they’ll wonder why some people are longtermists. I think it’s absolutely fine to disagree with longtermism, but I also think that longtermists are generally thoughtful and well-intentioned, and it’s worth understanding why they give to the causes they do.
Do you have specific reasons to believe this? It’s a possibility, but I could just as easily see most donations coming from non-EAs, or EAs who consider GiveDirectly a top pick anyways. Even if EA donors didn’t consider GiveDirectly a top pick on its own, they might have considered “GiveDirectly plus better relations with Émile with no extra cost” to be a top pick, and I feel hesitant to judge this more harshly than I would judge any other EA cause prioritization.
BTW, a mental model here is:
https://markfuentes1.substack.com/p/emile-p-torress-history-of-dishonesty
If Émile is motivated to attack EA because they feel rejected by it, it’s conceivable to me that their motivation for aggression would decrease if a super kind and understanding therapist-type person listened to them really well privately and helped them feel heard & understood. The fundraiser thing makes me think this could work if done well, although the Helen Pluckrose thing from Mark’s post makes me think it’s risky. But if it’s private, especially from a person who’s not particularly well-known, I assume it wouldn’t run the specific risk of legitimization.
[edited to fix pronouns, sorry!!]
FYI, Émile’s pronouns are they/them.
[Edit: I really don’t like that this comment got downvoted and disagree voted...]
I agree it’s a solid heuristic, but heuristics aren’t foolproof and it’s important to be able to realise where they’re not working.
I remembered your tweet about choosing intellectual opponents wisely because I think it be useful to show where we disagree on this:
1 - Choosing opponents is sometimes not up to you. As an analogy, being in a physical fight only takes one party to throw punches. When debates start to have significant consequences socially and politically, it’s worth considering that letting hostile ideas spread unchallenged may work out badly in the future.
2 - I’m not sure it’s clear that “the silent majority can often already see their mistakes” in this case. I don’t think this is a minor view on EA. I think a lot of people are sympathetic to Torres’ point of view, and a signficiant part of that is (in my opinion) because there wasn’t a lot of pushback when they started making these claims in major outlets.
On my first comment, I agree that I don’t think much could have been much done to stop Émile turning against EA,[1] but I absolutely don’t think it was inevitable that they would have had such a wide impact. They made the Bulletin of Atomic Scientists! They’re partnered with Timnit, who has large influence and sympathy in AI space! People who could have been potential allies in a coalition basically think our movement is evil.[2] They get sympathetically cited in academic criticisms of EA.
Was some pushback going to happen? Yes, but I don’t think inevitably at this scale. I do think more could have been done to actually push back on their claims that went over the line in terms of hostility and accuracy, and I think that could have led to a better climate at this critical juncture for AI discussions and policy where we need to build coalitions with communities who don’t fully agree with us.
My concern is that this new wave of criticism and attack on OpenPhil might not simply fade away but could instead cement an anti-EA narrative that could put the movement and the causes we care about at risk. So looking at what happened in the Torres case, and looking at the negative depictions of Open Philanthropy recently, ‘ignore it’ doesn’t look like a good strategy
Though they were part of EA space for a while, so there’s probably some ‘common knowledge’ that some people might have that paints a
I think the whole thread this tweet is a part of is worth reading
I don’t think the hostility between the near-term harm and the AI x-riskers would have been prevented by more attack rebutting Emille Torres.
The real problem is that the near-term AI harm people perceive Ai x-riskers as ignoring their concerns and actively making the near term harms worse.
Unfortunately, I think this sentiment is at least partly accurate. When Timnit got pushed out of google for pointing out near-term harms of AI, there was almost no support from the x-risk crowd (I can’t find any big name EA’s on this list, for example). This probably contributed to her current anti-EA stance.
As for real world harms, well, we can just say that openAI was started by an x-risker, and has kickstarted an AI race, causing a myriad of real world harms such as scams, art plagiarism, data theft, etc.
The way to actually fix this would be actual solidarity and bridge building on dealing with the short-term harms of AI.
I don’t want to fully re-litigate this history, as I’m more concerned about the future of OpenPhilanthropy being blindsided by a political attack (it might be low probability, but you’d think OpenPhil would be open to being concerned about low-chance high-impact threats to it)
Agreed. It predated Emile’s public anti-EA turn for sure. But it was never inevitable. Indeed, supporting Timnit during her firing from Google may have been a super low-cost way to show solidarity. It might have meant that Emile and Timnit wouldn’t have become allies who have strong ideological influence over a large part of AI research space.
I’d like to think so too, but this is a bridge that needs to build from both ends imo, as I wouldn’t recommend a unilateral action unless I really trusted the other parties involved.
There seems to have been some momentum towards more collaboration after the AI Safety Summit though. I hope the Bletchley Declaration can be an inflection point for more of this.
What do you see as the risk of building a bridge if it’s not reciprocated?
‘The way to actually fix this would be actual solidarity and bridge building on dealing with the short-term harms of AI.’
What would this look like? I feel like, if all you do is say nice things, that is a good idea usually, but it won’t move the dial that much (and also is potentially lying, depending on context and your own opinions; we can’t just assume all concerns about short-term harm, let alone proposed solutions, are well thought out). But if you’re advocating spending actual EA money and labour on this, surely you’d first need to make a case that stuff “dealing with the short term harms of AI” is not just good (plausible), but also better than spending the money on other EA stuff. I feel like a hidden crux here might be that you, personally, don’t believe in AI X-risk*, so you think it’s an improvement if AI-related money is spent on short term stuff, whether or not that is better than spending it on animal welfare or global health and development, or for that matter anti-racist/feminist/socialist stuff not to do with AI. But obviously, people who do buy that AI X-risk is comparable/better as a cause area than standard near-term EA stuff or biorisk, can’t take that line.
*I am also fairly skeptical it is a good use of EA money and effort for what it’s worth, though I’ve ended up working on it anyway.
This seems a little zero-sum, which is not how successful social movements tend to operate. I’ll freely confess that I am on the “near term risk” team, but that doesn’t mean the two groups can’t work together.
A simplified example: Say 30% of a council are concerned abut near term harms, and 30% are concerned about x-risk, and each wants policies passed that address their own concerns. If the two spend all their time shitting on each other for having misplaced priorities, neither of them will get what they want. But if they work together, they have a majority, and can pass a combined bill that addresses both near-term harm and AI x-risk, benefiting both.
Unfortunately, the best time to do this bridge building and alliance making was several years ago, and the distrust is already rather entrenched. But I genuinely think that working to mend those bridges will make both groups better off in the long run.
You haven’t actually addressed the main question of the previous comment: What would this bridge building look like? Your council example does not match the current reality very well.
It feels like you also sidestep other stuff in the comment, and it is unclear what your position is. Should we spend EA money (or other resources) on “short-term harms”? If yes, is the main reason because funding the marginal AI ethics research is better than the marginal bed-net and the marginal AI xrisk research? Or would the main reason for spending money on “short-term harms” that we buy sympathy with the group of people concerned about “short-term harms”, so we can later pass regulations together with them to reduce both “short-term harm” and AI x-risk.
https://forum.effectivealtruism.org/posts/Q4rg6vwbtPxXW6ECj/we-are-fighting-a-shared-battle-a-call-for-a-different (It’s been a while since I read this so I’m not sure it is what you are looking for, but Gideon Futerman had some ideas for what “bridge building” might look like.)
I just read most of the article. It was not that satisfying in this context. Most of it is arguments that we should work together (which I dont disagree with). And I imagine it will be quite hard to convince most AI xrisk people of “whether AI is closer to a stupid ‘stochastic parrot’ or on the ‘verge-of-superintelligence’ doesn’t really matter; ”. If we were to adopt Gideon’s desired framing, it looks like we would need to make sacrifices in epistemics. Related:
Some of Gideon’s suggestion such as protest or compute governance are already being pursued. Not sure if that counts as bridge building though, because these might be good ideas anyways.
.
For the record, I’m very willing to be corrected/amend my Quick Take (and my beliefs on this is in general) if “ignore it” isn’t an accurate summary of what was done. Perhaps there was internal action taken with academic spaces/EA organisations that I’m not aware of? I still think the net effect of EA actions in any case was closer to “ignore it”, but the literal strong claim may be incorrect.
Edit: I actually think these considertions should go for many of the comments in this sub-thread, not just my own. There’s a lot to disagree about, but I don’t think any comment in this chain is worthy of downvotes? (Especially strong ones)
A meta-thought on this take given the discussion its generated.
Currently this is at net 10 upvotes from 20 total votes at time of writing—but is ahead 8 to 6 on agree/disagree votes. Based on Forum voting norms, I don’t think this is particularly deserving of downvotes given the suggested criteria? Especially strong ones? Disagreevotes—go ahead, be my guest! Comments pointing out where I’ve gone wrong—I actively encourage you to do so!
I put this in a Quick Take, not a top-level post so it’s not as visible on the Forum front page. (and the point of a QT is for ’exploratory, draft-stage, rough thoughts like this).I led off with saying “I think”—I’m just voicing my concerns about the atmosphere surrounding OpenPhil and its perception. It’s written in good faith, albeit with a concerned tone. I don’t think it violates what the EA Forum should be about.[1]
I know these kind of comments are annoying but still, I wanted to point out that this vote distribution feels a bit unfair, or at least unexplained to me. Sure, silent downvoting is a signal, but it’s a crude and noisy signal and I don’t really have much to update off here.
If you downvoted but don’t want to get involved in a public discussion about it, feel free to send me a DM with feedback instead. We don’t have to get into a discussion about the merits (if you don’t want to!), I’m just confused at the vote distribution.
Again, especially in Quick Takes
The harsh crticism of EA has only been a good thing, forcing us to have higher standards and rigour. We don’t want an echochamber.
I would see it as a thoroughly good thing if Open Philanthropy were to combat the protrayal of itself as a shadowy cabal (like in the recent politico piece) for example by:
Having more democratic buy-in with the public
e.g. Having a bigger public presence in media, relying on a more diverse pool of funding than (i.e. less billionarie funding)
Engaged in less political lobbying
More transparent about the network of organisations around them
e.g. from the Politico article: ”… said Open Philanthropy’s use of Horizon … suggests an attempt to mask the program’s ties to Open Philanthropy, the effective altruism movement or leading AI firms”
I am not convinced that “having a bigger public presence in media” is a reliable way to get democratic buy-in. (There is also some “damned if you, damned if you don’t” dynamic going on here—if OP was constantly engaging in media interactions, they’d probably be accused of “unduly influencing the discourse/the media landscape”) Could you describe what a more democratic OP would look like?
You mention “less billionaire funding”—OP was built on the idea of giving away Dustin’s and Cari’s money in the most effective way. OP is not fundraising, it is grantmaking! So how could it, as you put it, “rely on a more diverse pool of funding”? (also: https://forum.effectivealtruism.org/posts/zuqpqqFoue5LyutTv/the-ea-community-does-not-own-its-donors-money) I also suspect we would see the same dynamic as above: If OP did actively try to secure additional money in the forms of government grants, they’d be maligned for absorbing public resources in spite of their own wealth.
I think a blanket condemnation of political lobbying or the suggestion to “do less” of it is not helpful. Advocating for better policies (in animal welfare, GHD, pandemic preparedness etc.) is in my view one of the most impactful things you can do. I fear we are throwing the baby out with the bathwater here.
Suing people nearly always makes you look like the assholes I think.
As for Torres, it is fine for people to push back against specific false things they say. But fundamentally, even once you get past the misrepresentations, there is a bunch of stuff that they highlight that various prominent EAs really do believe and say that genuinely does seem outrageous or scary to most people, and no amount of pushback is likely to persuade most of those people otherwise.
In some cases, I think that outrage fairly clearly isn’t really justified once you think things through very carefully: i.e. for example the quote from Nick Beckstead about saving lives being all-things-equal higher value in rich countries, because of flow-through effects which Torres always says makes Beckstead a white supremacist. But in other cases well, it’s hardly news that utilitarianism has a bunch of implications that strongly contradict moral commonsense, or that EAs are sympathetic to utilitarianism. And ‘oh, but I don’t endorse [outrageous sounding view], I merely think there is like a 60% chance it is true, and you should be careful about moral uncertainty’ does not sound very reassuring to a normal outside person.
For example, take Will on double-or-nothing gambles (https://conversationswithtyler.com/episodes/william-macaskill/) where you do something that has a 49% chance of destroying everyone, and a 51% chance of doubling the number of humans in existence (now and in the future). It’s a little hard to make out exactly what Will’s overall position on this, but he does say it is hard to justify not taking those gambles:
‘Then, in this case, it’s not an example of very low probabilities, very large amounts of value. Then your view would have to argue that, “Well, the future, as it is, is like close to the upper bound of value,” in order to make sense of the idea that you shouldn’t flip 50⁄50. I think, actually, that position would be pretty hard to defend, is my guess. My thought is that, probably, within a situation where any view you say ends up having pretty bad, implausible consequences’
And he does seem to say there are some gambles of this kind he might take:
’Also, just briefly on the 51/49: Because of the pluralism that I talked about — although, again, it’s meta pluralism — of putting weight on many different model views, I would at least need the probabilities to be quite a bit wider in order to take the gamble...’
Or to give another example, the Bostrom and Shulman paper on digital minds talks about how if digital minds really have better lives than us, than classical (total) utilitarianism says they should take all our resources and let us starve. Bostrom and Shulman are against that in the paper. But I think it is fair to say they take utilitarianism seriously as a moral theory. And lots of people are going to think taking seriously the idea that this could be right is already corrupt, and vaguely Hitler-ish/reminiscent of white settler expansionism against Native Americans.
In my view, EAs should be more clearly committed to rejecting (total*) utilitarianism in these sorts of cases than they actually are. Though I understand that moral philosophers correctly think the arguments for utilitarianism, or views which have similar implications to utilitarianism in these contexts, are disturbingly strong.
*In both of the cases described, person-affecting versions of classical utilitarianism which deny creating happy people is good doesn’t have the scary consequences.
First, I want to thank you for engaging David. I get the sense we’ve disagreed a lot on some recent topics on the Forum, so I do want to say I appreciate you explaining your point of view to me on them, even if I do struggle to understand. Your comment above covers a lot of ground, so if you want to switch to a higher-bandwidth way of discussing them, I would be happy to. I apologise in advance if my reply below comes across as overly hostile or in bad-faith—it’s not my intention, but I do admit I’ve somewhat lost my cool on this topic of late. But in my defence, sometimes that’s the appropriate response. As I tried to summarise in my earlier comment, continuing to co-operate when the other player is defecting is a bad approach.
As for your comment/reply though, I’m not entirely sure what to make of it. To try to clarify, I was trying to understand why the Twitter discourse between people focused on AI xRisk and the FAact Community[1] has been so toxic over the last week, almost entirely (as far as I can see) from the latter to the former. Instead, I feel like you’ve steered the conversation away to a discussion about the implications of naïve utilitariansim. I also feel we may disagree on how much Torres has legitimate criticisms and how of their work is simply wilful ‘misrepresentation’ (I wonder if you’ve changed your mind on Torres since last year?). There are definitely connections there, but I don’t think it’s quite the same conversation, and I think it somewhat telling that you responded to suggestions 3 & 4, and not 1 & 2, which I think are far less controversial (fwiw I agree that legal action should only be used once all other courses of actions have failed).
To clarify what I’m trying to get at here with some more examples, which I hope will be reasonably unobjectionable even if incorrect:
Yesterday Timnit again insinuated that William MacAskill was a eugenicist. You can read that tweet and I don’t think she means this in a ‘belongs to a historical tradition’ way I think she means it in a ‘this is what he believes’ way. I haven’t seen anyone from the FAact Community call this out. In fact, Margaret Mitchell’s response to Jess Whittlestone’s attempt to offer an olive branch was met with confusion that there’s any extreme behaviour amongst the AI Ethics field.
People working in AI Safety and/or associated with EA should therefore expect to be called eugencists, and the more Timnit’s perspective gains prominence that more they will have to deal with the consequences of this.
Noah Giansiracua’s thread that I linked in the last tweet is highly conspiratiorial, spreads reckless misinformation, and is often just wrong. But not only has he doubled down despite pushback,[2] but he today tried to bridge the Safety/Ethics divide today seemingly unware that trashing the other side in a 26 tweet screed is massively damaging to this goal.
This suggests that while AI Safety efforts to build bridges may have some success, there may a strong and connected group of scholars who will either not countenance it at all, or be happy to stick the knife in once the opportunity appears. If I were an AI Safety academic, I wouldn’t trust Noah.
In general, my hope is that work is going on behind the scenes and off Twitter to build bridges between the two camps. But a lot of names on the FAact side that seem to be more toxic are quite prominent, and given the culture of silence/bullying involved there (again, see the Rumman Chowdhury tweet in the original comment, with further evidence here) I’m not sure I feel as hopeful it will happen as I did in recent weeks.
The more I look into it, the more I see the hostility as asymmetric. I’d be very open to counter-evidence on this point, but I don’t see AI Safety people treating the other camp with such naked hostility, and definitely not from the more influential members of the movement, as far as I can tell. (And almost certainly not any more than usual over the past week or so? As I said, a lot of this seems to have kicked off post CAIS Letter).
My call to not ‘be passive’ was one in which I expect hostility to the field of AI Safety to continue, perhaps grow, and be amplified by influential figures in the AI space. I maintian the general EA media strategy of ignoring critics, and if engaging them only doing so with the utmost politeness, has been a net negative strategy, and will continue to be so if continued—with perhaps very bad consequences.
Anyway, I’d like to thank you for sharing your perspective, and I do hope my perceptions have been skewed to be too pessimistic. To others reading, I’d really appreciate hearing your thoughts on these topics, and points of view or explanations that might change my mind
I think this is better than the Safety/Ethics labelling, but I’m referring to the same divide here
Long may EA Twitter dunk on him until a retraction appears
I mean in a sense a venue that hosts torres is definitionally trashy due to https://markfuentes1.substack.com/p/emile-p-torress-history-of-dishonesty except insofar as they haven’t seen or don’t believe this Fuentes person.
I guess I thought my points about total utilitarianism were relevant, because ‘we can make people like us more by pushing back more against misrepresentation’ is only true insofar as the real views we have will not offend people. I’m also just generically anxious about people in EA believing things that feel scary to me. (As I say, I’m not actually against people correcting misrepresentations obviously.)
I don’t really have much sense of how reasonable critics are or aren’t being, beyond the claim that sometimes they touch on genuinely scary things about total utilitarianism, and that it’s a bit of a problem that the main group arguing for AI safety contains a lot of prominent people with views that (theoretically) imply that we should be prepared to take big chances of AI catastrophe rather than pass up small chances of lots of v. happy digital people.
On Torres specifically: I don’t really follow them in detail (these topics make me anxious), but I didn’t intend to be claiming that they are a fair or measured critic, just that they have decent technical understanding of the philosophical issues involved and sometimes puts their finger on real weaknesses. That is compatible with them also saying a lot of stuff that’s just false. I think motivated reasoning is a more likely explanation for why they says false things than conscious lying, but that’s just because that’s my prior about most people. (Edit: Actually, I’m a little less sure of that, after being reminded of the sockpuppetry allegations by quinn below. If those are true, that is deliberate dishonesty.)
Regarding Gebru calling Will a eugenicist. Well, I really doubt you could “sue” over that, or demonstrate to the people most concerned about this that he doesn’t count as one by any reasonable definition. Some people use “eugenicist” for any preference that a non-disabled person comes into existence rather than a different disabled person. And Will does have that preference. In What We Owe the Future, he takes it as obvious that if you have a medical condition that means if you conceive right now, your child will have awful painful migraines, then you should wait a few weeks to conceive so that you have a different child who doesn’t have migraines. I think plenty ordinary people would be fine with that and puzzled by Gebru-like reactions, but it probably does meet some literal definitions that have been given for “eugenics”. Just suggesting he is a “eugenicist” without further clarification is nonetheless misleading and unfair in my view, but that’s not quite what libel is. Certainly I have met philosophers with strong disability rights views who regard Will’s kind of reaction to the migraine case as bigoted. (Not endorsing that view myself.)
None of this is some kind of overall endorsement of how the ‘AI ethics’ crowd on Twitter talk overall, or about EAs specifically. I haven’t been much exposed to it, and when I have been, I generally haven’t liked it.
Does anyone work at, or know somebody who works at Cohere?
Last November their CEO Aidan Gomez published an anti-effective-altruism internal letter/memo (Bloomberg reporting here, apparently confirmed as true though no further comment)
I got the vibe from Twitter/X that Aidan didn’t like EA, but making an internal statement about it to your company seems really odd to me? Like why do your engineers and project managers need to know about your anti-EA opinions to build their products? Maybe it came after the AI Safety Summit?
Does anyone in the AI Safety Space of EA have any knowledge or context to help out here?
Ideally I’d like to read the memo and see what it’s definition of EA is and what its arguments are.
Given Cohere are a leading, though not frontier-level, AI company feels like it’s bad to have a fairly prominent CEO who hates EA so much he releases internal memos about it to the company.
I don’t think it’s odd at all. As the Bloomberg article notes, this was in response to the martyrdom of Saint Altman, when everyone thought the evil Effective Altruists were stealing/smashing OA for no reason and destroying humanity’s future (as well as the fortune of many of the individuals involved, to a degree few of them bothered to disclose) and/or turning it over to the Chinese commies. An internal memo decrying ‘Effective Altruism’ was far from the most extreme response at the time; but I doubt Gomez would write it today, if only because so much news has come out since then and it no longer looks like such a simple morality play. (For example, a lot of SV people were shocked to learn he had been fired from YC by Paul Graham et al for the same reason. That was a very closely held secret.)
Ah good point that it was in the aftermath of the OpenAI board weekend, but it still seems like a very extreme/odd reaction to me (though I have to note the benefit of hindsight as well as my own personal biases). I still think it’d be interesting to see what Aidan actually said, and/or why he’s formed such a negative opinion of EA, but I think your right than the simplest explanation here is:
I’d be a bit surprised if you could find people on this forum who (still) work at Cohere. Hard to see a stronger signal to interview elsewhere than your CEO explaining in a public memo why they hate you.
I agree it’s odd in the sense that most companies don’t do it. I see it as a attempt to enforce a certain kind of culture (promoting conformity, discouragement of dissent, “just build now” at the expense of ethics, etc) that I don’t much care for. But the CEO also made it abundantly clear he doesn’t like people who think like me either, so ¯\_(ツ)_/¯.
I’ve generally been quite optimistic that the increased awareness AI xRisk has got recently can lead to some actual progress in reducing the risks and harms from AI. However, I’ve become increasingly sad at the ongoing rivalry between the AI ‘Safety’ and ‘Ethics’ camps[1] 😔 Since the CAIS Letter was released, there seems to have been an increasing level of hostility on Twitter between the two camps, though my impression is that the holistility is mainly one-directional.[2]
I dearly hope that a coalition of some form can be built here, even if it is an uneasy one, but I fear that it might not be possible. It unfortunately seems like a textbook case of mistake vs conflict theory approaches at work? I’d love someone to change my mind, and say that Twitter amplifies the loudest voices,[3] and that in the background people are making attempts to build bridges. But I fear that instead that the centre cannot hold, and that there will be not just simmering resentment but open hostility between the two camps.
If that happens, then I don’t think those involved in AI Safety work can afford to remain passive in response to sustained attack. I think that this has already damaged the prospects of the movement,[4] and future consequences could be even worse. If the other player in your game is constantly defecting, it’s probably time to start defecting back.
Can someone please persuade me that my pessimism is unfounded?
FWIW I don’t like these terms, but people seem to intuitively grok what is meant by them
I’m open to be corrected here, but I feel like those sceptical of the AI xRisk/AI Safety communities have upped the ante in terms of the amount of criticism and its vitriol—though I am open to the explanation that I’ve been looking out for it more
It also seems very bad that the two camps do most of their talking to each other (if they do at all) via Twitter, that seems clearly suboptimal!!
The EA community’s silence regarding Torres has led to the acronym ‘TESCREAL’ gaining increasing prominence amongst academic circles—and it is not a neutral one, and gives them more prominence and a larger platform.
What does not “remaining passive” involve?
I can’t say I have a strategy David. I’ve just been quite upset and riled up by the discourse over the last week just as I had gained some optimism :( I’m afraid that by trying to turn the other cheek to hostility, those working to mitigate AI xRisk end up ceding the court of public opinion to those hostile to it.
I think some suggestions would be:
Standing up to, and callying out, bullying in these discussions can cause a preference cascade of pushback to it—see here—but someone needs to stand up for people to realise that dominant voices are not representative of a field, and silence may obscure areas for collaboration and mutual coalitions to form.
Being aware of what critiques of EA/AI xRisk get traction in adjacent communities. Some of it might be malicious, but a lot of it seems to be a default attitude of scepticism merged with misunderstandings. While not everyone would change their mind, I think people reaching ‘across the aisle’ might correct the record in many people’s minds. Even if not for the person making the claims, perhaps for those watching and reading online.
Publicly pushing back on Torres. I don’t know what went down when they were more involved in the EA movement that caused their opinion to flip 180 degrees, but I think the main ‘strategy’ has been to ignore their work and not respond to their criticism. The result: their ideas gaining prominence in the AI Ethics field, publications in notable outlets, despite acting consistently in bad faith. To their credit, they are voraciously productive in their output and I don’t expect to it slow down. Continuing with a failed strategy doesn’t sound like the right call here.
In cases of the most severe hostility, potential considering legal or institutional action? In this example, can you really just get away with calling someone a eugenicist when it’s so obviously false? But there have been cases where people have successfully sued for defamation for statements made on Twitter. That’s an extreme option though, but not worth ignoring entirely.
I would recommend trying to figure out how much loud people matter. Like it’s unclear if anyone is both susceptible to sneer/dunk culture and potentially useful someday. Kindness and rigor come with pretty massive selection effects, i.e., people who want the world to be better and are willing to apply scrutiny to their goals will pretty naturally discredit hostile pundits and just as naturally get funneled toward more sophisticated framings or literatures.
I don’t claim this attitude would work for all the scicomm and public opinion strategy sectors of the movement or classes of levers, but it works well to help me stay busy and focused and epistemically virtuous.
I wrote some notes about a way forward last february, I just CC’d them to shortform so I could share with you https://forum.effectivealtruism.org/posts/r5GbSZ7dcb6nbuWch/quinn-s-shortform?commentId=nskr6XbPghTfTQoag
related comment I made: https://forum.effectivealtruism.org/posts/nsLTKCd3Bvdwzj9x8/ingroup-deference?commentId=zZNNTk5YNYZRykbTu
Oh hi. Just rubber-ducking a failure mode some of my Forum takes[1] seem to fall into, but please add your takes if you think that would help :)
----------------------------------------------------------------------------
Some of my posts/comments can be quite long—I like responding with as much context as possible on the Forum, but as some of the original content itself is quite long, that means my responses can be quite long! I don’t think that’s necessarily a problem in itself, but the problem then comes with receiving disagree votes without comments elaborating them.
<I want to say, this isn’t complaining about disagreement. I like disagreement[2], it means I get to test my ideas and arguments>
However, it does pose an issue with updating my thoughts. A long post that has positive upvotes, negative disagree votes, and no (or few) comments means it’s hard for me to know where my opinion differs from other EA Forum users, and how far and in what direction I ought to update in. The best examples from my own history:
My takeaways from Cremer’s Vox piece. Do the disagree votes apply to Cremer’s piece, my (relatively) positive response to it, or the specific points I highlighted, or all of the above?
My fears about karma-downweighting Community posts. This one went in a few different directions—it has about 2.5 distinct but related points going on, so I don’t know where the disagreement is focused on.[3]
----------------------------------------------------------------------------
Potential Solutions?:
Post shorter takes: This means the agree/disagree signal will be more clearly linked to my context, but it means I won’t be able to add all the content that I do now, which I think adds value.
Post fewer takes: Ceteris paribus this might not be expected work, but the argument would be that with more time between fewer contributions, the quality of them would go up so they’d usually make one clear point for readers to agree/disagree with
Post more takes: Splitting a large comment into lots of sub-comments would do the same thing as ‘post shorter takes’, and keep context. The cost is a karma-farming effect, and splitting a thread into multiple parts, making it harder to follow.
Explicitly ask for comments: I feel like this wouldn’t work? It feels a bit whiny and has no enforcement mechanism?
Pre-commit to up-voting responses: Relies on others to trust me, might not be credible for those who strongly disagree with me.
Suggestions/thoughts on any of the above welcome
I’m using ‘takes’ to refer to both posts and comments, it’s not meant to imply low-quality contributions to the forum
Modulo good faith norms etc
Also, the agree-vote karma on this piece went all over the place, and comments might have been a place to hash this disagreement out
status: very rambly. This is an idea I want to explore in an upcoming post about longtermism, would be grateful to anyone’s thoughts. For more detailed context, see https://plato.stanford.edu/entries/time/ for debate on the nature of time in philosophy
Does rejecting longtermism require rejecting the B-Theory of Time (i.e. eternalism, the view that the past, present, and future have the same ontological status)? Saying that future people don’t exist (and therefore can’t be harmed, can’t lose out by not existing, don’t have the same moral rights as present ‘existing’ people) also implies that the future they live in (i.e. the city, country, planet, solar system, galaxy etc) doesn’t exist. This seems to fly in the face of our best understandings of physics (i.e. Relativity and Quantum Physics don’t have a place for a special ‘present’ afaik, in fact Relativity suggests that there is no such thing as a universal present)[1] I’m not sure people grapple with that contradiction.
Furthermore, if you want to claim the special ‘present’ moment does exist, and that our moral obligations only hold in that frame of time, doesn’t that mean that our obligations to the past don’t matter? Many have criticised Bostrom’s use of ‘mere ripples’ to describe historical atrocities,[2] but doesn’t believing in a unique present moment imply that the past doesn’t exist, and hence those atrocities don’t exist either and that we have no obligations that refer to past events or people? One could get around this by assuming a ‘growing block’ theory of time, where the past exists, as does the present, but the future doesn’t. But if you believe in a special, unique present then asserting the past exists but the future doesn’t seems no more valid than the other way around?
I actually think that has legs as more than a niche philosophical point, but would be very interested in hearing others’ thoughts.
Physicists please correct me if I’ve made egregious mistakes of interpretation here
And I’m not defending that language here
FWIW, someone could reject longtermism for reasons other than specific person-affecting views or even pure time preferences.
Even without a universal present, there’s still your present (and past), and you can do ethics relative to that. Maybe this doesn’t seem impartial enough, and could lead to agents with the same otherwise impartial ethical views and same descriptive views disagreeing about what to do, and those are undesirable?
OTOH, causality is still directed and we can still partially order events that way or via agreement across reference frames. The descendants of humanity, say, 1000 years from your present (well, humanity here in our part of the multiverse, say) are still all after your actions now, probably no matter what (physically valid) reference frame you consider, maybe barring time travel. This is because humans’ reference frames are all very similar to one another, as differences in velocity, acceleration, force and gravity are generally very small.
So, one approach could be to rank or estimate the value of your available options on each reference frame and weigh across them, or look for agreement or look for Pareto improvements. Right now for you, the different reference frames should agree, but they could come apart for you or other agents in the future if/when we or our descendants start colonizing space, traveling at substantial fractions of the speed of light.
Also, people who won’t come to exist don’t exist under the B-theory, so they can’t experience harm. Maybe they’re harmed by not existing, but they won’t be around to experience that. Future people could have interests, but if we only recognize interests for people that actually exist under the B-theory, then extinction wouldn’t be bad for those who never come to exist as a result, because they don’t exist under the B-theory.
Great response Michael! Made me realise I’m conflating a lot of things (so bear with me)
By longtermism, I really mean MacAskill’s:
future people matter
there could be a lot of them
our actions now could affect their future (in some morally significant way)
And in that sense, I really think only bullet 1 is the moral claim. The other two are empirical, about what our forecasts are, and what our morally relevant actions are. I get the sense that those who reject longtermism want to reject it on moral grounds, not empirical ones, so they must reject bullet-point 1. The main ways of doing so are, as you mention, person-affecting views or a pure-rate of time preference, both of which I am sceptical are without their difficult bullets to bite.[1]
The argument I want to propose is this:
The moral theories we regard as correct ought to cohere with our current best understandings of physics[2]
The Special Theory of Relativity (STR) is part of our current best understanding of physics
STR implies that there is no universal present moment (i.e. without an observer’s explicit frame of reference)
Some set of person-affecting views [P] assume that there is a universal present moment (i.e. that we can clearly separate some people as not in the present and therefore not worthy of moral concern, and others which do have this property)
From 3 & 4, STR and P are contradictory
From 1 & 5, we ought to reject all person-affecting views that are in P
And I would argue that a common naïve negative reactions to longtermism (along the liens of) “potential future people don’t matter, and it is evil to do anything for them at the expense of present people since they don’t exist” is in P, and therefore ought to be rejected. In fact, the only ways to really get out of this seem to be either that someone’s chosen person-affecting views are not in P, or that 1 is false. The former is open to question of course, and the latter seems highly suspect.
The point about certain people never existing under B-Theory is an interesting counter. Is it possible to have B-Theory without strong determinism (the kind that undermines moral responsibility)? I guess I’d reframe it as under B-Theory, future people exist, and so whoever can affect their wellbeing causally (whether from their reference frame of past, present, or future) ought to do so if they can. You could still believe that is empirically very difficult to do so, but it makes sense to me that outside some special sphere of moral obligation (say my family, children, or those who I have made promises too), the only difference between someone far away in time and close is the extent to which I can causally help them, and not their position in time—just the same way we might say that the reason to be sceptical of charity in different locations in the present is their effectiveness at aid and not their physical location in space.
That causal effectiveness (corresponding to opening bullet point 3) seems to me the most promising response to longtermism, not that those future people exist or not, or have moral value (bullet point 1), but it seems that’s what most objections lean on.
I don’t think having to bite bullets is a knockdown refutation, but if your view has them I want to see evidence you’ve included them in your meal, especially if criticise others for not biting their own bullets
Interestingly, the corollary of this is that while one cannot derive an ought from an is, one can derive an is from an ought!
Has anyone else listened to the latest episode of Clearer Thinking ? Spencer interviews Richard Lang about Douglas Harding’s “Headless Way”, and if you squint enough it’s related to the classic philosophical problems of consciousness, but it did remind me a bit of Scott A’s classic story “Universal Love, Said The Cactus Person” which made me laugh. (N.B. Spencer is a lot more gracious and inquisitive than the protagonist!)
But yeah if you find the conversation interesting and/or like practising mindfulness meditation, Richard has a series of guided meditations on the Waking Up App, so go and check those out.
In this comment I was going to quote the following from R. M. Hare:
I remember this being quoted in Mackie’s Ethics during my undergraduate degree, and it’s always stuck with me as a powerful argument against moral non-naturalism and a close approximation of my thoughts on moral philosophy and meta-ethics.
But after some Google-Fu I couldn’t actually track down the original quote. Most people think it comes from the Essay Nothing Matters in Hare’s Applications of Moral Philosophy. While this definitely seems to be in the same spirit of the quote, the online scanned pdf version of Nothing Matters that I found doesn’t contain this quote at all. I don’t have access to any of the academic institutions to check other versions of the paper or book.
Maybe I just missed the quote by skimreading too quickly? Are there multiple versions of the article? Is it possible that this is a case of citogenesis? Perhaps Mackie misquoted what R.M. Hare said, or perhaps misattributed it and and it actually came from somewhere else? Maybe it was Mackie’s quote all along?
Help me EA Forum, you’re my only hope! I’m placing a £50 bounty to a charity of your choice for anyone who can find the original source of this quote, in R. M. Hare’s work or otherwise, as long as I can verify it (e.g. a screenshot of the quote if it’s from a journal/book I don’t have access to).