âEA-Adjacentâ now I guess.
đ¸ 10% Pledger.
Likes pluralist conceptions of the good.
Dislikes Bay Culture being in control of the future.
JWS đ¸
Karma: 4,547
First, I want to say thanks for this explanation. It was both timely and insightful (I had no idea about the LLM screening, for instance). So wanted to give that a big đ
I think something Jan is pointing to (and correct me if Iâm wrong @Jan_Kulveit) is that because the default Community tag does downweight the visibility and coverage of a post, it could be implicitly used to deter engagement from certain posts. Indeed, my understanding was that this was pretty much exactly the case, and was driven by a desire to reduce Forum engagement on âCommunityâ issues in the wake of FTX. See for example:
âKarma overrates some topics; resulting issues and potential solutionsâ from Lizka and Ben in January 2023
My comment and Lizkaâs response in the comments to that post
The reasoning given in the change announcement post which confirms it was for the âother motivesâ that Jan mentions. Thatâs at least how I read it.
Now, it is also true that I think the Forum was broadly supportive about this at the time. People were exhausted by FTX, and there seemed like there was a new devasting EA scandal every week, and being able to downweight these discussions and focus on ârealâ EA causes was understandably very popular.[1] So it wasnât even necessarily a nefarious change, it was responding to user demand.
Nevertheless I think, especially since criticisms of EA also come with the âCommunityâ tag attached,[2] it has also had the effect of somewhat reducing criticism and community sense-making. In retrospect, I still feel like the damage wrought by FTX hasnât had a full accounting, and the change to down-weight Community posts was trying to solve the âsymptomsâ rather than the underling issues.
Sharing some planned Forum posts Iâm considering, mostly as a commitment device, but welcome thoughts from others:
I plan to add another post in my âEA EDAâ sequence analysing Forum trends in 2024. My pre-registered prediction is that weâll see 2023 patterns continue: declining engagement (though possibly plateauing) and AI Safety further cementing its dominance across posts, karma, and comments.
Iâll also try to do another end-of-year Forum awards post (see here for last yearâs) though with slightly different categories.
Iâm working on an analysis of EAâs post-FTX reputation using both quantitative metrics (Forum engagement, Wikipedia traffic) and qualitative evidence (public statements from influential figures inside and outside EA). The preliminary data suggests more serious reputational damage than the recent Pulse survey. I meaningful (as opposed to methodological or just a mistake) I suspect it might highlight the difference between public and elite perception.
I recently finished reading former US General Stanley McChrystalâs book: Team of Teams. Ostensibly itâs a book about his command of JSOC in the Iraq War, but itâs really about the concept of Auftragstaktik as a method of command, and there was more than one passage which I thought was relevant to Effective Altruism (especially for what âThird Waveâ EA might mean). This one is a stretch though, Iâm not sure how interested the Forum would be for this, or whether it would be the right place to post it.
My focus for 2025 will be to work towards developing my position on AI Safety, and share that through a series of posts AI Safety sequence.[1] The concept of AGI went mainstream in 2024, and it does look like we will see significant technological and social disruption in the coming decades due to AI development. Nevertheless, I find myself increasingly skeptical of traditional narratives and arguments about what Alignment is, the likelihood of risk, and what ought to be done about it. Instead, Iâve come to view âAlignmentâ primarily as a political philosophy rather than a technical computer science. Nevertheless, I could very well be wrong on most-all of these ideas, and getting critical discussion from the community will I think be good both for myself and (I hope) the Forum readership.[2]
As such, Iâm considering doing a deep-dive on the Apollo o1 report given the controversial reception itâs had.[3] I think this is the most unlikely one though, as Iâd want to research it as thoroughly as I could, and time is at a premium since Christmas is around the corner, so this is definitely a âstretch goalâ.
Finally, I donât expect to devote much more time[4] to adding to the âCriticism of EA Criticismâ sequence. I often finish the posts well after the initial discourse has died down, and Iâm not sure what effect they really have.[5] Furthermore, and Iâve started to notice my own views of a variety of topics start to diverge from âEA Orthodoxyâ, so Iâm not really sure Iâd make a good defender. This change may itself warrant a future post, though again Iâm not committing to that yet.
- ^
Which I will rename
- ^
It possibly may be more helpful for those without technical backgrounds concerned about AI, but Iâm not sure. I also think have a somewhat AGI-sceptical persepctive represented on the Forum might be useful for intellectual diversity purposes but I donât want to claim that. Iâm very uncertain about the future of AI and could easily see myself being convinced to change my mind.
- ^
Iâm slightly leaning towards the skeptical interpretation myself, as you might have guessed
- ^
if any at all, unless an absolutely egregious but widely-shared example comes up
- ^
Does Martin Sandbu read the EA Forum, for instance?
I think this is, to a significant extent, definitionally impossible with longtermist interventions, because the âlong-termâ part excludes having an empirical feedback loop quick enough to update our models of the world.
For example, if Iâm curious about whether malaria net distribution or vitamin A supplementation is more âcost-effectiveâ than another, I can fund interventions and run RCTs, and then model the resulting impact according to some metric like the DALY. This isnât cast-iron secure evidence, but it is at least causally connected to the result I care about.
For interventions that target the long-run future of humanity, this is impossible. We canât run counterfactuals of the future or past, and I at least canât wait 1000 years to see the long-term impact of certain decisions on the civilizational trajectory of the world. Thus, any longtermist intervention cannot really get empirical feedback on the parameters of action, and mostly rely on subjective human judgement about them.
To their credit, the EA Long-Term Future Fund says as much on their own web page:
Unfortunately, there is no robust way of knowing whether succeeding on these proxy measures will cause an improvement to the long-term future.
For similar thoughts, see Laura Duffyâs thread on empirical vs reason-driven EA
One potential weakness is that Iâm curious if it promotes the more well-known charities due to the voting system. Iâd assume that these are somewhat inversely correlated with the most neglected charities.
I guess this isnât necessarily a weakness if the more well-known charities are more effective? I can see the case that: a) they might not be neglected in EA circles, but may be very neglected globally compared to their impact and that b) there is often an inverse relationship between tractability/âneglectedness and importance/âimpact of a cause area and charity. Not saying youâre wrong, but itâs not necessarily a problem.
Furthermore, my anecdotal take from the voting patterns as well as the comments on the discussion thread seem to indicate that neglectedness is often high on the mind of votersâthough I admit that commenters on that thread are a biased sample of all those voting in the election.
It can be a bit underwhelming if an experiment to try to get the crowdâs takes on charities winds up determining to, âjust let the current few experts figure it out.â
Is it underwhelming? I guess if you want the donation election to be about spurring lots of donations to small, spunky EA-startups working in weird-er cause areas, it might be, but I donât think thatâs what I understand the intention of the experiment to be (though I could be wrong).
My take is that the election is an experiment with EA democratisation, where we get to see what the community values when we do a roughly 1-person-1-ballot system instead of those-with-the-moeny decide system which is how things work right now. Those takeaways seem to be:
The broad EA community values Animal Welfare a lot more than the current major funders
The broad EA community sees value in all 3 of the âbig cause areasâ with high-scoring charities in Animal Welfare, AI Safety, and Global Health & Development.
But you havenât provided any data đ¤ˇ
Like you could explain why you think so without de-anonymising yourself, e.g. sammy shouldnât put EA on his CV in US policy because:Republicans are in control of most positions and they see EA as heavily democrat-coded and arenât willing to consider hiring people with it
The intelligentsia who hire for most US policy positions see EA as cult-like and/âor disgraced after FTX
People wonât understand what EA is on a CV will and discount sammyâs chances compared to them putting down âran a discussion group at universityâ or something like that
You think EA is doomed/âlikely to collapse and sammy should pre-emptively dissasociate their career from it
Like I feel that would be interesting and useful to hear your perspective on, to the extend you can share information about it. Otherwise just jumping in with strong (and controversial?) opinions from anonymous accounts on the forum just serves to pollute the epistemic commons in my opinion.
Right but I donât know who you are, or what your position in the US Policy Sphere is, if you have one at all. I have no way to verify your potential background or the veracity of the information you share, which is one of the major problems with anonymous accounts.
You may be correct (though again that lack of explanation doesnât help give detail or a mechanism why or help sammy that much, as you said it depends on the section) but that isnât really the point, the only data point you provide is âintentionally anonymous person of the EAForum states opinion without supporting explanationsâ which is honestly pretty weak sauce
I donât find comments like these helpful without explanations or evidence, especially from throwaway accounts
Yeah again I just think this depends on oneâs definition of EA, which is the point I was trying to make above.
Many people have turned away from EA, both the beliefs, institutions, and community in the aftermath of the FTX collapse. Even Ben Todd seems to not be an EA by some definitions any more, be that via association or identification. Who is to say Leopold is any different, or has not gone further? What then is the use of calling him EA, or using his views to represent the âThird Waveâ of EA?
I guess from my PoV what Iâm saying is that Iâm not sure thereâs much âconnective tissueâ between Leopold and myself, so when people use phrases like âlisten to usâ or âHow could we have doneâ I end up thinking âwho the heck is we/âus?â
Iâm not sure to what extent the Situational Awareness Memo or Leopold himself are representatives of âEAâ
In the pro-side:
Leopold thinks AGI is coming soon, will be a big deal, and that solving the alignment problem is one of the worldâs most important priorities
He used to work at GPI & FTX, and formerly identified with EA
He (
probablyalmost certainly) personally knows lots of EA people in the Bay
On the con-side:
EA isnât just AI Safety (yet), so having short timelines/âhigh importance on AI shouldnât be sufficient to make someone an EA?[1]
EA shouldnât also just refer to a specific subset of the Bay Culture (please), or at least we need some more labels to distinguish different parts of it in that case
Many EAs have disagreed with various parts of the memo, e.g. Gideonâs well received post here
Since his EA institutional history he moved to OpenAI (mixed)[2] and now runs an AGI investment firm.
By self-identification, Iâm not sure Iâve seen Leopold identify as an EA at all recently.
This again comes down to the nebulousness of what âbeing an EAâ means.[3] I have no doubts at all that, given what Leopold thinks is the way to have the most impact heâll be very effective at achieving that.
Further, on your point, I think thereâs a reason to suspect that something like situational awareness went viral in a way that, say, Rethink Priorities Moral Weight project didnâtâthe promise many people see in powerful AI is power itself, and thatâs always going to be interesting for people to follow, so Iâm not sure that situational awareness becoming influential makes it more likely that other âEAâ ideas will
- ^
Plenty of e/âaccs have these two beliefs as well, they just expect alignment by default, for instance
- ^
I view OpenAI as tending implicitly/âexplicitly anti-EA, though I donât think there was an explicit âpurgeâ, I think the culture/âvision of the company was changed such that card-carrying EAs didnât want to work there any more
- ^
The 3 big defintions I have (self-identification, beliefs, actions) could all easily point in different directions for Leopold
I sort-off bounced of this one Richard. Iâm not a professor of moral philosophy, so some of what I say below may seem obviously wrong/âstupid/âincorrectâbut I think that were I a philosophy professor I would be able to shape it into a stronger objection than it might appear on first glance.
Now, when people complain that EA quantifies things (like cross-species suffering) that allegedly âcanât be precisely quantified,â what theyâre effectively doing is refusing to consider that thing at all.
I donât think this would pass an ideological Turing Test. I think what people who make this claim are saying is often that previous attempts to quantify the good precisely have ended up having morally bad consequences. Given this history, perhaps our takeaway shouldnât be âthey werenât precise enough in their quantificationâ and should be more âperhaps precise quantification isnât the right way to go about ethicsâ.
Because the realistic alternative to EA-style quantitative analysis is vibes-based analysis: just blindly going with whatâs emotionally appealing at a gut level.
Again, I donât think this is true. Would you say that before the publication of Famine, Affluence, and Morality that all moral philosophy was just âvibes-based analysisâ? I think, instead, all of moral reasoning is in some sense âvibes-basedâ and the quantification of EA is often trying to present arguments for the EA position.
To state it more clearly, what we care about is moral decision-making, not the quantification of moral decisions. And most decisions that have been made or have ever been made have been done so without quantification. What matters is the moral decisions we make, and the reasons we have for those decisions/âvalues, not what quantitative value we place on said decisions/âvalues.
the question that properly guides our philanthropic deliberations is not âHow can I be sure to do some good?â but rather, âHow can I (permissibly) do the most (expected) good?â
I guess Iâm starting to bounce of this because I now view this as a big moral commitment which I think goes beyond simple beneficentrism. Another view, for example, would be a contractualism, where what âdoing goodâ means is substantially different from what you describe here, but perhaps thatâs a base metaethical debate.
Itâs very conventional to think, âPrioritizing global health is epistemically safe; you really have to go out on a limb, and adopt some extreme views, in order to prioritize the other EA stuff.â This conventional thought is false. The truth is the opposite. You need to have some really extreme (near-zero) credence levels in order to prevent ultra-high-impact prospects from swamping more ordinary forms of do-gooding.
I think this is confusing two forms of âextremeâ. Like in one sense the default âanimals have little-to-no moral worthâ view is extreme for setting the moral value of animals so low as to be near zero (and confidently so at that). But I think the âextremeâ in your first sentence refers to âextreme from the point of view of societyâ.
Furthermore, if we argue that quantifying expected value in quantitative models is the right way to do moral reasoning (as opposed to sometimes being a tool), then you donât have to accept the âeven a 1% chance is enoughâ, I could just decline to find a tool that produces such dogmatism at 1% acceptable. You could counter with âyour default/âstatus-quo morality is dogmaticâ, which sure. But it doesnât convince me to accept strong longtermism any more, and Iâve already read a fair bit about it (though I accept probably not as much as you).
While youâre at it, take care to avoid the conventional dogmatism that regards ultra-high-impact as impossible.
One manâs âconventional dogmatismâ could be reframed as âthe accurate observation that people with totalising philosophies promising ultra-high-impact have a very bad track record that have often caused harm and those with similar philosophies ought to be viewed with suspicionâ
Sorry if the above was a bit jumbled. It just seemed this post was very unlike your recent Good Judgement with Numbers post, which I clicked with a lot more. This one seems to be you, instead of rejecting the âAll or Nothingâ Assumption, actually going âall inâ on quantitative reasoning. Perhaps it was the tone with which it was written, but it really didnât seem to actually engage with why people have an aversion to over-quantification of moral reasoning.
Thanks for sharing your thoughts. Iâll respond in turn to what I think are the two main parts of it, since as you said this post seems to be a combination of suffering-focused ethics and complex cluelessness.
On Suffering-focused Ethics: To be honest, Iâve never seen the intuitive pull of suffering-focused theories, especially since my read of your paragraphs here seems to tend towards a lexical view where the amount of suffering is the only thing that matters for moral consideration.[1]
Such a moral view doesnât really make sense to me, to be honest, so Iâm not particularly concerned by it, though of course everyone has different moral intuitions so YMMV.[2] Even if youâre convinced of SFE though, the question is how best to reduce suffering which hits into the clueless considerations you point out.
On complex cluelessness: On this side, I think youâre right about a lot of things, but thatâs a good thing not a bad one!
I think youâre right about the âtime of perilsâ assumption, but you really should increase your scepticism of any intervention which claims to have âlasting, positive effects over millenniaâ since we canât get the feedback on the millennia long impact of our interventions.
You are right that radical uncertainty is humbling, and it can be frustrating, but it is also the state that everyone is in, and thereâs no use beating yourself up for the default state that everyone is in.
You can only decide how to steer humanity toward a better future with the knowledge and tools that you have now. It could be something very small, and doesnât have to involve you spending hundreds of hours trying to solve the problems of cluelessness.
Iâd argue that reckoning with the radical uncertainty should point towards moral humility and pluralism, but I would say that since thatâs the perspective in my wheelhouse! I also hinted at such considerations in my last post about a Gradient-Descent approach to doing good, which might be a more cluessness-friendly attitude to take.
- ^
You seem to be asking e.g. âwill lowering existential risk increase the expected amount of future sufferingâ instead of âwill lowering existential risk increase the amount of total preferences satisfied/ânon frustratedâ for example.
- ^
To clairfy, this sentence specifically referred to lexical suffering views, not all forms of SFE that are less strong in their formulation
Edit: Confused about the downvoting hereâis it a âthe Forum doesnât need more of this community dramaâ feeling? I donât really include that much of a personal opinion to disagree with, and I also encourage people to check out Lincolnâs whole response đ¤ˇ
For visibility, on the LW version of this post Lincoln Quirkâmember of the EV UK board made some interesting comments (tagging @lincolnq to avoid sub-posting). I thought itâd be useful to have visibility of them on the Forum. A sentence which jumped out at me was this:
Personally, Iâm still struggling with my own relationship to EA. Iâve been on the EV board for a year+ - an influential role at the most influential meta orgâand I donât understand how to use this role to impact EA.
If one of the EV board members is feeling this way and doesnât know what to do, what hope for rank-and-file EAs? Is anyone driving the bus? Feels like a negative sign for the broader âEA projectâ[1] if this feeling goes right to the top of the institutional EA structure.
That sentence comes near the end of a longer, reflective comment, so I recommend reading the full exchange to take in Lincolnâs whole perspective. (Iâll probably post my thoughts on the actual post sometime next week)
- ^
Which many people reading this might feel positive about
- ^
A thought about AI x-risk discourse and the debate on how âPascalâs Muggingâ-like AIXR concerns are, and where this causes confusion between those concerned and sceptical.
I recognise a pattern where a sceptic will say âAI x-risk concerns are like Pascalâs wager/âare Pascalian and not validâ and then an x-risk advocate will say âBut the probabilities arenât Pascalian. Theyâre actually fairly largeâ[1], which usually devolves into a âThese percentages come from nowhere!â âBut Hinton/âBengio/âRussell...â âJust useful idiots for regulatory capture...â discourse doom spiral.
I think a fundamental miscommunication here is that, while the sceptic is using/âimplying the term âPascallianâ they arenât concerned[2] with the percentage of risk being incredibly small but high impact, theyâre instead concerned about trying to take actions in the worldâespecially ones involving politics and powerâon the basis of subjective beliefs alone.In the original wager, we donât need to know anything about the evidence record for a certain God existing or not, if we simply Pascalâs framing and premisses then we end up with the belief that we ought to believe in God. Similarly, when this term comes up, AIXR sceptics are concerned about changing beliefs/âbehaviour/âenact law based on arguments from reason alone that arenât clearly connected to an empirical track record. Focusing on which subjective credences are proportionate to act upon is not likely to be persuasive compared to providing the empirical goods, as it were.
Something which has come up a few times, and recently a lot in the context of Debate Week (and the reaction to Leifâs post) is things getting downvoted quickly and being removed from the Front Page, which drastically drops the likelihood of engagement.[1]
So a potential suggestion for the Frontpage might be:
Hide the vote score of all new posts if the absolute score of the post is below some threshold (Iâll use 20 as an example)
If a post hits â20, it drops off the front page
After a post hits 20+, itâs karma score is permanently revealed
Galaxy-brain version is that Community/âNon-Community grouping should only take effect once a post hits these thresholds[2]
This will still probably leave us with too many new posts to fit on the front page, so some rules to sort which stay and which get knocked off:
Some consideration to total karma should probably count (how much to weight it is debatable)
Some consideration to how recent the post is should count too (e.g. Iâd probably want to see a new post that got 20+ karma quickly than 100+ karma over weeks)
Some consideration should also go to engagementâsome metric related to either number of votes or comment count would probably indicate which posts are generating community engagement, though this could lead to bikeshedding/âMatthew effect if not implemented correctly. I still think itâs directionally correct though
Of course the userâs own personal weighting of topic importance can probably contribute to this as well
There will always be some trade-offs when designing some ranking on many posts with limited space. But the idea above is that no post should quickly drop off the front page because a few people quickly down-vote it into negative karma.
Maybe some code like this already exists, but this thought popped into my head and I thought it was worth sharing on this post.
- ^
My poor little piece on gradient descent got wiped out by debate week đ rip
- ^
In a couple of places Iâve seen people complain about the use of the Community tag to âhideâ particular discussions/âtopics. Not saying I fully endorse this view.
I think âmeat-eating problemâ > âmeat-eater problemâ came in my comment and associated discussion here, but possibly somewhere else.[1]
- ^
(I still stand by the comment, and I donât think itâs contradictory with my current vote placement on the debate week question)
- ^
GraÂdiÂent DesÂcent as an analÂogy for DoÂing Good
On the platonic/âphilosophical side Iâm not sure, I think many EAs werenât really bought into it to begin with and the shift to longtermism was in various ways the effect of deference and/âor cohort effects. In my case I feel that the epistemic/âcluelessness challenge to longtermism/âfar future effects is pretty dispositive, but Iâm just one person.
On the vibes side, I think the evidence is pretty damning:The launch of WWOTF was almost perfectly at the worst time possible and the idea seems indelibly linked with SBFâs risky/ânaĂŻve ethics and immoral actions.
Do a Google News or Twitter search for âlongtermismâ in its EA context and itâs ~broadly to universally negative. The Google trends data also points toward the term fading away.
No big EA org or âEA leaderâ however defined is going to bat for longtermism any more in the public sphere. The only people talking about it are the critics. When you get that kind of dynamic, itâs difficult to see how an idea can survive.
Even on the Forum, very little discussion on the Forum seems to be based on âlongtermismâ these days. People either seem to have left the Forum/âEA, or longtermist concerns have been subsumed into AI/âbio risk. Longtermism just seems superfluous to these discussions.
Thatâs just my personal read on things though. But yeah, seems very much like that SBF-Community Drama-OpenAI board triple whammy from Nov22-Nov23 marked the death knell for longtermism at least as the public facing justification of EA.
For the avoidance of doubt, not gaining knowledge from the Carl Shulman episodes is at least as much my fault as it is Rob and Carlâs![1] I think similar to his appearance on the Dwarkesh Podcast, it was interesting and full of information, but Iâm not sure my mind has found a good way to integrate it into my existing perspective yet. It feels unresolved to me, and something I personally want to explore more, so a version of the post written later in time might include those episodes high up. But writing this post from where I am now, I at least wanted to own my perspective/âbias leaning against the AI episodes rather than leave it implicit in the episode selection. But yeah, it was very much my list, and therefore inherits all of my assumptions and flaws.
I do think working in AI/âML means that the relative gain of knowledge may still be lower in this case compared to learning about the abolition of slavery (Brown #145) or the details of fighting Malaria (Tibenderana #129), so I think thatâs a bit more arguable, but probably an unimportant distinction.
- ^
(Iâm pretty sure I didnât listen to part 2, and canât remember how much I listened to of part 1 over reading some of the transcript on the 80k website, so these episodes may be a victim of the ânot listened to fully yetâ criteria)
- ^
Just flagging this for context of readers, I think Habrykaâs position/âreading makes more sense if you view it in the context of an ongoing Cold War between Good Ventures and Lightcone.[1]
Some evidence on the GV side:
The change of funding priorities from Good Ventures seems to include stopping any funding for Lightcone.
Dustin seems to associate the decoupling norms of Lightcone with supporting actors and beliefs that he wants to have nothing to do with.
Dustin and Oli went back and forth in the comments above, some particularly revealing comments from Dustin are here and here, which even if are an attempt at gallows humour, to me also show a real rift.
To Habrykaâs credit, itâs much easier to see what the âLightcone Ecosystemâ thinks of OpenPhil!
He thinks that the actions of GV/âOP were and currently are overall bad for the world.
I think the reason why is mostly given here by MichaelDickens on LW, Habryka adds some more concerns in the comments. My sense is that the LW commentariat is turning increasingly against OP but thatâs just a vibe I have when skim-reading.
Some of it also appears to be for reasons to do with the Lightcone-aversion-to-âdeceptionâ-broadly-defined, which one can see from the Habrykaâs reasoning in this post or replying here to Luke Muehlhauser. This philosophy doesnât seem to explained in one place, Iâve only gleaned what I can from various posts/âcomments so if someone does have a clearer example then feel free to point me in that direction.
This great comment during the Nonlinear saga I think helps make a lot of Lightcone v OP discourse make sense.
I was nervous about writing this because I donât want to start a massive flame war, but I think itâs helpful for the EA Community to be aware that two powerful forces in it/âadjacent to it[2] are essentially in a period of conflict. When you see comments from either side that seem to be more aggressive/âhostile than you otherwise might think warranted, this may make the behaviour make more sense.
Note: I donât personally know any of the people involved, and live half a world away, so expect it to be very inaccurate. Still, this âframeâ has helped me to try to grasp what I see as behaviours and attitudes which otherwise seem hard to explain to me, as an outsider to the âEA/âLW in the Bayâ scene.
To my understanding, the Lightcone position on EA is that it âshould be disavowed and dismantledâ but thereâs no denying the Lightcone is closer to EA than ~most all other organisations in some sense