I am a rising junior at the University of Chicago (co-president of UChicago EA and founder of Rationality Group). I am mostly interested in philosophy (particularly metaethics, formal epistemology, and decision theory), economics, and entrepreneurship.
I also have a Substack where I post about philosophy (ethics, epistemology, EA, and other stuff). Find it here: https://substack.com/@irrationalitycommunity?utm_source=user-menu.
Reach out to me via email @ dnbirnbaum@uchicago.edu
Oh, sorry. I asked my friend, and they said it was Stop AI—not Pause AI. They basically protested in the middle of a talk with the CEA CEO saying things like he is a murderer, which I just think is pretty nuts. That is my bad, though. I have edited that part out of my initial comment.
I think I disagree with a bunch of what you said there: I don’t think good AIS involves being on good side of AI labs necessarily (tho I think there are good arguments for this), I think large movement building without getting a full pause would be a big win for PauseAI and that many EAs would agree with this (despite the fact that I and maybe them know little about social movements), and I do think making simple statements reflects negatively from a whole host of perspectives and should be taken pretty seriously.
I’d be interested in hearing how you/others at pause AI have tracked how much of this marginal improvements to advocacy you have been doing so far is. What are the wins? What were the costs? Happy to also get on a call with someone about this.
If there are real numbers on this (I know it’s hard in spaces like yours), I’d be curious about hearing why they aren’t often posted on the forum/LW, as that, I think, would be more helpful to people (and tracking the cost-effectiveness of AIS in general is super underrated, so this would be good).
If you don’t have the numbers, I would ask why you are so confident that this is actually the right approach.
Can you say more about what these EA assumptions are?
Disagreement is cool and awesome. Even intense disagreement (“I think this view is deeply misguided”). I really see no room for antagonism among two people that could be having an epistemically healthy conversation.
Sorry about your father.
I think there’s a much more mundane and much more epistemically healthy way to understand this disagreement.
Perhaps this is naive but my view currently just is: most people are disagreeing about a few concrete empirical and strategic parameters: the likely effectiveness and public reception of a Pause movement, how much meaningful safety work can be done from inside labs, and (probably) estimates of p(doom). Given how uncertain and high-stakes these questions are, it seems completely unsurprising that reasonable people would land in very different places.
It’s fine to worry about incentives and institutional bias — that could matter — but treating this as if the disagreement is obviously resolved, or as if it cleanly divides the world into “accelerationists” and “pause-ers,” strikes me as bad epistemics.
Interesting. A few thoughts:
Beyond strengthening the case for non-existential risks, if Sisyphus risk is substantial it also weakens arguments that place extreme weight on reducing existential risk at a specific time. Some of the importance of the Time of Perils comes from comparative advantage, which is diluted if civilization plausibly gets multiple runs.
One additional Sisyphean mechanism worth flagging is resource exhaustion: collapsing before reaching renewable resource self-sufficiency could permanently worsen later runs. This probably relies on a setback happening much later or a large amount of resources being used before, but it’s worth flagging.
A caveat on donation timing: even if post-AGI x-risk declines slowly, aligned AGI plausibly generates enormous resources, so standard impatient-philanthropy arguments may still apply if we have setbacks. And if we assume those resources are lost in a collapse, the same would likely apply to resources saved in advance.
Finally, the plausible setbacks all seem to hinge on something like the loss of knowledge. Other worries (e.g. Butlerian backlash) tend to rely on path-dependent successes—historically contingent timing, unusually alignable models, or specific public perceptions that don’t automatically replicate—seem hard to change conditional on setbacks. If those aren’t mostly luck-based and the relevant knowledge survives, a post-setback society could plausibly re-instantiate the same mechanisms, making Sisyphus risk primarily an epistemic rather than, say, a governance problem.
I think the COVID case usefully illustrates a broader issue with how “EA/rationalist prediction success” narratives are often deployed.
That said, this is exactly why I’d like to see similar audits applied to other domains where prediction success is often asserted, but rarely with much nuance. In particular: crypto, prediction markets, LVT, and more recently GPT-3 / scaling-based AI progress. I wasn’t closely following these discussions at the time, so I’m genuinely uncertain about (i) what was actually claimed ex ante, (ii) how specific those claims were, and (iii) how distinctive they were relative to non-EA communities.
This matters to me for two reasons.
First, many of these claims are invoked rhetorically rather than analytically. “EAs predicted X” is often treated as a unitary credential, when in reality predictive success varies a lot by domain, level of abstraction, and comparison class. Without disaggregation, it’s hard to tell whether we’re looking at genuine epistemic advantage, selective memory, or post-hoc narrative construction.
Second, these track-record arguments are sometimes used—explicitly or implicitly—to bolster the case for concern about AI risks. If the evidential support here rests on past forecasting success, then the strength of that support depends on how well those earlier cases actually hold up under scrutiny. If the success was mostly at the level of identifying broad structural risks (e.g. incentives, tail risks, coordination failures), that’s a very different kind of evidence than being right about timelines, concrete outcomes, or specific mechanisms.
I think the COVID case usefully illustrates a broader issue with how “EA/rationalist prediction success” narratives are often deployed.
That said, this is exactly why I’d like to see similar audits applied to other domains where prediction success is often asserted, but rarely with much nuance. In particular: crypto, prediction markets, LVT, and more recently GPT-3 / scaling-based AI progress. I wasn’t closely following these discussions at the time, so I’m genuinely uncertain about (i) what was actually claimed ex ante, (ii) how specific those claims were, and (iii) how distinctive they were relative to non-EA communities.
This matters to me for two reasons.
First, many of these claims are invoked rhetorically rather than analytically. “EAs predicted X” is often treated as a unitary credential, when in reality predictive success varies a lot by domain, level of abstraction, and comparison class. Without disaggregation, it’s hard to tell whether we’re looking at genuine epistemic advantage, selective memory, or post-hoc narrative construction.
Second, these track-record arguments are sometimes used—explicitly or implicitly—to bolster the case for concern about AI risks. If the evidential support here rests on past forecasting success, then the strength of that support depends on how well those earlier cases actually hold up under scrutiny. If the success was mostly at the level of identifying broad structural risks (e.g. incentives, tail risks, coordination failures), that’s a very different kind of evidence than being right about timelines, concrete outcomes, or specific mechanisms.
I can’t join this Sunday (finals season whoo!), but this is a really good idea. I’d love to see more initiatives like this to encourage writing on the Forum—especially during themed weeks.
Also, I’m always down to do (probably remote) co-working sessions with people who want to write Forum posts.
Strongly agreed. Organizing a group is probably one of the best things one could do for both their present and future impact.
I (and many others) would be happy to get on a call/help anyone willing to take over (I have a bunch of experience from organizing the UChicago group)! Dm me here to take me up on that.
Many questions in this space rely on assumptions about whether insect lives are positive or negative, though I haven’t seen much discussion of this explicitly (mostly just heard them in conversations). Is there not much that can be done to learn more other than what has already been done?
If so, the insect welfare initiatives that are interested in creating more or less are going to need to be dropped by 1-2 OOMs to account for massive uncertainty (say, 5% higher that they are positive), which is weird. It also wouldn’t be very robust in that the probability we do something good vs bad is quite fragile (especially a problem for many non hedonistic utilitarian views).
Do you have further takes here, Vasco?
I would look out for Rethink Priorities’ digital consciousness model (and other work they are doing here) which should be coming out soon-ish. I don’t think they would call it definitive in any sense, but it could be helpful here.
I think a major way this could be wrong is if you think we could get lots of digital minds in some amount of decades and early research/public engagement can have an oversized impact on shaping the conversation. This might make digital minds way more important, I think.
I’m also generally pretty interested in people doing more digital minds cross-cause prio (I’m working on a piece now)!
Re the new 2024 Rethink Cause Prio survey: “The EA community should defer to mainstream experts on most topics, rather than embrace contrarian views. [“Defer to experts”]” 3% strongly agree, 18% somewhat agree, 35% somewhat disagree, 15% strongly disagree.
This seems pretty bad to me, especially for a group that frames itself as recognizing intellectual humility/we (base rate for an intellectual movement) are so often wrong.
(Charitable interpretation) It’s also just the case that EAs tend to have lots of views that they’re being contrarian about because they’re trying to maximize the the expected value of information (often justified with something like: “usually contrarians are wrong, but if they are right, they are often more valuable for information than average person who just agrees”).
If this is the case, though, I fear that some of us are confusing the norm of being contrarian instrumental reasons and for “being correct” reasons.
Tho lmk if you disagree.
Some questions that might be cruxy and important for money allocation:
Because there is some evidence that superforecaster aggregation might underperform in AI capabilities, how should epistemic weight be distributed between generalist forecasters, domain experts, and algorithmic prediction models? What evidence exists/can be gotten about their relative track records?
Are there better ways to do AIS CEA? What are they?
Is there productive work to be done in inter-cause comparison among new potential cause areas (i.e. digital minds, space governance, etc)? What types of assumptions do these rely on? I ask because it seems like people typically go into these fields because “woah, those numbers are really big,” but that sort of reasoning applies to lots of those fields and doesn’t tell you very much about resource distribution.
What are the reputational effects for EA (for people inside and outside the movement) going (more) all in on certain causes and then being wrong (i.e. AI is and continues to be a bubble)? Should this change how much EA should go in on things? Under what assumptions?
This is great, and people should do this for more cause areas!
While I don’t have the bandwidth for this atm, someone should make a public (or private for, say, policy/reputation reasons) list of people working in (one or multiple of) the very neglected cause areas — e.g., digital minds (this is a good start), insect welfare, space governance, AI-enabled coups, and even AI safety (more for the second reason than others). Optional but nice-to-have(s): notes on what they’re working on, time contributed, background, sub-area, and the rough rate of growth in the field (you probably don’t want to decide career moves purely on current headcounts). And remember: perfection is gonna be the enemy of the good here.
Coordination.
It’s surprisingly hard to know who’s in these niches (independent researchers, part-timers, new entrants, maybe donors). A simple list would make it easier to find collaborators, talk to the right people, and avoid duplicated work.
Neglectedness clarity.
A major reason to work on ultra-neglected causes is… neglectedness. But we often have no real headcount, and that may push people into (or out of) fields they wouldn’t otherwise choose. Even technical AI safety numbers are outdated — the last widely cited 80k estimate (2022) was ~200 people, which is clearly very false now. (To their credit, they emphasized the difficulty and tried to update.)
Even rough FTE (full time equivalent) estimates + who’s active in each area would be a huge service for some fields.
Yea, sorry about that. That really sucks.