I am a rising junior at the University of Chicago (co-president of UChicago EA and founder of Rationality Group). I am mostly interested in philosophy (particularly metaethics, formal epistemology, and decision theory), economics, and entrepreneurship.
I also have a Substack where I post about philosophy (ethics, epistemology, EA, and other stuff). Find it here: https://substack.com/@irrationalitycommunity?utm_source=user-menu.
Reach out to me via email @ dnbirnbaum@uchicago.edu
Whooooo!!!!
I agree with all of this. One thing missing is just joining AIS/EA before it was cool/got much more money.
The Fallacy of Similar Magnitudes
A name I came up with for a pattern that comes up often enough to deserve one. Here’s a particular instance:
Suppose you’re a longtermist who thinks the vast majority of the (net positive) moral weight is in the far future, that AI existential risk is above 1%, and that it’s reasonably tractable to reduce. Some people who hold all three premises still say things like: “AI safety has maybe two orders of magnitude more resources than animal welfare, so the marginal dollar is better spent on animal welfare.”
This is the fallacy of similar magnitudes: in this case, the application is that neglectedness comparisons only matter when the causes are within a couple orders of magnitude of each other in stakes. Under the stated premises, they aren’t: the a priori difference in expected value is way larger. Two orders of magnitude in resources doesn’t move the needle even a bit.
Note: this AW response could be ok if you have certain views about risk aversion or worldview diversification, but that’s besides the point.
I am helping co-run the Generator Residency, which enables generalist talent into AIS, and we will soon be posting lots of ideas we think generalists in AIS could work on. Be on the lookout!
Lots of people in the Bay seem to be thinking about/preparing for/making funding decisions based on the idea that lots of philanthropy will be given to AIS/EA cause areas very soon (i.e. end of year-ish). I would love for someone to write the comprehensive steel man case against this, as I think it’s probably underrated (some reasons to think they won’t give the money/it won’t be as much as some assume. Happy to comment/ speak to whoever is interested in doing this.
Post this on LW and youll prob get more offers
This is a very good post.
Lucas! How r u??
I wonder how much value there would be in a company that principally doesn’t engage in sports betting. From my understanding, lots of people that bet on these markets like betting on non-sports claims but just hate the sports market component.
The EA Forum seems super dead, which is pretty bad. I think part of the reason is that much of the Forum gets cross-posted to LessWrong, so you get EA Forum content there plus everything else (especially AI stuff) — which means there’s little reason to actually come here.
One take: don’t cross-post to LessWrong. There are real benefits to having two intellectual communities that take the same kinds of ideas seriously but develop distinct cultures, emphases, and standards around them.
Other ways to help:
Do lots of cause prio out loud — especially the parts that haven’t been written about much
If your private doc doesn’t need to be private, don’t make it private
More EA in da news: https://x.com/DavidSacks/status/2034047505336295904
And the spicy CAIS take: https://x.com/cais/status/2034389842076025164?s=46
Experts currently treat being persuaded as reasonably good evidence that something is true — their judgment is calibrated enough that when they find an argument convincing, that’s correlated with the argument actually being correct. This allows them to update readily in light of new evidence, and is a big part of how intellectual progress happens: lots of innovation and advances in basically every subject come down to experts taking sometimes weird new ideas seriously.
One worry I have about superpersuasive AI is that it could erode this. If a superpersuasive AI can convince experts of things regardless of whether those things are true, experts may cease to see themselves being persuaded as good evidence that something is true — and start treating it the way laypeople do. Laypeople are typically hesitant to take on new, truth-tracking beliefs in light of new information, and (to some degree) rationally so: the fact that someone was able to convince a layperson of something is just not very strong evidence that it is in fact true. Experts might end up in the same position — only updating rarely, and in ways that are often unrelated to the truth.
This would be quite bad. If experts lose their capacity to reliably update on genuine evidence, we could significantly slow the rate of intellectual progress (which could be very important for making AI go well!). This is, I think, an underappreciated argument for caring about AI for epistemics — curious what others think.
Some videos from Rational Animations and Rob Miles would be cool, I think.
How much of a post are you comfortable for AI to write?
Some posts are meant for literary beauty and some are meant for ideas. Writing ideas well takes effort, and writing ideas with AI takes basically zero effort. If your post is mainly about ideas, and the fact that it is written by an AI doesn’t make it annoying to read, I really just don’t see why you wouldn’t write basically entirely with AI.
This might feel obvious, but I think it’s under-appreciated how much disagreement on AI progress just comes down to priors (in a pretty specific way) rather than object-level reasoning.
I was recently arguing the case for shorter timelines to a friend who leans longer. We kept disagreeing on a surprising number of object-level claims, which was weird because we usually agree more on the kinda stuff we were arguing about.
Then I basically realized what I think was going on: she had a pretty strong prior against what I was saying, and that prior is abstract enough that there’s no clear mechanism by which I can push against it. So whenever I made a good object-level case, she’d just take the other side — not necessarily because her reasons were better all else equal, but because the prior was doing the work underneath without either of us really knowing it.
There’s something clearly rational here that’s kinda unintuitive to get a grip on. If you have a strong prior, and someone makes a persuasive argument against it, but you can’t identify the specific mechanism by which their argument defeats it, you should probably update that the arguments against their case are better than they appear, even if you can’t articulate them yet. From the outside, this totally just looks like motivated reasoning (and often is), but I think it can be pretty importantly different.
The reason this is so hard to disentangle is that (unless your belief web is extremely clear to you, which seems practically impossible) it’s just enormously complicated. Your prior on timelines isn’t an isolate thing — it’s load-bearing for a bunch of downstream beliefs all at once. So the resistance isn’t obviously irrational, it’s more like… the system protecting its own coherence.
I think this means that people should try their best to disentangle whether some object level argument they’re having comes from real object level beliefs or pretty abstract priors (in which case, it seems less worthwhile to press on them).
The Forum should normalize public red-teaming for people considering new jobs, roles, or project ideas.
If someone is seriously thinking about a position, they should feel comfortable posting the key info — org, scope, uncertainties, concerns, arguments for — and explicitly inviting others to stress-test the decision. Some of the best red-teaming I’ve gotten hasn’t come from my closest collaborators (whose takes I can often predict), but from semi-random thoughtful EAs who notice failure modes I wouldn’t have caught alone (or people think pretty differently so can instantly spot things that would have taken me longer to figure out).
Right now, a lot of this only happens at EAGs or in private docs, which feels like an information bottleneck. If many thoughtful EAs are already reading the Forum, why not use it as a default venue for structured red-teaming?
Public red-teaming could:
reduce unilateralist mistakes,
prevent coordination failures (I’ve almost spent serious time on things multiple people were already doing — reinventing the wheel is common and costly),
Obviously there are tradeoffs — confidentiality, social risk, signaling concerns — but I’d be excited to see norms shift toward “post early, get red-teamed, iterate publicly,” rather than waiting for a handful of coffee chats.
Here’s a random org/project idea: hire full-time, thoughtful EA/AIS red teamers whose job is to seriously critique parts of the ecosystem — whether that’s the importance of certain interventions, movement culture, or philosophical assumptions. Think engaging with critics or adjacent thinkers (e.g., David Thorstad, Titotal, Tyler Cowen) and translating strong outside critiques into actionable internal feedback.
The key design feature would be incentives: instead of paying for generic criticism, red teamers receive rolling “finder’s fees” for critiques that are judged to be high-quality, good-faith, and decision-relevant (e.g., identifying strategic blind spots, diagnosing vibe shifts that can be corrected, or clarifying philosophical cruxes that affect priorities).
Part of why I think this is important is because I generally think have the intuition that the marginal thoughtful contrarian is often more valuable than the marginal agreer, yet most movement funding and prestige flows toward builders rather than structured internal critics. If that’s true, a standing red-team org — or at least a permanent prize mechanism — could be unusually cost-effective.
There have been episodic versions of this (e.g., red-teaming contests, some longtermist critiquing stuff), but I’m not sure why this should come in waves rather than exist as ongoing infrastructure (org or just some prize pool that’s always open for sufficiently good criticisms).
Seems right, though plausibly there are some EA/EA-adjacent students at some of these schools.
As money increases, the thing that is needed is more ideas for where to put money and more people to act on big ideas to deploy it. As a result, I think EA CB (which results in both) is likely the best place to put money into!