pre-doc at Data Innovation & AI Lab
previously worked in options HFT and tried building a social media startup
founder of Northwestern EA club
Charlie_Guthmann
Karma: 1,154
Hmm well isn’t this basically the core project of EA aha? anyway thanks for sharing I chatted my two cents into chaptgpt 5.5 and had it format/write it out for me.
The Foundation should target market failures: public goods, externalities, information failures, coordination failures, etc. That means being suspicious of “important but already incentivized” domains. I start with this because of an example they have already expressed interest in—Alzheimer’s is important, but it is not obviously where a new mega-foundation has the highest marginal leverage. There are already enormous commercial incentives to develop effective treatments: rich patients, aging societies, and pharmaceutical companies all badly want a breakthrough. The philanthropic opportunity is not “Alzheimer’s” in general, but specific neglected bottlenecks inside Alzheimer’s: open datasets, repurposed generics, trial infrastructure, non-patentable interventions, prizeable biomarkers, fixing and standardizing medical ontologies, or other areas where private returns diverge from social returns.The bigger bottleneck is that we do not have a trustworthy system for converting hundreds of billions into impact. Conventional grantmaking is too opaque, too relationship-driven, too vulnerable to value lock-in, and too dependent on a small number of people’s cached worldviews. If you give that system $180 billion and scale quickly, you mostly get larger versions of the same failure modes.
So my first rule would be: do not spend the endowment quickly (yes I also have short timelines so need to balance that but won’t get into that rn). For the first two or three years, spend a small fraction building the allocation machine: the mechanisms, audits, forecasting systems, and public models that make later spending less dumb. The Foundation should have three arms.
First, a pull-funding arm. This should be the biggest. Wherever outcomes can be specified reasonably well, the Foundation should stop trying to guess the best grantees ex ante and instead pay for results. This is the logic behind results-based financing, advance market commitments, prizes, and market-shaping work. If you want pandemic preparedness, reward verified improvements in surveillance, cheap diagnostics, vaccine platform readiness, PPE resilience, or rapid clinical trial capacity. If you want AI governance capacity, reward usable evals, security benchmarks, model-control tools, compute-accounting systems, or policy infrastructure that actually gets adopted. If you want global health impact, pay for credible QALYs or DALYs averted, while being explicit about the moral weights and assumptions underneath. This is not a magic bullet. Pull funding Goodharts whatever it measures and favors legible outcomes. But it has one huge virtue: it forces the Foundation to say what it actually wants. If the Foundation pays for QALYs, animal welfare improvements, verified safety evals, or reductions in catastrophic risk, then people can argue about those metrics directly instead of reverse-engineering the worldview of a grants committee. For every major pull-funding program, I would reserve 5–10% of the budget for adversarial audits: rewards for showing how the metric can be gamed, why the measured outcome is not the real outcome, or why the program is selecting for fake impact.
Second, a push-funding arm. Some things cannot be bought through clean outcome contracts. You sometimes need to fund inputs: weird researchers, early science, institution-building, field creation, adversarial work, and long-horizon bets where the output is not immediately measurable. But push funding should be treated as the dangerous, high-discretion part of the portfolio, not the default. Every major push grant should come with a public theory of change, a forecast distribution over key outcomes, conflict-of-interest disclosures, and a plan for retrospective evaluation. Here I would borrow from Squiggle, Guesstimate, Metaculus, QURI, the longtermist wiki/crux project and the broader EA modeling tradition. The goal is not to pretend these models are precise. The goal is to make uncertainty explicit enough that people can find the weak points. If a grant depends on “this reduces p(doom) by 0.01%,” say that. If it depends on shrimp having nontrivial moral weight, say that. If it depends on institutional lock-in being more important than technical alignment, say that. Then pay smart critics to attack the model.
Third, an infrastructure-and-audit arm. This is the least glamorous and probably the highest-leverage part. The Foundation should build a grantmaking stack that includes financial audits, evidence synthesis, reference-class forecasting, red-team review, prediction markets, grant outcome tracking, and public postmortems. It should maintain a live map of cause areas, interventions, assumptions, evidence quality, and open cruxes.
The OpenAI conflict requires special rules. The Foundation should not fund evals, governance work, safety audits, or policy organizations that may affect OpenAI through ordinary discretionary grantmaking. Those grants should go through an independently governed firewall: external reviewers, public recusals, guaranteed publication rights, and a presumption that negative findings can be published. Otherwise, even good grants will look like reputation laundering or soft capture.
In the first six months, I would make only continuation grants and small exploratory grants. The main work would be hiring mechanism designers, economists, forecasters, auditors, AI safety people, domain experts, and institutional skeptics.
In year one, I would launch pilot programs: maybe $100–300 million across pull-funding experiments, 50 mil model-based push grants, and 300-500 mil audit systems. The goal would not be to maximize immediate impact. The goal would be calibration: which mechanisms produce real information, which get gamed, which attract talent, and which reveal hidden bottlenecks?
In years two and three, I would scale only mechanisms that survive adversarial review. The Foundation could then begin spending billions annually, but only through channels that have been stress-tested. I would heavily cap opaque discretionary grantmaking (with some sort of push through mechanism requiring super majority of disagreeable people) and require retrospective public evaluation for large grants.
The Foundation’s comparative advantage is not just money. It has the capability to be the most tech savy/automated/inference dense granter ever. If it just becomes a giant grantmaker with more zeros, it will lock in the worldview and social network of whoever happens to be close to the money. If it builds transparent pull funding, disciplined push funding, and serious audit infrastructure, it can make many other actors smarter too.
Glad you’re fleshing this out and pushing the community to take variety/diversity more seriously as part of population axiology. I’ve had similar thoughts in this direction, and I think the core intuition is very compelling.
Caveat that I’ve only skimmed maybe a quarter of the full post so far, and I can already see that it goes well beyond the simple claim that “variety matters”: it adds a lot of context, formal structure, and specific assumptions/conditions. So I’m not trying to say this isn’t a much needed contribution.
My reaction is more about framing. I worry that the “new theory” framing + all the new words may make the central intuition feel more novel or exotic than it is. Many people, including/especially many who would not identify as utilitarians or EAs, already have the intuition that the value of a world depends not only on total welfare, but also on the diversity, richness, or non-redundancy of the lives/experiences it contains — roughly, that additional near-duplicate lives have diminishing marginal value.
So I’d find it helpful to separate, as clearly as possible, the widely shared motivating intuition from the more specific Saturationist implementation. Otherwise I worry the jargon makes the view feel more alien or proprietary than it needs to be, when the underlying motivation may actually be quite intuitive to many people.
https://forum.effectivealtruism.org/posts/W4JksntnuFABZCkCw/funding-strategies-for-global-public-goods
^ I would read this to get some perspective on how to think about funding mechanisms and converting all the way to morally actionable outputs.
https://forum.effectivealtruism.org/posts/mopsmd3JELJRyTTty/ozzie-gooen-s-shortform?commentId=GHT2r3ubscoXPwfb3
https://www.longtermwiki.com/wiki/E411
https://ea-crux-project.vercel.app/ai-transition-model-views/graph
^ check out what ozzie is doing here (ea quick take + longtermism wiki + ea crux project).
https://forum.effectivealtruism.org/posts/zuQeTaqrjveSiSMYo/a-proposed-hierarchy-of-longtermist-concepts
^ good way to try to operationalize real world quantities into morally relevant long termism.
https://forum.effectivealtruism.org/s/y5n47MfgrKvTLE3pw
^ moral weights project to give baseline for moral circle sliders.
https://forum.effectivealtruism.org/posts/3hH9NRqzGam65mgPG/five-steps-for-quantifying-speculative-interventions
^ broad post on trying to do this for more speculative stuff.
but yea this is hard and goes exponential quickly
My take is we should start by really drilling down a db of charity financials, charity outputs (or intervention outputs and charities are composed of interventions), then pipes for converting between outputs into outcomes, with fungible philosophy pipes that take outcomes and pipe into rankings
ranking charities requires solving or assuming:Metaethics 📚
Population ethics 👥
Philosophy of mind 🧠
Decision theory 🎲
Astrobiology 👽
The nature of consciousness 💭
Whether distance matters (in space 🌌 AND time ⏳)
on top of all the boiler plate auditing.
As a (not super confident) non-believer in real-money prediction markets, curious if you have a best steelman link/post you like or if you want to tell me why you disagree with me (basically the exact 3 reasons you listed plus some more context)?
seems to me we could get a huge chunk of the benefits with play money markets + the existing binary event contracts the cme already allowed.
Well I’d guess first i’d just say I only ever thought much about this stuff in the context of Chicago, and even then just in my little slice of the world. I’m sure different places have different textures.
Your second paragraph doesn’t make that much sense to me. Do you think if meetup was free there would be way more events in your town? Seems unlikely to me. Don’t get me wrong, network effects are totally in play in these markets and can create natural monopolies which reduce supply (and then market quantity) from social optimum, but i think that if it was 0$ instead of 175 you might have like 10-30% more groups. But it seems to me supply could reasonably 5x if we had a healthy society.
“I also think you’re overemphasizing the need for group culture and leaders to be designed well, since I think this stuff just naturally arises in environments where typical people with shared interests come together.” Hmm yea my mind could be changed pretty easily, I’d just like to see the studies (or maybe there already are similar things done in psyche or soc, i haven’t looked much).
Re meetup and groups: You need to find a balance of inclusivity. Even though the purpose of a group is largely to socialize, the medium in which socializing takes place should be attractive in some sense orthogonally to how cool a person is. e.g. I like to go play basketball. I also socialize with the people at basketball. sometimes the people there are weird, it’s ok, i still like basketball and have a good time and come again. Vs. a group that’s basically just to socialize, the cool people get less out of it if the other people aren’t cool and so they leave and it spirals.
The reason I said “Developing online platforms that allow individuals to host in-person community events for free.” was because Meetup currently costs $175 a year and is the main platform in my city
If people really wanted to meetup more, there are tons of possible ways it could happen cheaper, both through existing competitors (facebook events/groups, reddit, listservs, 222, RA, and way more, trust me). While sorting and selecting the best events is not extremely easy, it’s not that hard. If you actually put a few hours asking around where events are posted and then 1-2 hours of effort scrolling through these things (local websites, instas) you can fill up your calendar with random stuff (although I will say I think a lot of young 20s people don’t realize that you could just ask a few store owners and librarian where the events are and you will learn a lot). It’s true we could drive the marginal cost down even further and this should help some but after thinking about this alot (and trying to get people to actually join things and or/post events) i’m not convinced this is the core of the problem. I think the core of the problem is more that we don’t have enough supply of actually good events and communities + increasingly entertaining other options that become hard to break habits so we might need some light paternal guidance in the right direction. Operationalizing this into a solution ends up looking closer to a religion than an app.
I would also be interested in seeing what some researchers could come up with, and I def think there is a lot of innovation on the side of thirdspace design and group norms/ activity setups that can improve social life, but OTOH so much of what makes a group is its people and leaders, and that’s not something that is easy to scale, repeating but scaling it seems more social movement/religion in nature than modern tech.
Hey James, very concerned with number 1 also. I actually spent a year building an events app (i gave up), it’s a very tough space. For one it’s not clear why people need more two sided market places (spontaneous event/hangouts apps are also considered one of the biggest “tarpits” for entrepreneurs, though i’d caution reading too deeply into that type of stuff). ATP partiful and lu.ma are solid enough in terms of e vites, for really small groups you use i message. then maybe you are gesturing more at meetup or pie but these apps suffer from weird sociological dynamics. You can not just get a bunch of lonely people together and have a good time exactly. The more you look into this stuff, the more it feels like an omni problem. It’s hard to say the tractability. There is a huge design space of things to improve on, you gesture at this. The problem spans addiction, habits, thirdspaces, culture, trust, etc. I don’t know if that’s a good or a bad thing in terms of being able to make headway.
I do believe ai might have some promise in terms of creating better events wikis (e.g. https://cguth7.github.io/events/ ).
Feel free to reach out whenever, I have much more to say on the topic.
more bang for your buck with soil nematodes
Our ancestors did not make this trade at all for the most part. Mostly they stayed hunter gathers, until the people who adopted farming out populated them and then expanded and killed/outcompeted them. (technically, I guess “our” ancestors are the ones who adopted the agriculture)
Very cool.
I work on text parsing / meta science and do a lot of stuff like this on the side and for my lab.
https://docgmedicalsummaries.com/rankings
I’ve done something similar for ranking clinical medicine articles, it’s pretty similar to your site but might be able to share some insights. (might comment more later regardless, just throwing this up for now so I remember).
edit: also signing up will auto subscribe you to emails just to note but should be easy to unsubscribe, can also see how we do rankings without signing up on the landing page.
What an insightful comment! Very well put I appreciate it.
YESSSSS didn’t think I would get to see some hoops on the forum, thanks.
But I would be remiss to not give credit to 2 other dudes who should get some love from EA, Morey and Harden! To be fair trying to assign exact credit for who spurred the 3 pt revolution is hard and I don’t claim to be confident, and also it is definitely true that curry helped accelerate the revolution, though I would probably put the curry warriors as the 2nd or 3rd most important group in doing so.
I think the Morey/Harden/rockets (and possibly seven seconds suns but will ignore for now) probably deserve more credit, although definitely curry/warriors if you mean who made the public think 3>2. (and I’m not claiming that the point of your post was to give curry all or the most credit, I just can’t help myself in filling in some more basketball history for those interested).
The thing about curry is he is the greatest to ever shoot it. You simply can’t acquire a curry. Also, my read of Steve Kerr is that he is honestly not that analytic pilled as a coach. Like he is certainly on the more forward thinking side but he’s not a math demon the way Morey was. He did have them running an incredible offensive scheme though don’t get me wrong, but it was highly artistic and free flowing.
Morey was kinda the one to realize that you really shouldn’t take midrange at all. This was the first domino in the revolution (although looking at the midrange chart above, seems like league had been slowly realizing that before him). You should never ever ever take a step in, which players often did. In fact, you should often take a step back even if you are open from the midrange (although the step back didn’t explode in popularity till later). And you should put your role players on the three point line in the corners/wings, not in the midrange (and similarly, you should acquire players who can hit those shotes).I also think while on first glance, it’s easy to think of the 3 pt revolution as completely analogous to something like the shift in baseball—basically pure math that would have been true at any point in the league—I think it’s probably at least a little less of a brain fart (though still mostly a brain fart) than it might initially seem. I think there was actually a series of (relatively simple) innovations that had to occur.
Just because league 3pt TS% > 2 pt TS% (in the halfcourt), this doesn’t mean that the marginal 3 pt is higher EV / TS than the marginal 2 pt. Now I happen to think that it still probably was (i.e. like a good shooter jacking up some contested 3 still better than replacement 2 from that team), but you have to figure out exactly how to generate those extra 3s. At first I think it’s obvious, just replace the middies with the threes. But then you have done picked all of this fruit, and now you have to figure out some more complicated ways to generate more.
Some (haters like myself) might argue this is where the warriors really came into play. The warriors abused moving screens harder than had ever been done in the history of the league, and in doing so, they were able to generate a few more clean looks a game. This definitely was very influential and you can see the proliferation today, with almost every screen set in the nba today being technically illegal (I hate to call this an innovation but...).
After everyone started abusing the moving screens, we needed even more innovations to generate new threes. Again I think here Harden and Morey shine, with the step back 3 revolution occuring around 2017-2018 by Harden.
Anyway I’m super pedantic and I don’t think this changes the implications of your post at all, just excited to write about basketball on the forum and wanted to add my 2 cents.
giving up ~0.00000000000000000000000000000000000001% of the lightcone is easily worth it on moral uncertainty grounds.
Agreed this seems prudent and plausible, but not so much so that I would feel confident that this would be the result of CEV ish stuff. Despite some of the technical hurdles mentioned involved with trying to meaningfully specify up front/value locking that we get to keep this solar system for us and the animals I feel like I could be convinced this is still the more likely path to end up in a good future for us (but not all sentient life throughout the lightcone).
But also, if the CEV of human values involves killing all humans, then doesn’t that kinda mean killing all humans is the correct thing to do?
yea but the correct thing (from a human CEV) to do isn’t equivalent to what is good for humans (and animals). I might be getting into button pushing semantics here.
I’m not sure why you think baking CEV into AI will result in a good future for animals (or humans), though if we are talking about “all sentient beings”, I guess I would say probably. It seems quite likely to me that if there is a “CEV attractor state” or similar, it involves killing us all—I don’t say this because I don’t love animals or humanity. I just don’t see how it could be remotely possible that we (earth evolved humans and animals) are efficient utility producers (by a wide range of definitions of “utility”). That being said, if CEV or similar is a real coherent concept, it almost certainly would prevent permanent torture/s-risks which would be nice.
but CEV is a fuzzy concept to me so might be misunderstanding (i’ve read the lw page and some other basic stuff and have a basic sense of stance deference and cosmopolitanism) .
Anthropic itself isn’t Molochian—The molochian-ness (idk if I’m using this term right I don’t read SSC) is that any time there is a disagreement in the community over some issue, and one side aligns more with the real worlds outer loops (money, status, intellectual sexiness), that side will naturally acquire more power within the movement because the movement does not have any way to counteract this other than persuasion which is increasingly difficult as the problems we deal with get more abstract and complex.
Yes I wrote it (with some help from Claude), glad you enjoyed it!
You are right your specific worry/content of the post is narrower, and generally I think you have approximately the right sense for what is going on and didn’t mean for the parable to be an exact fictional substitute for your post but just related. Also maybe I missed it but I think you forgot to mention the selection effect of those who ends up at anthropic itself, which is arguably bigger than the value drift inside of it—the tower was to some extent built by believers!. It’s always hard for me to write these comments because I feel as though I could write a 40 page book about the group dynamics in EA :P.
I really do love this community but I basically have given up on it (in terms of my views of the long run trajectory, i still love the people and read what they write religiously). I think it’s already too late and it’s been institutionally/culturally captured. I’m never certain of course but I think probably FTX was EAs last chance to put in real political/financial policies (some of which you mentioned/gestured at) that stop it from value drifting with outer status/money loops, and at this point it’s most likely a waste of time to try to fix it. I didn’t realize it at the time but this is how I feel looking back. I mean god damn we didn’t even clear house of the majority of the people directly implicated in the scandal! That would have been a bare minimum, I think.
The problems are real but increasingly my advice is: you are better of hopping to something like humanism and working on improving it if you want to see the solutions implemented. The forum and EA movement at large if you don’t live in a group house or for a prestigious EA org or have a bunch of money is basically when your older brother hands you an unplugged controller. I read the forum almost every day and have done so for years so one starts to pick up on some patterns. Since FTX I see a post like yours approx once a month (although more recently). They usually get between 20-50 upvotes so there is definitely some sort of coalition there but it’s small and basically never does someone powerful in the movement interact with these posts, you can decide if that’s a coincidence or not. And ultimately the posts always seem drift away with the wind. In this sense one can see how the movement would get accused of being paid opposition or something like that.
Personally, I will be hopping ship the first chance I get (i.e. as soon as another community has a close level of intellectual rigor without the horrible incentives and incoherent structure). And yes I will still call myself an effective altruist :), only if asked will I clarify the lowercaseness of that statement. (see I always write way too much—I’m working on it lol).
There was a question so simple that no honest person could refuse.
A child is drowning.Do you help?
From this, a city sprouted.
In the beginning there were no buildings. There was a leap.
And then a plunge.
Some cold wet socks.
And a coughing child firmly on solid earth.
Those who witnessed firsthand saw how vast and strong the river was, and how many more children they could not save. Word spread and one leap became many. Small structures began to rise along the great riverbank. As more came they brought new ideas.
One day, someone decided to start counting. If you mapped out the expected distribution of drownings, you could triage. One jump could save two. Or a smaller leap might go further than another requiring more bravery. This counting was not a betrayal of the original question. It was the question taken seriously.
From this the towers grew. In the towers the modelers worked and lived. At first the towers were short and adjacent to the riverbank. The modelers invented new tools—nets, boats, buoys, weather systems, river maps. These were real. They were the question taken seriously. With the help of those at the riverbanks children were rescued at rates never dreamt over.
Many of the people in the towers had been at the river once — stringing ropes, placing floats. They understood that an hour spent modeling could save more children than a year at the water. This was provably true.
The people who maintained the ropes and ladders were still respected. They were thanked at ceremonies.
Nonetheless the success of the towers spurred more towers. Each new tower asked a bigger question. What about the children far downstream? The river turned into a huge delta. The tools available would be much better deployed there than at the cities adjacent rapids. And so on.
And each answer to each question was bigger than the last, and at some point the answer was really big and the bigness was the point. But this was not a perversion of the question, it was the question taken seriously.
And so the towers shot into the sky. Big questions require big models and big tools and big solutions. The towers debated the hard questions. Honestly, rigorously, sometimes for years. People changed their minds. Studies were revised. The city prided itself on this — it was, in fact, better at updating than anywhere else. The debates were real. It was just that the city’s center of gravity never moved very far.
Some left. The city wished them well and did not study where they went.
To build these towers the modelers needed money. They recruited people of extreme wealth who were drawn by the very same question. These people were very generous and funded the towers, and the docks, and the nets, and the boats, and the medicine for the ear infections and anything else you could think of. This was all very real, and many lives were saved. The city had no elections, no recall votes, no formal process for anything. The billionaires simply funded the work, and the work followed the funding, and the funding followed the billionaires’ interests, and the billionaires’ interests followed from the models, which the billionaires had funded.One day one of these philanthropists made a bet. The bet was large, and it failed. The bet’s rationale at least had the appearance of being built on the machinery of the city. Not everyone thought the bet served the city’s purpose. But the reasoning was layered and the models were complex and it was genuinely hard to say whether the bet was a betrayal of the city’s logic or its fullest expression. The loss was large enough that programs closed and people at the river were called home. The city was shaken. The city had meetings about it. The city discussed accountability. The city discussed reform.
Some towers altered their appearance, and the riverbank looked different too, it had to after all, with the new lack of funds. But overall the city was still the same. To meaningfully change the city, the city would have to have decided that it was in fact a city, governed by interests. The city could not admit this. The core premise of the city was that the counting was not politics but math.
And so the city continued. In time the bet was old history and the children kept getting saved and the towers continued off into the sky.
The city has no architect. Nobody designed it. A thousand people made a thousand kind decisions and the decisions accumulated into a shape, and the shape made more of itself. There is no one to confront. There is nothing wrong with any single part of it.
I think we could use a documentary series where we just go follow around orgs or individual EAs for a couple days and see how they talk, live and act. It would be pretty cheap at the very least.
Matches my intuition, I think there aren’t that many experts and some of them already know how to make dangerous viruses and/or already have access to the labs. Practically speaking between 2027-2028 I’d assume the main uplift will be for people with like a bachelors in bio or chem and good at using frontier AI.
Also underrated: being able to quickly gather a list of biology experts and biology labs that work on dangerous stuff near you with a break down of how deadly/contagious each is. Don’t need to be an expert to rob a bank. Yesterday a friend who goes to Hopkins sent me a photo of a poster in front of a lab in the hallway that said “ZIKA VIRUS IS USED IN THIS LAB DO NOT PASS THROUGH AS A SHORTCUT”
It’s definitely interesting to think about the elegant edge case of a string less than ~50 words that can hack your brain. Probably have a similar intuition to you that it’s unlikely.
https://pmc.ncbi.nlm.nih.gov/articles/PMC11846175/ Just using this study to baseline, 22 year olds were averaging ~4.5 hours on screens per day. Let’s just guess half of that is on social media/entertainment stuff. Assuming 170 WPM on the videos (probably the real conversion is higher because videos are more info dense but w/e) − 170 * 135 = ~23,000 words per day.
In effect a social media company has ~2.5 orders of magnitude more string length to hack your brain per day and ~3.5 per week. I can already tell you there are multiple normal (not like so normal but not schitzo or drug addicts) 20-30 year old males in my life who have been brain hacked by the internet and gone full Hikikomori. To be fair it’s probably a lot easier to hack people into shutting down or becoming vegetative than being your slave, but depending on who you are and what you plan is just making 30% of the population vegetative might be quite useful.
How does social media work approximately (as far as I can tell)
- Uses input data (eyes, likes, time to scroll, etc. ) to embed you into entertainment space. Has some dynamic vector that describes what would entertain you the most.
- Then finds the nearest neighboring content.
Would LLMs improve this?
- can make the entertainment space less hollow, i.e. the distance between your entertainment vector and the nearest neighbor smaller. (seems plausible)
- can improve the algorithms that embed you/the content into the space (seems unlikely to see significant boost here, doesn’t seem to be that hard of a problem and we already have been working on this for a decade).
- can widen the dimensionality of the entertainment space (ai girlfriends).