I’m an AI Program Officer at Longview Philanthropy, though all views I express here are my own.
Before Longview I was a researcher at Open Philanthropy and a Charity Entrepreneurship incubatee.
Ben Stewart
Karma: 2,005
I don’t really want to focus on an individual’s career/financial decisions. I agree that in a just world, successful impact should be rewarded handsomely. But fwiw on the practical questions, I’m reasonably confident that:
Joey has influenced significantly more capital via AIM, Elevate, and other projects than he could have reasonably expected to donate via a for-profit career ex ante. Indeed, a key benchmark to compare impactful career options is a comparison to ‘earning to give’.
Joey is very thorough and systematic in his career choices, and I would guess he considered for-profit entrepreneurship and rated it lower EV (at least for him at that particular time).
Thanks for the substantial engagement! You cover a lot, so I’ll just pick out quotes and respond to ~most points.(As you note, I don’t work for CG and haven’t worked for them in over a year. I now work for Longview, but none of this is based on private information, and none of it represents views of either org).
it’s not obvious that the biggest corporations are the most impactful, like I wouldn’t allocate more philanthropic capital to NVIDIA or SpaceX
Even if they were impactful, why would you consider allocating any philanthropic capital to incredibly rich companies? The marginal case would be terrible.
allocation of credit between individual humans is much much better in forprofits—Jensen Huang or Elon Musk are much much richer than the median employee of their org
I think the right allocation of credit isn’t obviously better here. It is hard to figure out who were most responsible for an org’s success, in both for- and non-profits. CEOs get massive pay. There’s lots of reasons for that (high leverage, small market, rent-extraction via CEO’s influence on boards, equity-based pay, and so on). I don’t think you can assume that because CEO’s get enormous pay, they therefore deserve such pay (that may be true, but you’d want to actually try and investigate the credit allocation story—which is the question you started out with). I think there would be many cases of failed credit allocation in SpaceX, NVIDIA, and most other companies—i.e. many senior engineers or leaders who contributed much more value than the CEO, but were paid orders of magnitude less.
The pay is at least some signal of how the companies’ board values the executive’s performance. And if news of a CEO considering leaving or committing shifts investor sentiment and stock price, that is a good signal of how the market evaluates their performance. But these are merely okay instruments of assessing credit, and are usually strictly aligned with profit maximising. I don’t think they solve the credit allocation problem, and are a minor factor in comparing for- and non-profits.
Why should an impact-minded founder care? Because if they push really hard and actually succeed at doing something impactful, they then have to go and pray that the grant evaluators up above agree with their assessment.
I agree that for-profits reward extreme success much, much better than non-profits, especially given that extreme success may require taking personal risk and extreme levels of commitment. This is a problem that changes to the non-profit funding ecosystem should seek to fix.
Why should you, cG grantmaker shepherding the AIS ecosystem, care? Because without good credit allocation
I don’t agree that extreme CEO pay is a useful signal, so I don’t think it provides much useful information on credit allocation. It just shifts credit allocation from the grantmakers’ view to the view of a board—where such a board is significantly more constrained and incentivised away from impact optimisation.
Okay, but what happens to the founder managing a $100m/y charity? Will cG and 501c3 law allow them to draw a proportionally larger salary?
Higher salaries are possible, and may be probable if there is funding abundance. There are some restrictions around what is ‘reasonable’ executive pay in the non-profit sector—maybe this could be a limit. But what is reasonable should scale with the size of the charity. For example, the CEO of the Gates Foundation earns $1.7M/yr. Extremely high salaries in the for-profit world are significantly equity-based, and this is the harder hurdle. I guess a key question is whether you think non-profit founders should get compensation in excess of low-seven-figures. That might be reasonable, but it’s a weaker case than assuming they still get paid poorly.
There’s one kind of EA answer which is “who cares, I was going to donate it to charity anyways, so it’s equivalent to have this money be in my personal bank account vs in my charity vs in cG”. I find elements of that answer which are compelling and noble and good, but also elements which are suspect wrt ecosystem-level feedback effects. I think it’s important that successful individuals (and orgs) gain more ability to then influence what happens next in the ecosystem, and $ and equity are good numeric ways of building that feedback mechanism.
I don’t really get why very high salaries would lead to greater influence on the ecosystem. Is it by directing their personal capital? I think in an era of funding abundance such capital is unlikely to be a major factor. If it is by their capital signalling apparent excellence, I don’t think that would work. I trust people according to how well they think and argue, my perception of their character, and their track record. The amount of money they earn is irrelevant. There are countless people on very high salaries who I think are completely useless; there are countless people on very low salaires who I trust far, far more.
(also, seems good to nerdsnipe the non-fully-EA-aligned founders into working on problems we think are good)
Agreed. Though there’s a slight worry about how internalised their altruistic motivations are, with the risk of value drift over time and making the wrong high-stakes call when incentives clash. But in general, yes we should want to leverage this talent pool.
One thing that has made the forprofit tech scene work is that, when a founder exits, they have extra $ to invest, and also experience running new orgs
Yep agree that past success should translate into ability to influence future grantmaking. Regranting is a method of doing this. However, my impressions is that regranting has often struggled because it takes real time to do it well. The usual failure mode of regranting is that the regrantor doesn’t spend enough money. In general you kind of need it to be a part of their job to do it effectively (like 0.2 FTE or more if I had to guess). But those with past successes have high opportunity cost—you would need to think that the regranting is a better use of their time than their counterfactual efforts at continuing their successful track record. That will be only true in some cases.
To raise a recent example, our fundraising for our own projects (Surplus and Mox) has been annoyingly bogged down in classic traps of philanthropic funding norms, namely “funder chicken”. X asks “why isn’t Y funding you instead” and Y asks the same question in reverse and we’re waiting weeks to months to litigate. Grantmakers are already pressed for time and nobody has the chance to coordinate properly, and end up deferring too much.
Sorry that that’s been your experience, and yes these issues are still live. One reason for optimism is that if there is funding abundance, and funders are all opportunity-constrained, the reasons for worrying about funging and coordination become less important. Seems plausible that much better solutions are out there that should be pursued anyway, as you mention.
Tech forprofit venture funding is already operating at the 100x scale that AI safety philanthropic funding hopes to get to, and so I think copying their homework (where reasonable) is wise.
Yes, I agree there are many lessons and structures from VC land that should be of interest to funders. Two caveats:
VC-land also has many problems, which we’d want to avoid. Those problems are significantly worse in the context of chasing impact, because VCs basically only care about money, and bankruptcy caps downside risk. If you care about moral impact, you have to consider all the externalities, and you have to countenance extreme (indeed, existential) downside risk. It’s also harder to operationalise, since VCs can just use profits, but impacts needs very uncertain judgments or metrics that could be Goodharted on. My impression is that there have some attempts at impact metrics to guide capital allocation, without much success so far (though people should keep trying).
VCs move $4–8M per employee per year (Fable reckons for global VC there is $425B–513B deployed each year ÷ 60,000–100,000 employees). That’s only a little higher than than than the $-moved per employee at mainstream philanthropic orgs (typically $1M-$5M), and CG is about bang in the middle if we think that they do $1B in annual grants with about 170 employees ($5.9M/yr). So at least on these loose numbers, VC is not significantly more ‘efficient’ than grant making—maybe 2x compared to mainstream, and ~no difference compared to CG (and possibly other EA funders). VC just has orders of magnitude more employees.
My guess is that currently cG overall is trying to instead move down in size and get into earlier-stage funding and incubation.
I don’t think earlier stage funding and incubation is ‘moving down in size’. It’s what you would do if you wanted many more organisations to be able to receive money. Orgs can only grow so fast without other things breaking, so if all your good grantees are growing as fast as possible, moving earlier is the main way to scale the relevant ecosystem. Your other option is to give lots to already-big organisations that are outside the relevant ecosystem. Maybe in some cases that will make sense, but it doesn’t seem that promising a strategy.
outside cG’s core competency (eg have the people who are now in charge of those programs, previously founded successful orgs?)
Most VCs are not past founders. So if you like the VC approach, you shouldn’t worry about this. CG also has a great deal of experience in guiding the founding, early stages, and rapid growth of hundreds of organisations. I think that could actually be a more useful training ground than founding a single company, but they’re at least comparable.
I’m unsure whether it makes sense to house all of these initiatives under a single cG banner, vs have a somewhat more diversified ecosystem.
Agreed, a more diversified ecosystem would be good.
(All of this are my own views, and based on public information)
I agree with the main claim here—people should consider for-profit vehicles when they are exploring founding an org. And I agree with much of the content. But I wish the piece was more balanced, specifically: (1) adequatley noting the downsides of for-profit structures, and (2) updating the valuation of non-profit structures given a potential era of funding abundance.On (1):
There are always exceptions, so this is just a general principle: I think for-profits that are aiming for moral impact face a deep structural tension which is both powerful and hard to avoid . That is the problem of trying to aim for two things at once - impact and profits—which both have extreme distributions.
Digression/caveat: profits are power-lawed, but impact’s distribution is probably less extreme (probably because it lacks the revenue->investment positive feedback loop). That distribution could be power lawed, but across all causes/interventions—once you select for (A) important problems and (B) plausible interventions, maybe you’ve already put yourself up towards the tail anyway. Empirical analysis of this would be cool.
If both are extremely distributed, then you probably need to push hard to achieve that excellent outcome in either. And the tails probably come apart. The set of actions and sacrifices that would give you a chance at excellent profits will be different to those you would need to take for excellent impact. My general sense is that to achieve such great outcomes, a founder or org usually needs to make it an obsession, and the top priority. You cannot meaningfully have two separate and difficult aims be your obsessive top priority.
What if those aims are well correlated, so the tails don’t come apart? I’m sure there are cases like this, but I expect they are rare. In most cases I expect a successful org to do well at one of the aims and mediocre at the other.
If an org is good at profits but mediocre at impact, I don’t particularly care about it existing.
If an org is mediocre at profits but good at impact, I want it to exist. But if profits are mediocre, then it seems plausible that a non-profit structure would have been better anyway.
I expect the tails to be decently de-correlated here because profits are only a weak proxy for impact, as you note. Not only is profit determined by who can pay, but also by how much they can pay. Large powerful actors can pay a lot, but they are usually not the ones who need help.
This structural tension applies to impact investing, with little changed.
And to repeat, there are exceptions to this general principle. But it means your prior that a for-profit structure is the optimal one for a particular case should be significantly lower than if you haven’t considered this principle.
On (2):
There is likely to soon be a large volume of philanthropic capital, as you note. This means that many of the negatives of non-profit structures could be greatly reduced:
Grantmaking seems likely to become significantly faster and higher volume, making fundraising a much lower portion of a founders time.
Grantmaking seems likely to become able to fund more aggressive growth and larger scales.
You note “Historically, AI safety has relied on a small set of highly trusted grantmakers. But as funding in the space grows 100x, we don’t think we can 100x the number of human grantmakers.”
It seems weird to me for someone to countenance this fact and conclude that one should therefore try to be a for-profit instead, pursuing a very different kind of capital. Instead one should consider (and expect) that grant making could become much higher volume (even if the scale is below 100x), and that the near-term future may be the most accessible time in history for founding an ambitious non-profit.
I’m not trying to be unkind, and I apologise if I was. I’ll take this down if you ask here or via DM. I overreacted to what is a quick take because I think it was emblematic of a bad pattern—but that is unfair and disproportionate of me.
My main thing here is to push for better intermediate thinking. Like the standard EA/rat approach is so often based on dismissing mainstream or non-EA views, and then acting like their individual opinion is clearly superior, often reinventing current or past views that have had lots of non-EA examination. I want EA thinking to be better, and a lot of the time it would be improved by people reading more before opining, and not thinking the views of EA are so special.I think EAs if anything are far too epistemically modest and unwilling to stick their neck out for defending true and accurate positions.
We just have very different experiences then.
(this is the third time you’ve done this),
Do you mean critique someone on epistemic immodesty grounds? This is probably true but can you point me to the examples you have in mind? (I may indeed be doing this too much and seeing the examples would help)
Somewhat meta point on epistemic modesty, calling it out here because it is a pattern that has deeply frustrated me about EA/rationalism for as long as I have known them:
(making a quick take rather than commenting due to an app.operation_not_allowed error—I’m responding to @Linch’s quick take on war crimes)
I guess these are just EA/rationalist norms, but an approach that glosses major positions as being so quickly dismissible strikes me as insufficiently epistemically modest. I would expect such a treatment will fail to properly consider alternative answers or intuitions to the author’s own, especially the strongest versions of those answers (e.g. modern just war positions), won’t consider the most sophisticated counterpoints (e.g. your ‘oldest and clearest form’ gambit may just be bracketing out the counterexamples that don’t fit your definition, like genocide or sexual violence), and reinvent the wheel, e.g. the view seems to be exactly this from 2013:“A final rationale for the perfidy prohibition is to preserve the possibility of a return to peace. To prevent the degradation of trust and the bad faith between warring parties that would impede negotiation of peace terms. An effective perfidy prohibition preserves the good faith upon which ceasefires, armistices and conclusions of hostilities rely.”
I think deep engagement with the range of serious views on the topic is required to make your post “the best modern articulation of these ancient ideas”. I don’t think the quick take seems on a good track for that.
I think Henry’s skeptical that the AI safety community made a counterfactual difference in getting interpretability started earlier or growing faster. Not questioning interpretability’s prospects for reducing x-risk.
I think there’s a good case for AI safety having a pretty good counterfactual effect on a bunch of productive areas, but obviously that’s depends on a lot of details and there’s plenty of room for debate.
I think a stronger line of critique could be that early-mid AI safety efforts/thinking made the frontier race start earlier, go faster, and be more intense (e.g. roles in getting key frontier leaders obsessed, introducing Deepmind cofounders, boosting OpenAI’s founding, etc). I haven’t interrogated that history to know where to come down, but it’s a plausible way that the whole of AI safety has been net-negative. (This claim doesn’t really detract from future impact of AI safety though, if the cat’s out of the bag)
I was excited by ForecastBench and FutureEval both projecting that LLMs would reach superforecaster parity by June 2027. But I didn’t realise access to human crowd forecasts might be driving a lot of performance. If it is, that is massively disappointing.
The top LLM performers in ForecastBench have access to the crowd forecast (and it’s not clear to me if FutureEval hides crowd forecasts—Metaculus did for the Quarterly Cup in 2025 but I couldn’t find info about FutureEval). Skimming the literature with Claude, it seems like most studies either deliberately provide crowd forecasts or don’t prevent searching for it, and those that hide it tend to have significantly worse results (still interesting, but less exciting).
To me, the potential wonders of LLM superforecasting is being able to get excellent guesses at any questions I might come up with. If I need to already have a human crowd or market forecast for the guess to be any good, then the kind of LLM superforecasting being projected is about 10% as useful to me. I still expect ‘true’ parity eventually, but it becomes a story of general timelines rather than empirical projection.
I don’t know the field well, and I’m probably misunderstanding something. I’m posting this to find out I’m wrong. If I’m right, then it’s worth dampening the expectations of anyone else who was imagining having an instant team of supers at their beck-and-call in ~14 months time.
“very obviously their direct experience with thinking and working with existing AIs would be worth > $1M pa if evaluated anonymously based on understanding SOTA AIs, and likely >$10s M pa if they worked on capabilities.”
“Y&S endorsing some effort as good would likely have something between billions $ to tens of billions $ value.”
fwiw both of these claims strike me as close to nonsense, so I don’t think this is a helpful reaction.
Edit: I didn’t see how old this post was! It came up on my feed somehow and I’d assumed it was recent.
Thanks for this. I’ve only skimmed the report, and don’t have expertise in the area. So the below is lightly held.Section 4.2.3 talks about negative wellbeing effects. I think these are a serious downside risk, but other than noting that severe harms are indeed faced by guest workers, the report’s response is based on a single paper (Clemens 2018) and the idea that the intervention could improve things via surveys and ratings. Most of the benefit/harms considered in the rest of the report are financial (it appears on a skim).
I think the risk of facilitating severe harms against individuals (participating in the ‘repugnant transaction’) is very unsettling, and would be my main reason not to donate to such a charity. If I were a prospective donor I would want to see deeper exploration and red teaming of this worry.
I’d also note that this issue has characteristics that EA/AIM is likely to systematically underrate:
harms that are hard to quantify and compare against financial benefits, and
harms that may be wrong in a deontological sense to inflict, and not properly appreciated or respected by an all things considered cost-effectiveness analysis.
I thought this was great! Seems like a very accessible way to spread this info.
Two minor notes—when you perch you get a little stuck where you can’t move or unperch, I only got free by clicking socialize. And in the caged scenario, I shared the space with 2 other chickens, not 5-10 like the info says. Making the surroundings of the cage depicting other chickens would be more intense too, rather than the existing pattern.If you were you or others were going to extend it, I’d imagine gamifying it might be interesting. Eg you gain points by performing the natural behaviors, and points allow you to unlock the more elaborate natural behaviors. And then having some mechanic where you have to choose the deleterious behaviors (attacking other chickens, pulling own feathers). Maybe some stress bar that increases as a function of the space you have. Performing natural behaviors brings the stress down, and this is manageable in the kinder scenarios. But in cages, the stress is increasing rapidly and the natural behaviors aren’t available, so you’re forced to do the deleterious ones.)
The main benefit of doing this gamification would be to increase the chance people get interested, or that some streamer gives it a go.Nice work!
Thanks for this. Post-hoc theorizing:
‘doing good better’ calls to mind comparisons to the reader’s typical ideas about doing good. It implicitly criticizes those examples, which is a negative experience for the reader and could cause defensiveness.
‘Do the most good’ makes the reader attempt to imagine what that could be, which is a positive and interesting question, and doesn’t immediately challenge the reader’s typical ideas about doing good.
It wouldn’t have been obvious to me before the fact whether the above stuff wouldn’t be outweighed by worries about reactions to ‘the most good’ or what have you, so I appreciate you gathering empirical evidence here.
“Given leadership literature is rife with stories of rejected individuals going on to become great leaders”
The selection effect can be very misleading here — in that literature you usually don’t hear from all the individuals who were selected and failed, nor those who were rejected correctly and would have failed, and so on. Lots of advice from the start-up/business sector is super sus for this exact reason.
I would wait for METR’s actual evaluation — ’30 hours’ is just based on claims of continued effort, not actual successful performance on carefully measured tasks.
I think it’s possible to gain the efficiency of using LLM assistance without sacrificing style/tone — it just requires taste and more careful prompting/context, which seems worth it for a job ad. Maybe it works for their intended audience, but puts me off.
What can I read to understand the current and near-term state of drone warfare, especially (semi-)autonomous systems?
I’m looking for an overview of the developments in recent years, and what near-term systems are looking like. I’ve been enjoying Paul Scharre’s ‘Army of None’, but given it was published in 2018 it’s well behind the curve. Thanks!
I don’t know. My guess is that they give very slim odds to the Trump admin caring about carbon neutrality, and think that the benefit of including a mention in their submission to be close to zero (other than demonstrating resolve in their principles to others).
On the minus side, such a mention risks a reaction with significant cost to their AI safety/security asks. So overall, I can see them thinking that including a mention does not make sense for their strategy. I’m not endorsing that calculus, just conjecturing.
Object-level aside, I suspect they’re aware their audience is the hypersensitive-to-appearances Trump admin, and framing things accordingly. Even basic, common sense points regarding climate change could have a significant cost to the doc’s reception.
Thanks Carol! I think its reasonable to have these worries, and to be concerned that money doesn’t solve them (these orgs need to execute well, and more direct solutions will be needed in some cases). I think there’s reason for optimism here, but I don’t want to speak for any orgs or base things on private info.