Thank you, I appreciate that.
Benjamin_Todd
Karma: 9,094
Hey JWS,
These comments were off-hand and unconstructive, have been interpreted in ways I didn’t intend, and twitter isn’t the best venue for them, so I apologise for posting, and I’m going to delete them. My more considered takes are here. Hopefully I can write more in the future.
Thank you!
I’d be interested to hear a short explanation of why this seems like a different result from Leopold’s paper, especially the idea that it could be better to accelerate through the time of perils.
I should maybe have been more cautious—how messaging will pan out is really unpredictable.
However, the basic idea is that if you’re saying “X might be a big risk!” and then X turns out to be a damp squib, it looks like you cried wolf.
If there’s a big AI crash, I expect there will be a lot of people rubbing their hands saying “wow those doomers were so wrong about AI being a big deal! so silly to worry about that!”
That said, I agree if your messaging is just “let’s end AI!”, then there’s some circumstances under which you could look better after a crash e.g. especially if it looks like your efforts contributed to it, or it failed due to reasons you predicted / the things you were protesting about (e.g. accidents happening, causing it to get shut down).
However, if the AI crash is for unrelated reasons (e.g. the scaling laws stop working, it takes longer to commercialise than people hope), then I think the Pause AI people could also look silly – why did we bother slowing down the mundane utility we could get from LLMs if there’s no big risk?
Thanks that’s helpful!
I agree people often overlook that (and also future resources).
I think bio and climate change also have large cumulative resources.
But I see this as a significant reason in favour of AI safety, which has become less neglected on an annual basis recently, but is a very new field compared to the others.
Also a reason in favour of the post-TAI causes like digital sentience.
Or you might like to look into Christian’s grantmaking at Founders Pledge: https://80000hours.org/after-hours-podcast/episodes/christian-ruhl-nuclear-catastrophic-risks-philanthropy/
Thanks that’s helpful background!
I agree tractability of the space is the main counterargument, and MacArthur might have had good reasons to leave. Like I say in the post, I’d suggest people think about this issue carefully if you’re interested in giving to this area.
I don’t focus exclusively on philanthropic funding. I added these paragraphs to the post to clarify my position:
I agree that a full accounting of neglectedness should consider all resources going towards the cause (not just philanthropic ones), and that ‘preventing nuclear war’ more broadly receives significant attention from defence departments. However, even considering those resources, it still seems similarly neglected as biorisk.
And the amount of philanthropic funding still matters because certain important types of work in the space can only be funded by philanthropists (e.g. lobbying or other policy efforts you don’t want to originate within a certain national government).
I’d add that if if there’s almost no EA-inspired funding in a space, there’s likely to be some promising gaps by someone applying that mindset.
In general, it’s a useful approximation to think of neglectedness as a single number, but the ultimate goal is to find good grants, and to do that it’s also useful to break down neglectedness into different types of resources, and consider related heuristics (e.g. that there was a recent drop).
--
Causes vs. interventions more broadly is a big topic. The very short version is that I agree doing cost-effectiveness estimates of specific interventions is a useful input into cause selection. However, I also think the INT framework is very useful. One reason is it seems more robust. Another reason is that in many practical planning situations that involve accumulating expertise over years (e.g. choosing a career, building a large grantmaking programme) it seems better to focus on a broad cluster of related interventions.
E.g. you could do a cost-effectiveness estimate of corporate campaigns and determine ending factory farming is most cost-effective. But once you’ve spent 5 years building career capital in that factory farming, the available interventions or your calculations about them will likely very different.
It might take more than $1bn, but around that level, you could become a major funder of one of the causes like AI safety, so you’d already be getting significant benefits within a cause.
Agree you’d need to average 2x for the last point to work.
Though note the three pathways to impact—talent, intellectual diversity, OP gaps—are mostly independent, so you’d only need one of them to work.
Also agree in practice there would be some funging between the two, which would limit the differences, that’s a good point.
I’d also be interested in that. Maybe worth adding that the other grantmaker, Matthew, is younger. He graduated in 2015 so is probably under 32.
Intellectual diversity seems very important to figuring out the best grants in the long term.
If atm the community, has, say $20bn to allocate, you only need a 10% improvement to future decisions to be worth +$2bn.
Funder diversity also seems very important for community health, and therefore our ability to attract & retain talent. It’s not attractive to have your org & career depend on such a small group of decision-makers.
I might quantify the value of the talent pool around another $10bn, so again, you only need a ~10% increase here to be worth a billion, and over centralisation seems like one of the bigger problems.
The current situation also creates a single point of failure for the whole community.
Finally it still seems like OP has various kinds of institutional bottlenecks that mean they can’t obviously fund everything that would be ‘worth’ funding in abstract (and even moreso to do all the active grantmaking that would be worth doing). They also have PR constraints that might make some grants difficult. And it seems unrealistic to expect any single team (however good they are) not to have some blindspots.
$1bn is only 5% of the capital that OP has, so you’d only need to find a 1 grant for every 20 that OP makes that they’ve missed with only 2x the effectiveness of marginal OP grants in order to get 2x the value.
One background piece of context is that I think grants often vary by more than 10x in cost-effectiveness.
One quick point is divesting, while it would help a bit, wouldn’t obviously solve the problems I raise – AI safety advocates could still look like alarmists if there’s a crash, and other investments (especially including crypto) will likely fall at the same time, so the effect on the funding landscape could be similar.
With divestment more broadly, it seems like a difficult question.
I share the concerns about it being biasing and making AI safety advocates less credible, and feel pretty worried about this.
On the other side, if something like TAI starts to happen, then the index will go from 5% AI-companies to 50%+ AI companies. That’ll mean AI stocks will outperform the index by ~10x or more, while non-AI stocks will underperform by 2x or more.
So by holding the index, you’d be forgoing 90%+ of future returns (in the most high leverage scenarios), and being fully divested, giving up 95%+.
So the costs are really big (far far greater than divesting from oil companies).
Moreover, unless your p(doom) is very high, it’s plausible a lot of the value comes from what you could do in post-TAI worlds. AI alignment isn’t the only cause to consider.
On balance, it doesn’t seem like the negatives are so large as to reduce the value of your funds by 10x in TAI worlds. But I feel uneasy about it.
I want to be clear it’s not obvious to me OP is making a mistake. I’d lean towards guessing AI safety and GCBRs are still more pressing than nuclear security. OP also have capacity constraints (which make it e.g. less attractive to pursue smaller grants in areas they’re not already covering, since it uses up time that could have been used to make even larger grants elsewhere). Seems like a good fit for some medium-sized donors who want to specialise in this area.
Interesting. I guess a key question is whether another wave of capabilities (e.g. gpt-5, agent models) comes in soon or not.
Agree it’s most likely already in the price.
Though I’d stand behind the idea that markets are least efficient when it comes to big booms and busts involving large asset classes (in contrast to relative pricing within a liquid asset class), which makes me less inclined to simply accept market prices in these cases.
You could look for investments that do neutral-to-well in a TAI world, but have low-to-negative correlation to AI stocks in the short term. That could reduce overall portfolio risk but without worsening returns if AI does well.
This seems quite hard, but the best ideas I’ve seen so far are:
The cluster of resources companies, electricity producers, commodities, land. There’s reason to think these could do quite well during a TAI transition, but in the short term they do well when inflation rises, which tends to be bad for AI stocks. (And they were effective hedges in the most recent draw down and in 2022.) Some of them also look quite cheap at the moment. However, in a recession, they will fall at the same time as AI stocks.
Short long-dated government bonds or AI company credit. In the short term helps to hedge out the interest rate and inflation exposure in AI companies, and should also do well in the long term if an AI boom increases interest rates. Credit spreads are narrow so you’re not paying much for the hedge. However, if there’s a recession, these will also do badly.
Index shorts (especially focused on old economy stocks). This could reduce overall market risk, and AI stocks will most likely fall at the same time as other stocks. If you buy long dated put options there’s some reason to think AI will increase volatility, so you might also benefit a little there. However, on net it might be desirable to have high market exposure / this trade most likely loses money.
Long-short multi-asset trend-following. This is an active strategy (so you might be skeptical that it works) but tends to do well during macro regime changes / big market crashes / high volatility, which will likely be times when AI stocks are doing badly. But for the same reasons it could also do well during an AI boom.
However, all of these have important downsides and someone would need to put billions of dollars behind them to have much impact on the overall portfolio.
(Also this is not investment advice and these ideas are likely to lose a lot of money in many scenarios.)
Thanks for the analysis! I think it makes sense to me, but I’m wondering if you’ve missed an important parameter: diminishing returns to resources.
If there are 100 community members they can take the 100 most impactful opportunities (e.g. writing DGB, publicising that AI safety is even a thing), while if there are 1000 people, they will need to expand into opportunities 101-1000, which will probably be lower impact than the first 100 (e.g. becoming the 50th person working on AI safety).
I’d guess a 10x increase to labour or funding working on EA things (even setting aside coordination and reputation issues) only increases impact by ~3x.
It seems like that might make significant difference to the model—if I’ve understood, currently the impact of marginal members in the model is actually increasing due coordination benefits, whereas this could mean it’s decreasing.
I’d still guess marginal growth is net positive, but I feel less confident than the post suggests.