I think this post significantly overstates its conclusion and is plausibly poorly calibrated on the relative value of forecasting.
My main “directional” issues with the post as it’s currently written:
I think it overstates the amount of funding devoted to forecasting on a “worldview” basis.
Most forecasting funding is (iiuc) not going to neartermist causes or particularly fungible with neartermist causes, so pointing to a bunch of neartermist causes to justify better funding options seems irrelevant.
From my perspective, it seems like:
Within Animal welfare fungible money, very little goes into forecasting e.g. less than $2M per year
Tbh—I would probably prefer that more money went into some kinds of forecasting on the margin. For example, I think that people are generally too bullish on clean meat, and Linch/Open Phil’s work investigating the difficulty of clean meat has plausibly resulted in better allocation of millions of dollars because there are, in fact, good alternatives (like cage-free campaigns).
Within Longtermist/AI fungible money, maybe $10M/year goes into forecasting, which seems pretty reasonable to me but i think to get to 10M you need to be including projects that seems very promising to me for different reasons to mainstream forecasting infrastructure e.g. AI 2027, METR.
I think the strongest version of the argument would be attacking AI evals but I’m unsure about whether those are in-scope for this post—my impression is that evals are useful for forecasting capabilities are a pretty great bet relative to other funding opportunities within the AI space.
So the argument actually seems to be “longtermist funding is not as cost-effective as neartermist funding” which is not totally unreasonable, but clearly needs to engage with the long/neartermist worldview (e.g. moral size of the future) as opposed to just engaging with tangible short-term impact indicators.
I’m less convinced than the OP that funders in particular are overrating forecasting—I just don’t see much effort going into forecasting grantmaking compared to ~every other grantmaking area.
My impression is that a lot of forecasting dollars are funded by organisations that are incentivised to use the money well (e.g. AI companies paying FRI to produce forecasts around safety and capability evaluations for safety planning). I see that others have weighed in on this already so not planning to elaborate on this more.
I agree with some of the post’s vibes and think it’s pointing at real cultural traits of rationalist communities. Though tbh, I think OP is too bearish on the usefulness of betting/making falsifiable predictions for people in EA-spaces. I suspect that OP seen lots of people getting very distracted by futarchy/manifold etc. (and I do think this is a risk), but culturally I think EA should be pretty into “betting/making falsifiable predictions” and that cluster of epistemic traits AND I think forecasting infrastructure has a meaningful effect on this. E.g In two office spaces (out of three that I’ve spent substantial time in), I think Manifold/prediction markets have very clearly made the communities more forecasting-y, and this has had tangible effects on people’s research/choice of projects—this is probably the most explicit example, though most changes are harder to hyperlink.
I think far more than $10M/year is going into forecasting. Many grants for forecasting are awarded outside the forecasting fund, such as the Navigating Transformative AI Fund. It depends on what you count, but I think it is closer to $25M/year.
I really question if people are really getting much, if anything, from all these forecasts that they didn’t already have before.
I think this post significantly overstates its conclusion and is plausibly poorly calibrated on the relative value of forecasting.
My main “directional” issues with the post as it’s currently written:
I think it overstates the amount of funding devoted to forecasting on a “worldview” basis.
Most forecasting funding is (iiuc) not going to neartermist causes or particularly fungible with neartermist causes, so pointing to a bunch of neartermist causes to justify better funding options seems irrelevant.
From my perspective, it seems like:
Within Animal welfare fungible money, very little goes into forecasting e.g. less than $2M per year
Tbh—I would probably prefer that more money went into some kinds of forecasting on the margin. For example, I think that people are generally too bullish on clean meat, and Linch/Open Phil’s work investigating the difficulty of clean meat has plausibly resulted in better allocation of millions of dollars because there are, in fact, good alternatives (like cage-free campaigns).
Within Longtermist/AI fungible money, maybe $10M/year goes into forecasting, which seems pretty reasonable to me but i think to get to 10M you need to be including projects that seems very promising to me for different reasons to mainstream forecasting infrastructure e.g. AI 2027, METR.
I think the strongest version of the argument would be attacking AI evals but I’m unsure about whether those are in-scope for this post—my impression is that evals are useful for forecasting capabilities are a pretty great bet relative to other funding opportunities within the AI space.
So the argument actually seems to be “longtermist funding is not as cost-effective as neartermist funding” which is not totally unreasonable, but clearly needs to engage with the long/neartermist worldview (e.g. moral size of the future) as opposed to just engaging with tangible short-term impact indicators.
I’m less convinced than the OP that funders in particular are overrating forecasting—I just don’t see much effort going into forecasting grantmaking compared to ~every other grantmaking area.
My impression is that a lot of forecasting dollars are funded by organisations that are incentivised to use the money well (e.g. AI companies paying FRI to produce forecasts around safety and capability evaluations for safety planning). I see that others have weighed in on this already so not planning to elaborate on this more.
I agree with some of the post’s vibes and think it’s pointing at real cultural traits of rationalist communities. Though tbh, I think OP is too bearish on the usefulness of betting/making falsifiable predictions for people in EA-spaces. I suspect that OP seen lots of people getting very distracted by futarchy/manifold etc. (and I do think this is a risk), but culturally I think EA should be pretty into “betting/making falsifiable predictions” and that cluster of epistemic traits AND I think forecasting infrastructure has a meaningful effect on this. E.g In two office spaces (out of three that I’ve spent substantial time in), I think Manifold/prediction markets have very clearly made the communities more forecasting-y, and this has had tangible effects on people’s research/choice of projects—this is probably the most explicit example, though most changes are harder to hyperlink.
(Caveat: Slightly self-promoting, sorry, but I hope it’s germane/helpful.) By the way, on the animal welfare forecasting front, see Support Metaculus’ First Animal-Focused Forecasting Tournament and Rethinking the Future of Cultured Meat: An Unjournal Evaluation. I’d leave room for some doubt as to whether the “clean meat forecasting” work led to updates in the right direction.
We’re trying to take the next steps on this with a workshop involving some belief elicitation and forecasting (workshop page, belief elicitation page).
I think far more than $10M/year is going into forecasting. Many grants for forecasting are awarded outside the forecasting fund, such as the Navigating Transformative AI Fund. It depends on what you count, but I think it is closer to $25M/year.
I really question if people are really getting much, if anything, from all these forecasts that they didn’t already have before.