While I think it’s valuable to share thoughts about the value of different types of work candidly, I am very appreciative of both people working on forecasting projects and grantmakers in the space for their work trying to make the world a better place (and am friendly with many of them). As I maybe should have made more obvious, I am myself affiliated with Samotsvety Forecasting, and Sage which has done several forecasting projects. And I’m also doing AI forecasting research atm, though not the type that would be covered under the grantmaking program.
I’m not trying to claim with significant confidence that this program shouldn’t exist. I am trying to share my current views on the value of previous forecasting grants and the areas that seem most promising to me going forward. I’m also open to changing my mind on lots of this!
Thoughts on some of your bullet points:
2. I think that for further funding in this field to be exciting, funders should really work on designing/developing this field to emphasize the very best parts. The current median doesn’t seem great to me, but I think the potential has promise, and think that smart funding can really triple-down on the good stuff. I think it’s sort of unfair to compare forecasting funding (2024) to AI Safety funding (2024), as the latter has had much more time to become mature. This includes having better ideas for impact and attracting better people. I think that if funders just “funded the median projects”, then I’d expect the field to wind up in a similar place to it is now—but if funders can really optimize, then I’d expect them to be taking a decent-EV risk. (Decent chance of failure, but some chance at us having a much more exciting field in 3-10 years).
I was trying to compare previous OP forecasting funding to previous AI Safety. It’s not clear to me how different these were; sure, OP didn’t have a forecasting program but AI safety was also very short-staffed. And re: the field maturing idk Tetlock has been doing work on this for a long time, my impression is that AI safety also had very little effort going into it until like mid-late 2010s. I agree that funding of potentially promising exploratory approaches is good though.
3. I’d prefer funders focus on “increasing wisdom and intelligence” or “epistemic infrastructure” than on “forecasting specifically”. I think that the focus on forecasting is over-limiting. That said, I could see an argument to starting from a forecasting angle, as other interventions in “wisdom and intelligence / epistemic infrastructure” are more speculative.
Seems reasonable. I did like that post!
4. If I were deploying $50M here, I’d probably start out by heavily prioritizing prioritization work itself—work to better understand this area and what is exciting within it. (I explain more of this in the wisdom/intelligence post above). I generally think that there’s been way too little good investigation and prioritization work in this area.
Perhaps, but I think you gain a ton of info from actually trying to do stuff and iterating. I think prioritization work can sometimes seem more intuitively great than it ends up being, relative to the iteration strategy.
6. I’d like to flag that I think that Metaculus/Manifold/Samotsvety/etc forecasting has been valuable for EA decision-making. I’d hate to give this up or de-prioritize this sort of strategy.
I would love for this to be true! Am open to changing mind based on a compelling analysis.
7. I don’t particularly trust EA decision-making right now. It’s not that I think I could personally do better, but rather that we are making decisions about really big things, and I think we have a lot of reason for humility. When choosing between “trying to better figure out how to think and what to do” vs. “trying to maximize the global intervention that we currently think is highest-EV,” I’m nervous about us ignoring the former and going all-in on the latter. That said, some of the crux might be that I’m less certain about our current marginal AI Safety interventions than I think Eli is.
There might be some difference in perceptions of the direct EV of marginal AI Safety interventions. There might also be differences in beliefs in the value of (a) prioritization research vs. (b) trying things out and iterating, as described above (perhaps we disagree on absolute value of both (a) and (b)).
8. Personally, around forecasting, I’m most excited about ambitious, software-heavy proposals. I imagine that AI will be a major part of any compelling story here.
Seems reasonable, though I’d guess we have different views on which ambitious AI-related software-heavy projects.
9. I’d also quickly flag that around AI Safety—I agree that in some ways AI safety is very promising right now. There seems to have been a ton of great talent brought in recently, so there are some excellent people (at very least) to give funding to. I think it’s very unfortunate how small the technical AI safety grantmaking team is at OP. Personally I’d hope that this team could quickly get to 5-30 full time equivalents. However, I don’t think this needs to come at the expense of (much) forecasting/epistemics grantmaking capacity.
I think you might be understating how fungible OpenPhil’s efforts are between AI safety (particularly governance team) and forecasting. Happy to chat in DM if you disagree. Otherwise reasonable point, though you’d ofc still have to do the math to make sure the forecasting program is worth it.
(edit: actually maybe the disagreement is still in the relative value of the work, depending on what you mean by “much” grantmaking capacity)
10. I think you can think of a lot of “EA epistemic/evaluation/forecasting work” as “internal tools/research for EA”. As such, I’d expect that it could make a lot of sense for us to allocate ~5-30% of our resources to it. Maybe 20% of that would be on the “R&D” to this part—perhaps more if you think this part is unusually exciting due to AI advancements. I personally am very interested in this latter part, but recognize it’s a fraction of a fraction of the full EA resources.
Seems unclear what should count as internal research for EA, e.g. are you counting OP worldview investigation team / AI strategy research in general? And re: AI advancements, it both improves the promise of AI for forecasting/epistemics work but also shortens timelines which points toward direct AI safety technical/gov work.
First, again, overall, I think we generally agree on most of this stuff.
Perhaps, but I think you gain a ton of info from actually trying to do stuff and iterating. I think prioritization work can sometimes seem more intuitively great than it ends up being, relative to the iteration strategy.
I agree to an extent. But I think there are some very profound prioritization questions that haven’t been researched much, and that I don’t expect us to gain much insight from by experimentation in the next few years. I’d still like us to do experimentation (If I were in charge of a $50Mil fund, I’d start spending it soon, just not as quickly as I would otherwise). For example:
How promising is it to improve the wisdom/intelligence of EAs vs. others?
How promising are brain-computer-interfaces vs. rationality training vs. forecasting?
What is a good strategy to encourage epistemic-helping AI, where philanthropists could have the most impact?
What kinds of benefits can we generically expect from forecasting/epistemics? How much should we aim for EAs to spend here?
I would love for this to be true! Am open to changing mind based on a compelling analysis.
We might be disagreeing a bit on what the bar for “valuable for EA decision-making” is. I see a lot of forecasting like accounting—it rarely leads to a clear and large decision, but it’s good to do, and steers organizations in better directions. I personally rely heavily on prediction markets for key understandings of EA topics, and see that people like Scott Alexander and Zvi seem to. I know less about the inner workings of OP, but the fact that they continue to pay for predictions that are very much for their questions seems like a sign. All that said, I think that ~95%+ of Manifold and a lot of Metaculus is not useful at all.
I think you might be understating how fungible OpenPhil’s efforts are between AI safety (particularly governance team) and forecasting
I’m not sure how much to focus on OP’s narrow choices here. I found it surprising that Javier went from governance to forecasting, and that previously it was the (very small) governance team that did forecasting. It’s possible that if I evaluated the situation, and had control of the situation, I’d recommend that OP moved marginal resources to governance from forecasting. But I’m a lot less interested in this question than I am, “is forecasting competitive with some EA activities, and how can we do it well?”
Seems unclear what should count as internal research for EA, e.g. are you counting OP worldview diversification team / AI strategy research in general?
Thanks Ozzie for sharing your thoughts!
A few things I want to clarify up front:
While I think it’s valuable to share thoughts about the value of different types of work candidly, I am very appreciative of both people working on forecasting projects and grantmakers in the space for their work trying to make the world a better place (and am friendly with many of them). As I maybe should have made more obvious, I am myself affiliated with Samotsvety Forecasting, and Sage which has done several forecasting projects. And I’m also doing AI forecasting research atm, though not the type that would be covered under the grantmaking program.
I’m not trying to claim with significant confidence that this program shouldn’t exist. I am trying to share my current views on the value of previous forecasting grants and the areas that seem most promising to me going forward. I’m also open to changing my mind on lots of this!
Thoughts on some of your bullet points:
I was trying to compare previous OP forecasting funding to previous AI Safety. It’s not clear to me how different these were; sure, OP didn’t have a forecasting program but AI safety was also very short-staffed. And re: the field maturing idk Tetlock has been doing work on this for a long time, my impression is that AI safety also had very little effort going into it until like mid-late 2010s. I agree that funding of potentially promising exploratory approaches is good though.
Seems reasonable. I did like that post!
Perhaps, but I think you gain a ton of info from actually trying to do stuff and iterating. I think prioritization work can sometimes seem more intuitively great than it ends up being, relative to the iteration strategy.
I would love for this to be true! Am open to changing mind based on a compelling analysis.
There might be some difference in perceptions of the direct EV of marginal AI Safety interventions. There might also be differences in beliefs in the value of (a) prioritization research vs. (b) trying things out and iterating, as described above (perhaps we disagree on absolute value of both (a) and (b)).
Seems reasonable, though I’d guess we have different views on which ambitious AI-related software-heavy projects.
I think you might be understating how fungible OpenPhil’s efforts are between AI safety (particularly governance team) and forecasting. Happy to chat in DM if you disagree. Otherwise reasonable point, though you’d ofc still have to do the math to make sure the forecasting program is worth it.
(edit: actually maybe the disagreement is still in the relative value of the work, depending on what you mean by “much” grantmaking capacity)
Seems unclear what should count as internal research for EA, e.g. are you counting OP worldview investigation team / AI strategy research in general? And re: AI advancements, it both improves the promise of AI for forecasting/epistemics work but also shortens timelines which points toward direct AI safety technical/gov work.
Thanks for the replies! Some quick responses.
First, again, overall, I think we generally agree on most of this stuff.
I agree to an extent. But I think there are some very profound prioritization questions that haven’t been researched much, and that I don’t expect us to gain much insight from by experimentation in the next few years. I’d still like us to do experimentation (If I were in charge of a $50Mil fund, I’d start spending it soon, just not as quickly as I would otherwise). For example:
How promising is it to improve the wisdom/intelligence of EAs vs. others?
How promising are brain-computer-interfaces vs. rationality training vs. forecasting?
What is a good strategy to encourage epistemic-helping AI, where philanthropists could have the most impact?
What kinds of benefits can we generically expect from forecasting/epistemics? How much should we aim for EAs to spend here?
We might be disagreeing a bit on what the bar for “valuable for EA decision-making” is. I see a lot of forecasting like accounting—it rarely leads to a clear and large decision, but it’s good to do, and steers organizations in better directions. I personally rely heavily on prediction markets for key understandings of EA topics, and see that people like Scott Alexander and Zvi seem to. I know less about the inner workings of OP, but the fact that they continue to pay for predictions that are very much for their questions seems like a sign. All that said, I think that ~95%+ of Manifold and a lot of Metaculus is not useful at all.
I’m not sure how much to focus on OP’s narrow choices here. I found it surprising that Javier went from governance to forecasting, and that previously it was the (very small) governance team that did forecasting. It’s possible that if I evaluated the situation, and had control of the situation, I’d recommend that OP moved marginal resources to governance from forecasting. But I’m a lot less interested in this question than I am, “is forecasting competitive with some EA activities, and how can we do it well?”
Yep, I’d count these.