Response to Kat, intended as a devil’s advocate stance:
As Tyler said, funders can already query other funders regarding projects they think might have been rejected. I think the unilateralist’s curse argument holds if the funding platform has at least one risk-averse big spender. I’m particularly scared about random entrepreneurs with typical entrepreneurial risk tolerance entering this space and throwing money at projects without concern for downsides, not about e.g. Open Phil + LTFF + Longview accessing a central database (though such a database should be administered by those orgs, probably).
I’m very open to hearing solutions to the risk of a miscommunication-induced unilateralist’s curse. I think a better solution than a centralized funding database would be a centralized query database, where any risk-averse funders can submit a request for information to a trusted third party, who knows every proposal that was rejected by all parties and can connect the prospective funders with the funders who rejected the proposal for more information. This reduces the chances that potentially risk-tolerant funders get pinged with every grant proposal but increases the chances that risk-averse funders request information that might help them reject too-risky proposals. I know it’s complicated, but it seems like a much better mechanism design if one is risk-averse.
It seems pretty unlikely that small projects will get noticed or critiqued on the EA Forum, but low-quality small projects might be bad en masse. Future Fund gave a lot of money to projects and people that got low visibility but might have contributed to “mass movement building concerns” around “diluted epistemics”, “counterfactually driving the wheel of AI hype and progress,” and “burning bridges for effective outreach.”
Open-sourcing funding analysis is a trade-off between false positives and false negatives for downside risk. Currently, I’m much more convinced that Open Phil and other funders are catching the large downside projects than an open source model would avoid these. False alarms seem safer than downside risk to me too, but this might be because I have a particularly low opinion of entrepreneurial risk tolerance and feel particularly concerned about “doing movement building right” (happy to discuss MATS’ role in this, btw).
A few key background claims:
AI safety is hard to do right, even for the experts. Doing it wrong but looking successful at it just makes AI products more marketable but doesn’t avert AGI tail-risks (the scary ones).
The market doesn’t solve AI risk by default and probably makes it worse, even if composed of (ineffectively) altruistic entrepreneurs. Silicon Valley’s optimism bias can be antithetical to a “security mindset.” “Deploy MVP + iterate” fails if we have to get it right first on the first real try. Market forces cannot distinguish between AI “saints” and “sycophants” unaided.
Big AI x-risk funders are generally anchoring on the “sign-value” of impact rather than “penny-pinching” when rejecting projects. Projects might sometimes get submaximal funding because the risk/benefit ratio increases with scale.
Agree with your background claims. But think we should be pivoting toward advocacy for slowing down / pausing / shutting down AI capabilities in general, in the post GPT-4+AgentGPT era. Short timelines means we should lower the bar for funding, and not worry quite so much about downside risks (especially if we only have months to get a moratorium in place).
I think that one’s level of risk aversion in grantmaking should depend on the upside and the downside risk of grantees’ action space. I see a potentially high upside to AI safety standards or compute governance projects that are specific, achievable, and verifiable and are rigorously determined by AI safety and policy experts. I see a potentially high downside to low-context and high-bandwidth efforts to slow down AI development that are unspecific, unachievable, or unverifiable and generate controversy or opposition that could negatively affect later, better efforts.
One might say, “If the default is pretty bad, surely there are more ways to improve the world than harm it, and we should fund a broad swathe of projects!” I think that the current projects to determine specific, achievable, and verifiable safety standards and compute governance levers are actually on track to be quite good, and we have a lot to lose through high-bandwith, low-context campaigns.
How much later are these efforts happening? I feel like EA leadership is asleep at the wheel here, and the EA community is not cut out for the emergency response we need in terms of how fast it can react (judging by the reaction so far).
Copying over the Facebook comments I just made.
Response to Kat, intended as a devil’s advocate stance:
As Tyler said, funders can already query other funders regarding projects they think might have been rejected. I think the unilateralist’s curse argument holds if the funding platform has at least one risk-averse big spender. I’m particularly scared about random entrepreneurs with typical entrepreneurial risk tolerance entering this space and throwing money at projects without concern for downsides, not about e.g. Open Phil + LTFF + Longview accessing a central database (though such a database should be administered by those orgs, probably).
I’m very open to hearing solutions to the risk of a miscommunication-induced unilateralist’s curse. I think a better solution than a centralized funding database would be a centralized query database, where any risk-averse funders can submit a request for information to a trusted third party, who knows every proposal that was rejected by all parties and can connect the prospective funders with the funders who rejected the proposal for more information. This reduces the chances that potentially risk-tolerant funders get pinged with every grant proposal but increases the chances that risk-averse funders request information that might help them reject too-risky proposals. I know it’s complicated, but it seems like a much better mechanism design if one is risk-averse.
It seems pretty unlikely that small projects will get noticed or critiqued on the EA Forum, but low-quality small projects might be bad en masse. Future Fund gave a lot of money to projects and people that got low visibility but might have contributed to “mass movement building concerns” around “diluted epistemics”, “counterfactually driving the wheel of AI hype and progress,” and “burning bridges for effective outreach.”
Open-sourcing funding analysis is a trade-off between false positives and false negatives for downside risk. Currently, I’m much more convinced that Open Phil and other funders are catching the large downside projects than an open source model would avoid these. False alarms seem safer than downside risk to me too, but this might be because I have a particularly low opinion of entrepreneurial risk tolerance and feel particularly concerned about “doing movement building right” (happy to discuss MATS’ role in this, btw).
A few key background claims:
AI safety is hard to do right, even for the experts. Doing it wrong but looking successful at it just makes AI products more marketable but doesn’t avert AGI tail-risks (the scary ones).
The market doesn’t solve AI risk by default and probably makes it worse, even if composed of (ineffectively) altruistic entrepreneurs. Silicon Valley’s optimism bias can be antithetical to a “security mindset.” “Deploy MVP + iterate” fails if we have to get it right first on the first real try. Market forces cannot distinguish between AI “saints” and “sycophants” unaided.
Big AI x-risk funders are generally anchoring on the “sign-value” of impact rather than “penny-pinching” when rejecting projects. Projects might sometimes get submaximal funding because the risk/benefit ratio increases with scale.
Agree with your background claims. But think we should be pivoting toward advocacy for slowing down / pausing / shutting down AI capabilities in general, in the post GPT-4+AgentGPT era. Short timelines means we should lower the bar for funding, and not worry quite so much about downside risks (especially if we only have months to get a moratorium in place).
I think that one’s level of risk aversion in grantmaking should depend on the upside and the downside risk of grantees’ action space. I see a potentially high upside to AI safety standards or compute governance projects that are specific, achievable, and verifiable and are rigorously determined by AI safety and policy experts. I see a potentially high downside to low-context and high-bandwidth efforts to slow down AI development that are unspecific, unachievable, or unverifiable and generate controversy or opposition that could negatively affect later, better efforts.
One might say, “If the default is pretty bad, surely there are more ways to improve the world than harm it, and we should fund a broad swathe of projects!” I think that the current projects to determine specific, achievable, and verifiable safety standards and compute governance levers are actually on track to be quite good, and we have a lot to lose through high-bandwith, low-context campaigns.
How much later are these efforts happening? I feel like EA leadership is asleep at the wheel here, and the EA community is not cut out for the emergency response we need in terms of how fast it can react (judging by the reaction so far).