How high quality do you think the grants the NSF will make would be?
Right now there is a very large amount of EA money available for AI safety research, between at least four different major groups. Each makes use of well connected domain experts to solicit and evaluate grants, and have application processes designed to be easy for applicants to fill in and then distribute the funds in a rapid fashion. Awards can be predicted sufficiently confidently that it is possible to create a career from a single one of these funders. However, the pool of good applicants is extremely small—all the organisations have struggled to spend this money effectively, and actively look for new ways of spending money faster.
In contrast I would be pessimistic about the quality of potential NSF grants. My concern is that, while we might be able to influence them to fund something called ‘AI safety’, it would not actually be that closely related to the thing we care about. Chosen grant reviewers could have many prestigious qualifications but lack a deep understanding of the problem. NSF grants can, I understand, take a long time to apply to, and the evaluation is also very slow—taking over six months. Even then success is not assured. So it’s possible a lot of high quality safety researchers would prefer dedicated EA funding anyway. Who does this leave getting NSF grants? I worry it would be largely existing academics in related fields who are able to reframe their research as corresponding to ‘safety’, thereby diluting the field, without significantly contributing to the project—similar to ‘AI Ethics’.
I do agree that NSF funding could help significantly raise the prestige of the field, but I would want to be confident in how high fidelity the idea transmission to grantmakers would be before promoting the idea, and I’m not sure how we can ensure this degree of alignment.
I agree that we need to care with high fidelity idea transmission, and there is some risk of diluting the field. But I think the reasonable chance of this spurring some more good research in AI safety is worth it, even if there will also be some wasted money.
One thing that’s interesting in the RfI is that it links to something called THE NATIONAL ARTIFICIAL INTELLIGENCE RESEARCH AND DEVELOPMENT STRATEGIC PLAN: 2019 UPDATE. This PDF outlines a federal committee’s strategic plan for dealing with AI. Strategy #4 is Ensure the Safety and Security of AI Systems and they are saying a lot of the “right things”. For example, it includes discussion of emergent behavior, goal misspecification, explainability/transparency and long-term AI safety and value-alignment. Whether this will help translate into useful actions isn’t certain, but it’s heartening to see some acknowledgment of AI concerns from the US government besides just “develop it before China does”.
As for the current funding situation in the EA/AI risk community, I have also heard about this issue of there being too much funding for not enough good researchers/applicants right now. I don’t think we should get to used to this dynamic though. The situation could easily reverse in a short time if awareness about AI risk causes a wave of new research interest, or if 80,000 Hours, AGI Safety Fundamentals Curriculum, AI Safety Camp and related programs are able to introduce more people into the field. So just because we have a funding glut now doesn’t mean we should assume that will continue through 2023 which is the time period that this NSF RfI pertains to.
I sadly don’t have time to explain all of my models here, but I do want to publicly state that I think the NSF getting involved here is likely net-negative for the world, given the abundance of funding in the space, and people’s bad experiences interfacing with governmentally-funded research. I don’t currently assign any substantial probability to our funding abundance changing over the next few years, also since funding seems to be growing substantially faster than interested and deployable talent.
I hopefully can get back to this thread in a few days when I have more time to explain my models, but for now it seemed good to at least share the outputs of my models.
Habryka, I appreciate you sharing your outputs. Do you have a few minutes to follow up with a little explanation of your models yet? It’s ok if it’s a rough/incomplete explanation. But it would help to know a bit more about what you’ve seen with government-funded research etc. that makes you think this would be net-negative for the world.
Alas, sorry, I do think it would take me a while to write things up in any comprehensive way, and sadly I’ve been sick the last few days and so ended up falling behind a number of other commitments.
Here is a very very rough outline:
There really is already a lot of money in the space. Indeed, enough money that even substantial contributions from the NSF are unlikely to increase funding in any substantial proportion.
I’ve talked to multiple people at various organizations in EA and AI Alignment over the years who accepted NSF and other government funding over the years, and I think they regretted it in every instance, and found the experience very strongly distorting on the quality and alignment of the research.
I think there are indeed multiple fields that ended up derailed by actors like the NSF entering them, and then strongly distorting the incentives of the field. Nanotechnology for example I think ended up derailed in this kind of way, and there are a number of other subfields that I studied that seemed kind of similar.
I also expect the NSF getting involved will attract a number of adversarial actors that I expect will be quite distracting and potentially disruptive.
There are some more complicated feelings I have about having high-prestige research that deals with the potential negative consequences of AGI being net-negative, by increasing the probability of arms-races towards AGI. E.g. it’s pretty plausible to me that publishing Superintelligence was quite bad for the world. I don’t have super settled thoughts here, and am still quite confused, but I think it’s an important dimension to think about, and I think it adds some downside risk to this NSF situation, with relatively limited upside.
The situation could easily reverse in a short time if awareness about AI risk causes a wave of new research interest, or if 80,000 Hours, AGI Safety Fundamentals Curriculum, AI Safety Camp and related programs are able to introduce more people into the field. So just because we have a funding glut now doesn’t mean we should assume that will continue through 2023 which is the time period that this NSF RfI pertains to.
Could you put some numbers around this please—e.g. how much you think we might be able to get the NSF to spend on this? I think we have a big difference in our models here; I can’t think of any scenario you’re thinking of where this seems plausible.
For context, it looks like the NSF currently spends around $8.5bn a year, and this particular program was only $12.5m. It seems unlikely to me that we could get them to spend 2% of the budget ($170m) on AI safety in 2023. In contrast, if there was somehow $170m dollars of high quality grant proposals I’m pretty confident the existing EA funding system would be able to fund it all.
This might make sense if all the existing big donors suddenly decided that AI safety was not very important, so we were very short on money. But if that happens it’s probably because they have become aware of compelling new arguments not to fund AI safety, in which case the decision is probably reasonable!
How high quality do you think the grants the NSF will make would be?
Right now there is a very large amount of EA money available for AI safety research, between at least four different major groups. Each makes use of well connected domain experts to solicit and evaluate grants, and have application processes designed to be easy for applicants to fill in and then distribute the funds in a rapid fashion. Awards can be predicted sufficiently confidently that it is possible to create a career from a single one of these funders. However, the pool of good applicants is extremely small—all the organisations have struggled to spend this money effectively, and actively look for new ways of spending money faster.
In contrast I would be pessimistic about the quality of potential NSF grants. My concern is that, while we might be able to influence them to fund something called ‘AI safety’, it would not actually be that closely related to the thing we care about. Chosen grant reviewers could have many prestigious qualifications but lack a deep understanding of the problem. NSF grants can, I understand, take a long time to apply to, and the evaluation is also very slow—taking over six months. Even then success is not assured. So it’s possible a lot of high quality safety researchers would prefer dedicated EA funding anyway. Who does this leave getting NSF grants? I worry it would be largely existing academics in related fields who are able to reframe their research as corresponding to ‘safety’, thereby diluting the field, without significantly contributing to the project—similar to ‘AI Ethics’.
I do agree that NSF funding could help significantly raise the prestige of the field, but I would want to be confident in how high fidelity the idea transmission to grantmakers would be before promoting the idea, and I’m not sure how we can ensure this degree of alignment.
I agree that we need to care with high fidelity idea transmission, and there is some risk of diluting the field. But I think the reasonable chance of this spurring some more good research in AI safety is worth it, even if there will also be some wasted money.
One thing that’s interesting in the RfI is that it links to something called THE NATIONAL ARTIFICIAL INTELLIGENCE RESEARCH AND DEVELOPMENT STRATEGIC PLAN: 2019 UPDATE. This PDF outlines a federal committee’s strategic plan for dealing with AI. Strategy #4 is Ensure the Safety and Security of AI Systems and they are saying a lot of the “right things”. For example, it includes discussion of emergent behavior, goal misspecification, explainability/transparency and long-term AI safety and value-alignment. Whether this will help translate into useful actions isn’t certain, but it’s heartening to see some acknowledgment of AI concerns from the US government besides just “develop it before China does”.
As for the current funding situation in the EA/AI risk community, I have also heard about this issue of there being too much funding for not enough good researchers/applicants right now. I don’t think we should get to used to this dynamic though. The situation could easily reverse in a short time if awareness about AI risk causes a wave of new research interest, or if 80,000 Hours, AGI Safety Fundamentals Curriculum, AI Safety Camp and related programs are able to introduce more people into the field. So just because we have a funding glut now doesn’t mean we should assume that will continue through 2023 which is the time period that this NSF RfI pertains to.
I sadly don’t have time to explain all of my models here, but I do want to publicly state that I think the NSF getting involved here is likely net-negative for the world, given the abundance of funding in the space, and people’s bad experiences interfacing with governmentally-funded research. I don’t currently assign any substantial probability to our funding abundance changing over the next few years, also since funding seems to be growing substantially faster than interested and deployable talent.
I hopefully can get back to this thread in a few days when I have more time to explain my models, but for now it seemed good to at least share the outputs of my models.
Habryka, I appreciate you sharing your outputs. Do you have a few minutes to follow up with a little explanation of your models yet? It’s ok if it’s a rough/incomplete explanation. But it would help to know a bit more about what you’ve seen with government-funded research etc. that makes you think this would be net-negative for the world.
Alas, sorry, I do think it would take me a while to write things up in any comprehensive way, and sadly I’ve been sick the last few days and so ended up falling behind a number of other commitments.
Here is a very very rough outline:
There really is already a lot of money in the space. Indeed, enough money that even substantial contributions from the NSF are unlikely to increase funding in any substantial proportion.
I’ve talked to multiple people at various organizations in EA and AI Alignment over the years who accepted NSF and other government funding over the years, and I think they regretted it in every instance, and found the experience very strongly distorting on the quality and alignment of the research.
I think there are indeed multiple fields that ended up derailed by actors like the NSF entering them, and then strongly distorting the incentives of the field. Nanotechnology for example I think ended up derailed in this kind of way, and there are a number of other subfields that I studied that seemed kind of similar.
I also expect the NSF getting involved will attract a number of adversarial actors that I expect will be quite distracting and potentially disruptive.
There are some more complicated feelings I have about having high-prestige research that deals with the potential negative consequences of AGI being net-negative, by increasing the probability of arms-races towards AGI. E.g. it’s pretty plausible to me that publishing Superintelligence was quite bad for the world. I don’t have super settled thoughts here, and am still quite confused, but I think it’s an important dimension to think about, and I think it adds some downside risk to this NSF situation, with relatively limited upside.
Could you put some numbers around this please—e.g. how much you think we might be able to get the NSF to spend on this? I think we have a big difference in our models here; I can’t think of any scenario you’re thinking of where this seems plausible.
For context, it looks like the NSF currently spends around $8.5bn a year, and this particular program was only $12.5m. It seems unlikely to me that we could get them to spend 2% of the budget ($170m) on AI safety in 2023. In contrast, if there was somehow $170m dollars of high quality grant proposals I’m pretty confident the existing EA funding system would be able to fund it all.
This might make sense if all the existing big donors suddenly decided that AI safety was not very important, so we were very short on money. But if that happens it’s probably because they have become aware of compelling new arguments not to fund AI safety, in which case the decision is probably reasonable!