Both founders don’t seem to have a background in technical AI safety research. Why do you think Nonlinear will be able to research and prioritize these interventions without having prior experience or familiarity in technical AI safety research?
Relatedly, wouldn’t the organization be better if it hired for a full-time researcher or have a co-founder who has a background in technical AI safety research? Is this something you’re considering doing?
Relatedly, I’d be interested to hear more about Nonlinear’s thoughts on what the downside risks of this org/approach might be, and how you plan to mitigate them. I appreciated the section “What about the risks?”, and I think gathering that board of advisors seems like a great step, but I’d be interested to hear you expand on that topic.
I think the main downside risks I’d personally have in mind would be risks to the reputations and relationships of other people working on related issues (particularly AI safety, other existential risks, and other longtermist stuff). It seems important to avoid seeming naive, slapdash, non-expert, or weird when working on these topics, or at least to find ways of minimising the chances that such perceptions would rub off on other people working in the same spaces.
(To be clear, I do not mean to imply that the existence of concerns like this means no one should ever do any projects like this. All projects will have at least some degree of downside risks, and some projects are very much worth doing even given those risks. It’s always just a matter of assessing risks vs benefits, thinking of mitigation options, and trying to account for biases and the unilateralist’s curse. So I’m just asking questions, rather than trying to imply a criticism.)
P.S. Some sources that inform/express my views on this sort of topic (in a generic rather than AI-safety-specific way):
From an overarching viewpoint, I am personally extremely motivated to avoid accidentally doing more harm than good. I have seen how very easy it is to do that in the relatively forgiving fields of poverty and animal welfare and the stakes are much higher and the field much smaller in AI safety. I literally (not figuratively or hyperbolically) lose sleep over this concern. So when I say we take it seriously, it’s not corporate speak for appeasing the masses, but a deeply, genuinely held concern. I say this to point towards the fact that whatever our current methods are for avoiding causing harm, we are motivated to find and become aware of other ways to increase our robustness.
More specifically, another approach we’re using is being extremely cautious in launching things, even if we are not convinced by an advisor’s object level arguments. Last year I was considering launching a project but before I went for it, I asked a bunch of experts in the area. Lots of people liked the idea but some were worried about it for various reasons. I wasn’t convinced by their reasoning, but I am convinced by epistemic modesty arguments and they had more experience in the area, so I nixxed the project. We intend to have a similar mindset moving forwards, while still keeping in mind that no project will ever be universally considered good.
I wasn’t convinced by their reasoning, but I am convinced by epistemic modesty arguments and they had more experience in the area, so I nixxed the project.
I agree that the epistemic modesty/humility idea of “defer at least somewhat to other people, and more so the more relevant experience/expertise they have” makes sense in general.
I also think that the unilateralist’s curse provides additional reason to take that sort of approach in situations (like this one) where “some number of altruistically minded actors each have the ability to take an action that would cause accidental harm to others” (quoting from that link). So it’s good to hear you’re doing that :)
On a somewhat related note, I’d be interested to hear more about what you mean by “advocacy” when you say “Once a top idea has been vetted, we use a variety of tools to turn it into a reality, including grantmaking, advocacy, RFPs, and incubating it ourselves.” Do you mean like advocacy to the general public? Or like writing EA Forum posts about the idea to encourage EAs to act on it?
Part of me wonders if a better model than the one outlined in this post is for Nonlinear to collaborate with well-established AI research organisations who can advise on the high-impact interventions, for which Nonlinear then proceeds to do the grunt work to turn into a reality.
Even in this alternative model I agree that Nonlinear would probably benefit from someone with in-depth knowledge of AI safety as a full-time employee.
This is indeed part of our plan! No need to re-invent the wheel. :)
One of our first steps will be to canvas existing AI Safety organizations and compile a comprehensive list of ideas they want done. We will do our own due diligence before launching any of them, but I would love for it to be that Nonlinear is the organization people come to when they have a great idea that they want to have happen.
For hiring full-time RAs, we have plans to do that in the future. Right now we are being slow on hiring full-timers. We want to get feedback from external people first (thank you!) and have a more solidified strategy before taking on permanent employees.
We are, however, working on developing a technical advisory board of people who are experts in ML. If you know anybody who’d be keen, please send them our way!
Sorry for the miscommunication. We are not intending to do technical AI safety work. We are going to focus on non-technical for the time being.
I am in the process of learning ML but am very far from being able to make contributions in that area. This is mostly so that I have a better understanding of the area and can better communicate with people with more technical expertise.
However, I’m still a bit confused. When you say “We are not intending to do technical AI safety work. We are going to focus on non-technical for the time being.”, do you mean you will only be researching high leverage, non-technical AI Safety interventions? Or do you mean that the research work you’re doing is non-technical?
I understand that the research work you’re doing is non-technical (in that you probably aren’t going to directly use any ML to do your research), but I’m not that aware of what the non-technical AI Safety interventions are, aside from semi-related things like working on AI strategy and policy (i.e. FHI’s GovAI, The Partnership on AI) and advocating against shorter-term AI risks (i.e. Future of Life Institute’s work on Lethal Autonomous Weapons Systems). Could you elaborate on what you mean when you say you will focus on non-technical AI safety work for the time being? Maybe you could give some examples of possible non-technical AI safety interventions? Thanks!
For sure. One example that we’ll be researching is scaling up getting PAs for high impact people in AI safety. It seems like one of the things that’s bottlenecking the movement is talent. Getting more talent is one solution which we should definitely be working on. Another is helping the talent we already have be more productive. Setting up an organization that specializes in hiring PAs and pairing them with top AI safety experts seems like a potentially great way to boost the impact of already high impact people.
I’m not that aware of what the non-technical AI Safety interventions are, aside from semi-related things like working on AI strategy and policy (i.e. FHI’s GovAI, The Partnership on AI) and advocating against shorter-term AI risks (i.e. Future of Life Institute’s work on Lethal Autonomous Weapons Systems).
Just wanted to quickly flag: I think the more popular interpretation of the term AI safety points to a wide landscape that includes AI policy/strategy as well as technical AI safety (which is also often referred to by the term AI alignment).
I thought the term AI safety was shorthand for technical AI safety, and didn’t really include AI policy/strategy. I personally use the term AI risk (or sometimes AI x-risk) to group together work on AI safety and AI strategy/policy/governance, i.e. work on AI risk = work on AI safety or AI strategy/policy.
I was aware though of AI safety being referred to as AI alignment.
Both founders don’t seem to have a background in technical AI safety research. Why do you think Nonlinear will be able to research and prioritize these interventions without having prior experience or familiarity in technical AI safety research?
Relatedly, wouldn’t the organization be better if it hired for a full-time researcher or have a co-founder who has a background in technical AI safety research? Is this something you’re considering doing?
Similar questions came to mind for me as well.
Relatedly, I’d be interested to hear more about Nonlinear’s thoughts on what the downside risks of this org/approach might be, and how you plan to mitigate them. I appreciated the section “What about the risks?”, and I think gathering that board of advisors seems like a great step, but I’d be interested to hear you expand on that topic.
I think the main downside risks I’d personally have in mind would be risks to the reputations and relationships of other people working on related issues (particularly AI safety, other existential risks, and other longtermist stuff). It seems important to avoid seeming naive, slapdash, non-expert, or weird when working on these topics, or at least to find ways of minimising the chances that such perceptions would rub off on other people working in the same spaces.
(To be clear, I do not mean to imply that the existence of concerns like this means no one should ever do any projects like this. All projects will have at least some degree of downside risks, and some projects are very much worth doing even given those risks. It’s always just a matter of assessing risks vs benefits, thinking of mitigation options, and trying to account for biases and the unilateralist’s curse. So I’m just asking questions, rather than trying to imply a criticism.)
P.S. Some sources that inform/express my views on this sort of topic (in a generic rather than AI-safety-specific way):
Ways people trying to do good accidentally make things worse, and how to avoid them
How to avoid accidentally having a negative impact with your project
Good and bad ways to think about downside risks
Hard-to-reverse decisions destroy option value
Thanks for the links and thoughtful question!
From an overarching viewpoint, I am personally extremely motivated to avoid accidentally doing more harm than good. I have seen how very easy it is to do that in the relatively forgiving fields of poverty and animal welfare and the stakes are much higher and the field much smaller in AI safety. I literally (not figuratively or hyperbolically) lose sleep over this concern. So when I say we take it seriously, it’s not corporate speak for appeasing the masses, but a deeply, genuinely held concern. I say this to point towards the fact that whatever our current methods are for avoiding causing harm, we are motivated to find and become aware of other ways to increase our robustness.
More specifically, another approach we’re using is being extremely cautious in launching things, even if we are not convinced by an advisor’s object level arguments. Last year I was considering launching a project but before I went for it, I asked a bunch of experts in the area. Lots of people liked the idea but some were worried about it for various reasons. I wasn’t convinced by their reasoning, but I am convinced by epistemic modesty arguments and they had more experience in the area, so I nixxed the project. We intend to have a similar mindset moving forwards, while still keeping in mind that no project will ever be universally considered good.
That sounds good to me.
I agree that the epistemic modesty/humility idea of “defer at least somewhat to other people, and more so the more relevant experience/expertise they have” makes sense in general.
I also think that the unilateralist’s curse provides additional reason to take that sort of approach in situations (like this one) where “some number of altruistically minded actors each have the ability to take an action that would cause accidental harm to others” (quoting from that link). So it’s good to hear you’re doing that :)
On a somewhat related note, I’d be interested to hear more about what you mean by “advocacy” when you say “Once a top idea has been vetted, we use a variety of tools to turn it into a reality, including grantmaking, advocacy, RFPs, and incubating it ourselves.” Do you mean like advocacy to the general public? Or like writing EA Forum posts about the idea to encourage EAs to act on it?
Part of me wonders if a better model than the one outlined in this post is for Nonlinear to collaborate with well-established AI research organisations who can advise on the high-impact interventions, for which Nonlinear then proceeds to do the grunt work to turn into a reality.
Even in this alternative model I agree that Nonlinear would probably benefit from someone with in-depth knowledge of AI safety as a full-time employee.
This is indeed part of our plan! No need to re-invent the wheel. :)
One of our first steps will be to canvas existing AI Safety organizations and compile a comprehensive list of ideas they want done. We will do our own due diligence before launching any of them, but I would love for it to be that Nonlinear is the organization people come to when they have a great idea that they want to have happen.
Sounds good!
Replied to hiring full-timer above https://forum.effectivealtruism.org/posts/fX8JsabQyRSd7zWiD/introducing-the-nonlinear-fund-ai-safety-research-incubation?commentId=ANTbuSPrNTwRHvw73
For hiring full-time RAs, we have plans to do that in the future. Right now we are being slow on hiring full-timers. We want to get feedback from external people first (thank you!) and have a more solidified strategy before taking on permanent employees.
We are, however, working on developing a technical advisory board of people who are experts in ML. If you know anybody who’d be keen, please send them our way!
I see, makes sense!
Good and important points!
Sorry for the miscommunication. We are not intending to do technical AI safety work. We are going to focus on non-technical for the time being.
I am in the process of learning ML but am very far from being able to make contributions in that area. This is mostly so that I have a better understanding of the area and can better communicate with people with more technical expertise.
Thanks for the reply Kat!
However, I’m still a bit confused. When you say “We are not intending to do technical AI safety work. We are going to focus on non-technical for the time being.”, do you mean you will only be researching high leverage, non-technical AI Safety interventions? Or do you mean that the research work you’re doing is non-technical?
I understand that the research work you’re doing is non-technical (in that you probably aren’t going to directly use any ML to do your research), but I’m not that aware of what the non-technical AI Safety interventions are, aside from semi-related things like working on AI strategy and policy (i.e. FHI’s GovAI, The Partnership on AI) and advocating against shorter-term AI risks (i.e. Future of Life Institute’s work on Lethal Autonomous Weapons Systems). Could you elaborate on what you mean when you say you will focus on non-technical AI safety work for the time being? Maybe you could give some examples of possible non-technical AI safety interventions? Thanks!
For sure. One example that we’ll be researching is scaling up getting PAs for high impact people in AI safety. It seems like one of the things that’s bottlenecking the movement is talent. Getting more talent is one solution which we should definitely be working on. Another is helping the talent we already have be more productive. Setting up an organization that specializes in hiring PAs and pairing them with top AI safety experts seems like a potentially great way to boost the impact of already high impact people.
Great, I think that’s a good idea actually! I’m looking forward to see other potential good ideas like that from Nonlinear’s research.
Just wanted to quickly flag: I think the more popular interpretation of the term AI safety points to a wide landscape that includes AI policy/strategy as well as technical AI safety (which is also often referred to by the term AI alignment).
Thanks for clarifying! I wasn’t aware.
I thought the term AI safety was shorthand for technical AI safety, and didn’t really include AI policy/strategy. I personally use the term AI risk (or sometimes AI x-risk) to group together work on AI safety and AI strategy/policy/governance, i.e. work on AI risk = work on AI safety or AI strategy/policy.
I was aware though of AI safety being referred to as AI alignment.