Relatedly, I’d be interested to hear more about Nonlinear’s thoughts on what the downside risks of this org/approach might be, and how you plan to mitigate them. I appreciated the section “What about the risks?”, and I think gathering that board of advisors seems like a great step, but I’d be interested to hear you expand on that topic.
I think the main downside risks I’d personally have in mind would be risks to the reputations and relationships of other people working on related issues (particularly AI safety, other existential risks, and other longtermist stuff). It seems important to avoid seeming naive, slapdash, non-expert, or weird when working on these topics, or at least to find ways of minimising the chances that such perceptions would rub off on other people working in the same spaces.
(To be clear, I do not mean to imply that the existence of concerns like this means no one should ever do any projects like this. All projects will have at least some degree of downside risks, and some projects are very much worth doing even given those risks. It’s always just a matter of assessing risks vs benefits, thinking of mitigation options, and trying to account for biases and the unilateralist’s curse. So I’m just asking questions, rather than trying to imply a criticism.)
P.S. Some sources that inform/express my views on this sort of topic (in a generic rather than AI-safety-specific way):
From an overarching viewpoint, I am personally extremely motivated to avoid accidentally doing more harm than good. I have seen how very easy it is to do that in the relatively forgiving fields of poverty and animal welfare and the stakes are much higher and the field much smaller in AI safety. I literally (not figuratively or hyperbolically) lose sleep over this concern. So when I say we take it seriously, it’s not corporate speak for appeasing the masses, but a deeply, genuinely held concern. I say this to point towards the fact that whatever our current methods are for avoiding causing harm, we are motivated to find and become aware of other ways to increase our robustness.
More specifically, another approach we’re using is being extremely cautious in launching things, even if we are not convinced by an advisor’s object level arguments. Last year I was considering launching a project but before I went for it, I asked a bunch of experts in the area. Lots of people liked the idea but some were worried about it for various reasons. I wasn’t convinced by their reasoning, but I am convinced by epistemic modesty arguments and they had more experience in the area, so I nixxed the project. We intend to have a similar mindset moving forwards, while still keeping in mind that no project will ever be universally considered good.
I wasn’t convinced by their reasoning, but I am convinced by epistemic modesty arguments and they had more experience in the area, so I nixxed the project.
I agree that the epistemic modesty/humility idea of “defer at least somewhat to other people, and more so the more relevant experience/expertise they have” makes sense in general.
I also think that the unilateralist’s curse provides additional reason to take that sort of approach in situations (like this one) where “some number of altruistically minded actors each have the ability to take an action that would cause accidental harm to others” (quoting from that link). So it’s good to hear you’re doing that :)
On a somewhat related note, I’d be interested to hear more about what you mean by “advocacy” when you say “Once a top idea has been vetted, we use a variety of tools to turn it into a reality, including grantmaking, advocacy, RFPs, and incubating it ourselves.” Do you mean like advocacy to the general public? Or like writing EA Forum posts about the idea to encourage EAs to act on it?
Similar questions came to mind for me as well.
Relatedly, I’d be interested to hear more about Nonlinear’s thoughts on what the downside risks of this org/approach might be, and how you plan to mitigate them. I appreciated the section “What about the risks?”, and I think gathering that board of advisors seems like a great step, but I’d be interested to hear you expand on that topic.
I think the main downside risks I’d personally have in mind would be risks to the reputations and relationships of other people working on related issues (particularly AI safety, other existential risks, and other longtermist stuff). It seems important to avoid seeming naive, slapdash, non-expert, or weird when working on these topics, or at least to find ways of minimising the chances that such perceptions would rub off on other people working in the same spaces.
(To be clear, I do not mean to imply that the existence of concerns like this means no one should ever do any projects like this. All projects will have at least some degree of downside risks, and some projects are very much worth doing even given those risks. It’s always just a matter of assessing risks vs benefits, thinking of mitigation options, and trying to account for biases and the unilateralist’s curse. So I’m just asking questions, rather than trying to imply a criticism.)
P.S. Some sources that inform/express my views on this sort of topic (in a generic rather than AI-safety-specific way):
Ways people trying to do good accidentally make things worse, and how to avoid them
How to avoid accidentally having a negative impact with your project
Good and bad ways to think about downside risks
Hard-to-reverse decisions destroy option value
Thanks for the links and thoughtful question!
From an overarching viewpoint, I am personally extremely motivated to avoid accidentally doing more harm than good. I have seen how very easy it is to do that in the relatively forgiving fields of poverty and animal welfare and the stakes are much higher and the field much smaller in AI safety. I literally (not figuratively or hyperbolically) lose sleep over this concern. So when I say we take it seriously, it’s not corporate speak for appeasing the masses, but a deeply, genuinely held concern. I say this to point towards the fact that whatever our current methods are for avoiding causing harm, we are motivated to find and become aware of other ways to increase our robustness.
More specifically, another approach we’re using is being extremely cautious in launching things, even if we are not convinced by an advisor’s object level arguments. Last year I was considering launching a project but before I went for it, I asked a bunch of experts in the area. Lots of people liked the idea but some were worried about it for various reasons. I wasn’t convinced by their reasoning, but I am convinced by epistemic modesty arguments and they had more experience in the area, so I nixxed the project. We intend to have a similar mindset moving forwards, while still keeping in mind that no project will ever be universally considered good.
That sounds good to me.
I agree that the epistemic modesty/humility idea of “defer at least somewhat to other people, and more so the more relevant experience/expertise they have” makes sense in general.
I also think that the unilateralist’s curse provides additional reason to take that sort of approach in situations (like this one) where “some number of altruistically minded actors each have the ability to take an action that would cause accidental harm to others” (quoting from that link). So it’s good to hear you’re doing that :)
On a somewhat related note, I’d be interested to hear more about what you mean by “advocacy” when you say “Once a top idea has been vetted, we use a variety of tools to turn it into a reality, including grantmaking, advocacy, RFPs, and incubating it ourselves.” Do you mean like advocacy to the general public? Or like writing EA Forum posts about the idea to encourage EAs to act on it?