Relatedly, Iād be interested to hear more about Nonlinearās thoughts on what the downside risks of this org/āapproach might be, and how you plan to mitigate them. I appreciated the section āWhat about the risks?ā, and I think gathering that board of advisors seems like a great step, but Iād be interested to hear you expand on that topic.
I think the main downside risks Iād personally have in mind would be risks to the reputations and relationships of other people working on related issues (particularly AI safety, other existential risks, and other longtermist stuff). It seems important to avoid seeming naive, slapdash, non-expert, or weird when working on these topics, or at least to find ways of minimising the chances that such perceptions would rub off on other people working in the same spaces.
(To be clear, I do not mean to imply that the existence of concerns like this means no one should ever do any projects like this. All projects will have at least some degree of downside risks, and some projects are very much worth doing even given those risks. Itās always just a matter of assessing risks vs benefits, thinking of mitigation options, and trying to account for biases and the unilateralistās curse. So Iām just asking questions, rather than trying to imply a criticism.)
P.S. Some sources that inform/āexpress my views on this sort of topic (in a generic rather than AI-safety-specific way):
From an overarching viewpoint, I am personally extremely motivated to avoid accidentally doing more harm than good. I have seen how very easy it is to do that in the relatively forgiving fields of poverty and animal welfare and the stakes are much higher and the field much smaller in AI safety. I literally (not figuratively or hyperbolically) lose sleep over this concern. So when I say we take it seriously, itās not corporate speak for appeasing the masses, but a deeply, genuinely held concern. I say this to point towards the fact that whatever our current methods are for avoiding causing harm, we are motivated to find and become aware of other ways to increase our robustness.
More specifically, another approach weāre using is being extremely cautious in launching things, even if we are not convinced by an advisorās object level arguments. Last year I was considering launching a project but before I went for it, I asked a bunch of experts in the area. Lots of people liked the idea but some were worried about it for various reasons. I wasnāt convinced by their reasoning, but I am convinced by epistemic modesty arguments and they had more experience in the area, so I nixxed the project. We intend to have a similar mindset moving forwards, while still keeping in mind that no project will ever be universally considered good.
I wasnāt convinced by their reasoning, but I am convinced by epistemic modesty arguments and they had more experience in the area, so I nixxed the project.
I agree that the epistemic modesty/āhumility idea of ādefer at least somewhat to other people, and more so the more relevant experience/āexpertise they haveā makes sense in general.
I also think that the unilateralistās curse provides additional reason to take that sort of approach in situations (like this one) where āsome number of altruistically minded actors each have the ability to take an action that would cause accidental harm to othersā (quoting from that link). So itās good to hear youāre doing that :)
On a somewhat related note, Iād be interested to hear more about what you mean by āadvocacyā when you say āOnce a top idea has been vetted, we use a variety of tools to turn it into a reality, including grantmaking, advocacy, RFPs, and incubating it ourselves.ā Do you mean like advocacy to the general public? Or like writing EA Forum posts about the idea to encourage EAs to act on it?
Similar questions came to mind for me as well.
Relatedly, Iād be interested to hear more about Nonlinearās thoughts on what the downside risks of this org/āapproach might be, and how you plan to mitigate them. I appreciated the section āWhat about the risks?ā, and I think gathering that board of advisors seems like a great step, but Iād be interested to hear you expand on that topic.
I think the main downside risks Iād personally have in mind would be risks to the reputations and relationships of other people working on related issues (particularly AI safety, other existential risks, and other longtermist stuff). It seems important to avoid seeming naive, slapdash, non-expert, or weird when working on these topics, or at least to find ways of minimising the chances that such perceptions would rub off on other people working in the same spaces.
(To be clear, I do not mean to imply that the existence of concerns like this means no one should ever do any projects like this. All projects will have at least some degree of downside risks, and some projects are very much worth doing even given those risks. Itās always just a matter of assessing risks vs benefits, thinking of mitigation options, and trying to account for biases and the unilateralistās curse. So Iām just asking questions, rather than trying to imply a criticism.)
P.S. Some sources that inform/āexpress my views on this sort of topic (in a generic rather than AI-safety-specific way):
Ways people trying to do good accidentally make things worse, and how to avoid them
How to avoid accidentally having a negative impact with your project
Good and bad ways to think about downside risks
Hard-to-reverse decisions destroy option value
Thanks for the links and thoughtful question!
From an overarching viewpoint, I am personally extremely motivated to avoid accidentally doing more harm than good. I have seen how very easy it is to do that in the relatively forgiving fields of poverty and animal welfare and the stakes are much higher and the field much smaller in AI safety. I literally (not figuratively or hyperbolically) lose sleep over this concern. So when I say we take it seriously, itās not corporate speak for appeasing the masses, but a deeply, genuinely held concern. I say this to point towards the fact that whatever our current methods are for avoiding causing harm, we are motivated to find and become aware of other ways to increase our robustness.
More specifically, another approach weāre using is being extremely cautious in launching things, even if we are not convinced by an advisorās object level arguments. Last year I was considering launching a project but before I went for it, I asked a bunch of experts in the area. Lots of people liked the idea but some were worried about it for various reasons. I wasnāt convinced by their reasoning, but I am convinced by epistemic modesty arguments and they had more experience in the area, so I nixxed the project. We intend to have a similar mindset moving forwards, while still keeping in mind that no project will ever be universally considered good.
That sounds good to me.
I agree that the epistemic modesty/āhumility idea of ādefer at least somewhat to other people, and more so the more relevant experience/āexpertise they haveā makes sense in general.
I also think that the unilateralistās curse provides additional reason to take that sort of approach in situations (like this one) where āsome number of altruistically minded actors each have the ability to take an action that would cause accidental harm to othersā (quoting from that link). So itās good to hear youāre doing that :)
On a somewhat related note, Iād be interested to hear more about what you mean by āadvocacyā when you say āOnce a top idea has been vetted, we use a variety of tools to turn it into a reality, including grantmaking, advocacy, RFPs, and incubating it ourselves.ā Do you mean like advocacy to the general public? Or like writing EA Forum posts about the idea to encourage EAs to act on it?