Cool, thanks for sharing, I’m a big fan of Elicit! Some spontaneous thoughts:
We want AI to be more helpful for qualitative research, long-term forecasting, planning, and decision-making than for persuasion, keeping people engaged, and military robotics.
Are you worried that your work will be used for more likely regretable things like
improving the competence of actors who are less altruistic and less careful about unintended consequences (e.g. many companies, militaries and government insitutions), and
speeding up AI capabilities research, and speeding it up more than AI safety research?
I suppose it will be difficult to have much control over insights you generate and it will be relatively easy to replicate your product if you make it publicly available?
Have you considered deemphasizing trying to offer a commercially successful product that will find broad application in the world, and focussing more strongly on designing systems that are safe and aligned with human values?
Regarding the competition between process-based vs. outcome-based machine learning
Today, process-based systems are ahead: Most systems in the world don’t use much machine learning, and to the extent that they use it, it’s for small, independently meaningful, fairly interpretable steps like predictive search, ranking, or recommendation as part of much larger systems. [from your referenced LessWrong post]
My first reaction was thinking that today’s ML systems might not be the best comparison, and instead you might want to include all information processing systems, which include human brains. I guess human brains are mostly outcome-based systems with processed-based features:
we’re monitoring our own thinking and adjust it if it fails to live up to standards we hold, and
we communicate our thought processes for feedback and to teach others
But most of it seems outcome-based and fairly inscrutable?
Are you worried that your work will be used for more likely regretable things like
improving the competence of actors who are less altruistic and less careful about unintended consequences (e.g. many companies, militaries and government insitutions), and
Less careful actors: Our goal is for Elicit to help people reason better. We want less careful people to use it and reason better than they would have without Elicit, recognizing more unintended consequences and finding actions that are more aligned with their values. The hope is that if we can make good reasoning cheap enough, people will use it. In a sense, we’re all less careful actors right now.
Less altruistic actors: We favor more altruistic actors in deciding who to work with, give access to, and improve Elicit for. We also monitor use so that we can prevent misuse.
speeding up AI capabilities research, and speeding it up more than AI safety research?
I expect the overall impact on x-risk to be a reduction by (a) causing more and better x-risk reduction thinking to happen and (b) shifting ML efforts to a more alignable paradigm, even if (c) Elicit has a non-zero contribution to ML capabilities.
The implicit claim in the concern about speeding up capabilities is that Elicit has a large impact on capabilities because it is so useful. If that is true, we’d expect that it’s also super useful for other domains e.g. AI safety. The larger Elicit’s impact on (c), the larger the corresponding impacts on (a) and (b).
To shift the balance away from (c) we’ll focus on supporting safety-related research and researchers, especially conceptual research. We’re not doing this very well today but are actively thinking about it and moving in that direction. Given that, it would be surprising if Elicit helped a lot with ML capabilities relative to tools and organizations that are explicitly pushing that agenda.
Have you considered deemphasizing trying to offer a commercially successful product that will find broad application in the world, and focussing more strongly on designing systems that are safe and aligned with human values?
We’re a non-profit so have no obligation to make a commercially successful product. We’ll only focus on it to the extent that it furthers aligned reasoning. That said, I think the best outcome is that we make a widely adopted product that makes it easier for everyone to think through the consequences of their actions and act in alignment with their values.
I was fuzzy about what I wanted to communicate with the term “careful”, thanks for spelling out your perspective here. I’m still a little uneasy about the idea that generally improving the ability to plan better will also make sufficiently many actors more careful about avoiding problems that are particularly risky for our future. It just seems so rare that important actors care enough about such risks, even for things that humanity is able to predict and plan for reasonably well, like pandemics.
We’re also only reporting our current guess for how things will turn out. We’re monitoring how Elicit is used and we’ll study its impacts and the anticipated impacts of future features, and if it turns out that the costs outweigh the benefits we will adjust our plans.
Cool, thanks for sharing, I’m a big fan of Elicit! Some spontaneous thoughts:
Are you worried that your work will be used for more likely regretable things like
improving the competence of actors who are less altruistic and less careful about unintended consequences (e.g. many companies, militaries and government insitutions), and
speeding up AI capabilities research, and speeding it up more than AI safety research?
I suppose it will be difficult to have much control over insights you generate and it will be relatively easy to replicate your product if you make it publicly available?
Have you considered deemphasizing trying to offer a commercially successful product that will find broad application in the world, and focussing more strongly on designing systems that are safe and aligned with human values?
Regarding the competition between process-based vs. outcome-based machine learning
My first reaction was thinking that today’s ML systems might not be the best comparison, and instead you might want to include all information processing systems, which include human brains. I guess human brains are mostly outcome-based systems with processed-based features:
we’re monitoring our own thinking and adjust it if it fails to live up to standards we hold, and
we communicate our thought processes for feedback and to teach others
But most of it seems outcome-based and fairly inscrutable?
Less careful actors: Our goal is for Elicit to help people reason better. We want less careful people to use it and reason better than they would have without Elicit, recognizing more unintended consequences and finding actions that are more aligned with their values. The hope is that if we can make good reasoning cheap enough, people will use it. In a sense, we’re all less careful actors right now.
Less altruistic actors: We favor more altruistic actors in deciding who to work with, give access to, and improve Elicit for. We also monitor use so that we can prevent misuse.
I expect the overall impact on x-risk to be a reduction by (a) causing more and better x-risk reduction thinking to happen and (b) shifting ML efforts to a more alignable paradigm, even if (c) Elicit has a non-zero contribution to ML capabilities.
The implicit claim in the concern about speeding up capabilities is that Elicit has a large impact on capabilities because it is so useful. If that is true, we’d expect that it’s also super useful for other domains e.g. AI safety. The larger Elicit’s impact on (c), the larger the corresponding impacts on (a) and (b).
To shift the balance away from (c) we’ll focus on supporting safety-related research and researchers, especially conceptual research. We’re not doing this very well today but are actively thinking about it and moving in that direction. Given that, it would be surprising if Elicit helped a lot with ML capabilities relative to tools and organizations that are explicitly pushing that agenda.
We’re a non-profit so have no obligation to make a commercially successful product. We’ll only focus on it to the extent that it furthers aligned reasoning. That said, I think the best outcome is that we make a widely adopted product that makes it easier for everyone to think through the consequences of their actions and act in alignment with their values.
Thanks a lot for elaborating, makes sense to me.
I was fuzzy about what I wanted to communicate with the term “careful”, thanks for spelling out your perspective here. I’m still a little uneasy about the idea that generally improving the ability to plan better will also make sufficiently many actors more careful about avoiding problems that are particularly risky for our future. It just seems so rare that important actors care enough about such risks, even for things that humanity is able to predict and plan for reasonably well, like pandemics.
We’re also only reporting our current guess for how things will turn out. We’re monitoring how Elicit is used and we’ll study its impacts and the anticipated impacts of future features, and if it turns out that the costs outweigh the benefits we will adjust our plans.