Where do you draw the line between AI startups that do vs don’t contribute excessively to capabilities externalities and existential risk? I think you’re right that your particular startup wouldn’t have a significant effect of accelerating timelines. But if we’re thinking AI startups in general, this could be another OpenAI or Adept, which probably have more of an effect on timelines.
I could imagine that even if one’s startup doesn’t working on scaling and making models generally smarter, a relatively small amount of applications work to make them more useful could put them at notably greater risk of having dangerous capabilities or intent. As an example, imagine if OpenAI only made GPT-3 and never produced InstructGPT or ChatGPT. It feels a lot harder to steer GPT-3 to do useful things, so I think that there would have been noticeably less adoption of LLMs and interest in advancing their capabilities, at least for a while. (For clarification, my claim isn’t that InstructGPT and ChatGPT necessarily contributed to existential risk, but they do have capabilities externalities and I think affected timelines, in part due to the hype they generated.)
If your claim is that ‘applying AI models for economically valuable tasks seems dangerous, i.e. the AIs themselves could be dangerous’ then I agree. A scrappy applications company might be more likely to end the world than OpenAI/DeepMind… it seems like it would be good, then, if more of these companies were run by safety conscious people.
A separate claim is the one about capabilities externalities. I basically agree that AI startups will have capabilities externalities, even if I don’t expect them to be very large. The question, then, is how much expected money we would be trading for expected time and what is the relative value between these two currencies.
Where do you draw the line between AI startups that do vs don’t contribute excessively to capabilities externalities and existential risk? I think you’re right that your particular startup wouldn’t have a significant effect of accelerating timelines. But if we’re thinking AI startups in general, this could be another OpenAI or Adept, which probably have more of an effect on timelines.
I could imagine that even if one’s startup doesn’t working on scaling and making models generally smarter, a relatively small amount of applications work to make them more useful could put them at notably greater risk of having dangerous capabilities or intent. As an example, imagine if OpenAI only made GPT-3 and never produced InstructGPT or ChatGPT. It feels a lot harder to steer GPT-3 to do useful things, so I think that there would have been noticeably less adoption of LLMs and interest in advancing their capabilities, at least for a while. (For clarification, my claim isn’t that InstructGPT and ChatGPT necessarily contributed to existential risk, but they do have capabilities externalities and I think affected timelines, in part due to the hype they generated.)
If your claim is that ‘applying AI models for economically valuable tasks seems dangerous, i.e. the AIs themselves could be dangerous’ then I agree. A scrappy applications company might be more likely to end the world than OpenAI/DeepMind… it seems like it would be good, then, if more of these companies were run by safety conscious people.
A separate claim is the one about capabilities externalities. I basically agree that AI startups will have capabilities externalities, even if I don’t expect them to be very large. The question, then, is how much expected money we would be trading for expected time and what is the relative value between these two currencies.