In general, it is a bad idea to trade increased probability that the world ends for money if your goal is to decrease probability that the world ends. People are usually bad at this kind of consequentialism, and this definitely strikes my ‘galaxybrain take’ detector.
I interpreted this post as the author saying that they thought general AI capabilities would be barely advanced by this kind of thing, if they were advanced by it at all. The author doesn’t seem to suggest building an AGI startup, but rather some kind of AI application startup.
I’m curious if you think your reasoning applies to anything with a small chance of accelerating timelines by a small amount, or if you instead disagree with the object-level claim that such a startup would only have a small chance of accelerating timelines by a small amount.
I think it has a large chance of accelerating timelines by a small amount, and a small chance of accelerating timelines by a large amount. You can definitely increase capabilities, even if they’re not doing research directly into increasing the size of our language models. Figuring out how you milk language models for all the capabilities they have, the limits of such milking, and making highly capable APIs easy for language models to use are all things which shorten timelines. You go from needing a super duper AGI to take over the world to a barely super-human AGI, if all it can do is output text.
Adjacently, contributing to the AGI hype shortens timelines too.
I also think the above assumes monetary or prestige pressures won’t cause organizational value drift. I think its quite likely whoever starts this will see pressure from funders, staff, and others to turn it into an AGI firm. You need good reason to believe your firm is going to not cave in, and I see nothing addressing this concern in the original post.
Also, from what I’ve heard, you cannot in fact use ungodly amounts of money to move talent. Generally, if top researchers were swayable that way, they’d be working in industry. Mostly, they just like working on their research, and don’t care much about how much they’re paid.
Thanks for the referral. I agree that the distinction between serial time and parallel time is important and that serial time is more valuable. I’m not sure if it is astronomically more valuable though. There are two points we could have differing views on:
- the amount of expected serial time a successful (let’s say $10 billion dollar) AI startup is likely to counterfactually burn. In the post I claimed that this seems unlikely to be more than a few weeks. Would you agree with this? - the relative value of serial time to money (which is exchangeable with parallel time). If you agree with the first statement, would you trade $10 billion dollars for 3 weeks of serial time at the current margin?
If you would not trade $10 billion for 3 weeks that could be because: - I’m more optimistic about empirical research / think the time iterating later when we have the systems is significantly more important than the time now when we can only try to reason about them. - you think money will be much less useful than I expect it to be
I would be interested in pinning down where I disagree with you (and others who probably disagree with the post for similar reasons).
the amount of expected serial time a successful (let’s say $10 billion dollar) AI startup is likely to counterfactually burn. In the post I claimed that this seems unlikely to be more than a few weeks. Would you agree with this?
No, see my comment above. Its the difference between a super duper AGI and only a super-human AGI, which could be years or months (but very very critical months!). Plus whatever you add to the hype, plus worlds where you somehow make $10 billion from this are also worlds where you’ve had an inordinate impact, which makes me more suspicious the $10 billion company world is the one where someone decided to just make the company another AGI lab.
the relative value of serial time to money (which is exchangeable with parallel time). If you agree with the first statement, would you trade $10 billion dollars for 3 weeks of serial time at the current margin?
Definitely not! Alignment is currently talent and time constrained, and very much not funding constrained. I don’t even know what we’d buy that’d be worth $10 billion. Maybe some people have some good ideas. Perhaps we could buy lots of compute? But we can already buy lots of compute. I don’t know why we aren’t, but I doubt its because we can’t afford it.
Maybe I’d trade a day for $10 billion? I don’t think I’d trade 2 days for $20 billion though. Maybe I’m just not imaginative enough. Any ideas yourself?
If you would not trade $10 billion for 3 weeks that could be because:
I’m more optimistic about empirical research / think the time iterating at the end when we have the systems is significantly more important than the time now when we can only try to reason about them.
you think money will be much less useful than I expect it to be
I wouldn’t trade $10 billion, but I think empirical research is good. It just seems like we can already afford a bunch of the stuff we want, and I expect we will continue to get lots of money without needing to sacrifice 3 weeks.
I also think people are generally bad consequentialists on questions like these. There is an obvious loss, and a speculative gain. The speculative gain looks very shiny because you make lots of money and end up doing something cool. The obvious loss does not seem very important because its not immediately world destroying, and somewhat boring.
This is an interesting post, and I’m glad you wrote it.
People are driven by incentives, which can be created with cash in a variety of ways
I agree with these ways, but I think it’s quite hard to manage incentives properly. You mention DARPA, but DARPA is a major bureacracy comprised of people who are aligned to their own incentive structures, and ultimately the most powerful organization in the world (the US government). Such a thing does not exist in AI safety—not even close. Money would certainly help with this, but it certainly can’t just be straightforwardly turned into good research.
Where is that money going to come from? How are they going to fine-tune the latest models when OpenAI is spending billions?
It’s unclear to me that having EA people starting an AI startup is more tractable than convincing other people that the work is worth funding. It certainly works faster to convince people who already have money now, vs. creating people who might make money later. I don’t really have a strong opinion, but this doesn’t seem wholly justified.
It’s frustratingly difficult to predict what will actually be useful for AI Safety...But money is flexible. It’s hard to imagine a world where another billion doesn’t come in handy.
I don’t see how the flexibility of money makes any difference? Isn’t it frustratingly difficult to predict which uses of money will actually be useful for AI safety? In that case, you still have the same problem.
It’s unclear to me that having EA people starting an AI startup is more tractable than convincing other people that the work is worth funding
Yeah, this is unclear to me to. But you can encourage lots of people to pursue earn-to-give paths (maybe a few will succeed). Not many are in a position to persuade people, and more people having this as an explicit goal seems dangerous.
Also, as an undergraduate student with short timelines, the startup path seems like a better fit.
I don’t see how the flexibility of money makes any difference? Isn’t it frustratingly difficult to predict which uses of money will actually be useful for AI safety? In that case, you still have the same problem.
I have to make important career decisions right now. It’s hard to know what will be useful in the future, but it seems likely that money will be. I could have made that point clearer.
I agree with the general premise of earning to give through entrepreneurship.
I’ve never been very convinced by the talent-constraint concept. With the right wage you can hire talent. I think the push from earning to give has been a mistake.
Where do you draw the line between AI startups that do vs don’t contribute excessively to capabilities externalities and existential risk? I think you’re right that your particular startup wouldn’t have a significant effect of accelerating timelines. But if we’re thinking AI startups in general, this could be another OpenAI or Adept, which probably have more of an effect on timelines.
I could imagine that even if one’s startup doesn’t working on scaling and making models generally smarter, a relatively small amount of applications work to make them more useful could put them at notably greater risk of having dangerous capabilities or intent. As an example, imagine if OpenAI only made GPT-3 and never produced InstructGPT or ChatGPT. It feels a lot harder to steer GPT-3 to do useful things, so I think that there would have been noticeably less adoption of LLMs and interest in advancing their capabilities, at least for a while. (For clarification, my claim isn’t that InstructGPT and ChatGPT necessarily contributed to existential risk, but they do have capabilities externalities and I think affected timelines, in part due to the hype they generated.)
If your claim is that ‘applying AI models for economically valuable tasks seems dangerous, i.e. the AIs themselves could be dangerous’ then I agree. A scrappy applications company might be more likely to end the world than OpenAI/DeepMind… it seems like it would be good, then, if more of these companies were run by safety conscious people.
A separate claim is the one about capabilities externalities. I basically agree that AI startups will have capabilities externalities, even if I don’t expect them to be very large. The question, then, is how much expected money we would be trading for expected time and what is the relative value between these two currencies.
Thanks for writing this. Do you already have an idea(s) for such a startup? I’d be happy to forward the doc to friends who fit what you’re looking for but it might go farther if you included potential ideas you’re deliberating on.
FWIW, I see a huge potential for AI startups to do a lot of good + make money in India. The amount of inefficiencies + market size bc of population + increasing exposure to the digital + Indian-specific things like the need to translate b/w 30+ languages = equals tons of opportunity. Ofc a lot of ppl are working on capitalizing on such (ik ppl working on applications of AI in law, education, healthcare) but there’s still a lot of low-middle hanging fruit.
For what it’s worth, I don’t think this needed a retraction. It’s true the original post was pretty overconfident about things. Instead of asserting something and defending it, it would probably have been better to assert it with the explicit aim of hearing criticism from people on here. That’s what happened anyway, but your framing was more “here is a thing I think is good.”
Post summary (feel free to suggest edits!): AI startups can be big money-makers, particularly as capabilities scale. The author argues that money is key to AI safety, because money:
Can convert into talent (eg. via funding AI safety industry labs, offering compute to safety researchers, and funding competitions, grants, and fellowships). Doubly so if the bottleneck becomes engineering talent and datasets instead of creative researchers.
Can convert into influence (eg. lobbying, buying board seats, soft power).
Is flexible and always useful.
The author thinks another $10B AI company would be unlikely to counterfactually accelerate timelines by more than a few weeks, and that money / reduced time to AGI tradeoff seems worth it. They also argue that the transformative potential of AI is becoming well-known, and now is the time to act to benefit from our foresight on it. They’re looking for a full-stack developer as a cofounder.
(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)
In general, it is a bad idea to trade increased probability that the world ends for money if your goal is to decrease probability that the world ends. People are usually bad at this kind of consequentialism, and this definitely strikes my ‘galaxybrain take’ detector.
And to the “but we’ll do it safer than the others” or “we’ll use our increased capabilities for alignment!” responses, I refer you to Nate’s excellent post rebutting that line of thought.
I interpreted this post as the author saying that they thought general AI capabilities would be barely advanced by this kind of thing, if they were advanced by it at all. The author doesn’t seem to suggest building an AGI startup, but rather some kind of AI application startup.
I’m curious if you think your reasoning applies to anything with a small chance of accelerating timelines by a small amount, or if you instead disagree with the object-level claim that such a startup would only have a small chance of accelerating timelines by a small amount.
I think it has a large chance of accelerating timelines by a small amount, and a small chance of accelerating timelines by a large amount. You can definitely increase capabilities, even if they’re not doing research directly into increasing the size of our language models. Figuring out how you milk language models for all the capabilities they have, the limits of such milking, and making highly capable APIs easy for language models to use are all things which shorten timelines. You go from needing a super duper AGI to take over the world to a barely super-human AGI, if all it can do is output text.
Adjacently, contributing to the AGI hype shortens timelines too.
I also think the above assumes monetary or prestige pressures won’t cause organizational value drift. I think its quite likely whoever starts this will see pressure from funders, staff, and others to turn it into an AGI firm. You need good reason to believe your firm is going to not cave in, and I see nothing addressing this concern in the original post.
Also, from what I’ve heard, you cannot in fact use ungodly amounts of money to move talent. Generally, if top researchers were swayable that way, they’d be working in industry. Mostly, they just like working on their research, and don’t care much about how much they’re paid.
Thanks for the referral. I agree that the distinction between serial time and parallel time is important and that serial time is more valuable. I’m not sure if it is astronomically more valuable though. There are two points we could have differing views on:
- the amount of expected serial time a successful (let’s say $10 billion dollar) AI startup is likely to counterfactually burn. In the post I claimed that this seems unlikely to be more than a few weeks. Would you agree with this?
- the relative value of serial time to money (which is exchangeable with parallel time). If you agree with the first statement, would you trade $10 billion dollars for 3 weeks of serial time at the current margin?
If you would not trade $10 billion for 3 weeks that could be because:
- I’m more optimistic about empirical research / think the time iterating later when we have the systems is significantly more important than the time now when we can only try to reason about them.
- you think money will be much less useful than I expect it to be
I would be interested in pinning down where I disagree with you (and others who probably disagree with the post for similar reasons).
No, see my comment above. Its the difference between a super duper AGI and only a super-human AGI, which could be years or months (but very very critical months!). Plus whatever you add to the hype, plus worlds where you somehow make $10 billion from this are also worlds where you’ve had an inordinate impact, which makes me more suspicious the $10 billion company world is the one where someone decided to just make the company another AGI lab.
Definitely not! Alignment is currently talent and time constrained, and very much not funding constrained. I don’t even know what we’d buy that’d be worth $10 billion. Maybe some people have some good ideas. Perhaps we could buy lots of compute? But we can already buy lots of compute. I don’t know why we aren’t, but I doubt its because we can’t afford it.
Maybe I’d trade a day for $10 billion? I don’t think I’d trade 2 days for $20 billion though. Maybe I’m just not imaginative enough. Any ideas yourself?
Didn’t see the second part there.
I wouldn’t trade $10 billion, but I think empirical research is good. It just seems like we can already afford a bunch of the stuff we want, and I expect we will continue to get lots of money without needing to sacrifice 3 weeks.
I also think people are generally bad consequentialists on questions like these. There is an obvious loss, and a speculative gain. The speculative gain looks very shiny because you make lots of money and end up doing something cool. The obvious loss does not seem very important because its not immediately world destroying, and somewhat boring.
This is an interesting post, and I’m glad you wrote it.
I agree with these ways, but I think it’s quite hard to manage incentives properly. You mention DARPA, but DARPA is a major bureacracy comprised of people who are aligned to their own incentive structures, and ultimately the most powerful organization in the world (the US government). Such a thing does not exist in AI safety—not even close. Money would certainly help with this, but it certainly can’t just be straightforwardly turned into good research.
It’s unclear to me that having EA people starting an AI startup is more tractable than convincing other people that the work is worth funding. It certainly works faster to convince people who already have money now, vs. creating people who might make money later. I don’t really have a strong opinion, but this doesn’t seem wholly justified.
I don’t see how the flexibility of money makes any difference? Isn’t it frustratingly difficult to predict which uses of money will actually be useful for AI safety? In that case, you still have the same problem.
Yeah, this is unclear to me to. But you can encourage lots of people to pursue earn-to-give paths (maybe a few will succeed). Not many are in a position to persuade people, and more people having this as an explicit goal seems dangerous.
Also, as an undergraduate student with short timelines, the startup path seems like a better fit.
I have to make important career decisions right now. It’s hard to know what will be useful in the future, but it seems likely that money will be. I could have made that point clearer.
I agree with the general premise of earning to give through entrepreneurship.
I’ve never been very convinced by the talent-constraint concept. With the right wage you can hire talent. I think the push from earning to give has been a mistake.
Where do you draw the line between AI startups that do vs don’t contribute excessively to capabilities externalities and existential risk? I think you’re right that your particular startup wouldn’t have a significant effect of accelerating timelines. But if we’re thinking AI startups in general, this could be another OpenAI or Adept, which probably have more of an effect on timelines.
I could imagine that even if one’s startup doesn’t working on scaling and making models generally smarter, a relatively small amount of applications work to make them more useful could put them at notably greater risk of having dangerous capabilities or intent. As an example, imagine if OpenAI only made GPT-3 and never produced InstructGPT or ChatGPT. It feels a lot harder to steer GPT-3 to do useful things, so I think that there would have been noticeably less adoption of LLMs and interest in advancing their capabilities, at least for a while. (For clarification, my claim isn’t that InstructGPT and ChatGPT necessarily contributed to existential risk, but they do have capabilities externalities and I think affected timelines, in part due to the hype they generated.)
If your claim is that ‘applying AI models for economically valuable tasks seems dangerous, i.e. the AIs themselves could be dangerous’ then I agree. A scrappy applications company might be more likely to end the world than OpenAI/DeepMind… it seems like it would be good, then, if more of these companies were run by safety conscious people.
A separate claim is the one about capabilities externalities. I basically agree that AI startups will have capabilities externalities, even if I don’t expect them to be very large. The question, then, is how much expected money we would be trading for expected time and what is the relative value between these two currencies.
Thanks for writing this. Do you already have an idea(s) for such a startup? I’d be happy to forward the doc to friends who fit what you’re looking for but it might go farther if you included potential ideas you’re deliberating on.
FWIW, I see a huge potential for AI startups to do a lot of good + make money in India. The amount of inefficiencies + market size bc of population + increasing exposure to the digital + Indian-specific things like the need to translate b/w 30+ languages = equals tons of opportunity. Ofc a lot of ppl are working on capitalizing on such (ik ppl working on applications of AI in law, education, healthcare) but there’s still a lot of low-middle hanging fruit.
Yep, I have some ideas. Please DM me and give some info about yourself if you are interested in hearing them :)
For what it’s worth, I don’t think this needed a retraction. It’s true the original post was pretty overconfident about things. Instead of asserting something and defending it, it would probably have been better to assert it with the explicit aim of hearing criticism from people on here. That’s what happened anyway, but your framing was more “here is a thing I think is good.”
Post summary (feel free to suggest edits!):
AI startups can be big money-makers, particularly as capabilities scale. The author argues that money is key to AI safety, because money:
Can convert into talent (eg. via funding AI safety industry labs, offering compute to safety researchers, and funding competitions, grants, and fellowships). Doubly so if the bottleneck becomes engineering talent and datasets instead of creative researchers.
Can convert into influence (eg. lobbying, buying board seats, soft power).
Is flexible and always useful.
The author thinks another $10B AI company would be unlikely to counterfactually accelerate timelines by more than a few weeks, and that money / reduced time to AGI tradeoff seems worth it. They also argue that the transformative potential of AI is becoming well-known, and now is the time to act to benefit from our foresight on it. They’re looking for a full-stack developer as a cofounder.
(If you’d like to see more summaries of top EA and LW forum posts, check out the Weekly Summaries series.)
I think I agree with this. I’ve thought of starting one myself. Not sure if I will yet.
Here, I wrote about how AI applied to non-profits could be neglected at the moment.