In general, it is a bad idea to trade increased probability that the world ends for money if your goal is to decrease probability that the world ends. People are usually bad at this kind of consequentialism, and this definitely strikes my ‘galaxybrain take’ detector.
I interpreted this post as the author saying that they thought general AI capabilities would be barely advanced by this kind of thing, if they were advanced by it at all. The author doesn’t seem to suggest building an AGI startup, but rather some kind of AI application startup.
I’m curious if you think your reasoning applies to anything with a small chance of accelerating timelines by a small amount, or if you instead disagree with the object-level claim that such a startup would only have a small chance of accelerating timelines by a small amount.
I think it has a large chance of accelerating timelines by a small amount, and a small chance of accelerating timelines by a large amount. You can definitely increase capabilities, even if they’re not doing research directly into increasing the size of our language models. Figuring out how you milk language models for all the capabilities they have, the limits of such milking, and making highly capable APIs easy for language models to use are all things which shorten timelines. You go from needing a super duper AGI to take over the world to a barely super-human AGI, if all it can do is output text.
Adjacently, contributing to the AGI hype shortens timelines too.
I also think the above assumes monetary or prestige pressures won’t cause organizational value drift. I think its quite likely whoever starts this will see pressure from funders, staff, and others to turn it into an AGI firm. You need good reason to believe your firm is going to not cave in, and I see nothing addressing this concern in the original post.
Also, from what I’ve heard, you cannot in fact use ungodly amounts of money to move talent. Generally, if top researchers were swayable that way, they’d be working in industry. Mostly, they just like working on their research, and don’t care much about how much they’re paid.
Thanks for the referral. I agree that the distinction between serial time and parallel time is important and that serial time is more valuable. I’m not sure if it is astronomically more valuable though. There are two points we could have differing views on:
- the amount of expected serial time a successful (let’s say $10 billion dollar) AI startup is likely to counterfactually burn. In the post I claimed that this seems unlikely to be more than a few weeks. Would you agree with this? - the relative value of serial time to money (which is exchangeable with parallel time). If you agree with the first statement, would you trade $10 billion dollars for 3 weeks of serial time at the current margin?
If you would not trade $10 billion for 3 weeks that could be because: - I’m more optimistic about empirical research / think the time iterating later when we have the systems is significantly more important than the time now when we can only try to reason about them. - you think money will be much less useful than I expect it to be
I would be interested in pinning down where I disagree with you (and others who probably disagree with the post for similar reasons).
the amount of expected serial time a successful (let’s say $10 billion dollar) AI startup is likely to counterfactually burn. In the post I claimed that this seems unlikely to be more than a few weeks. Would you agree with this?
No, see my comment above. Its the difference between a super duper AGI and only a super-human AGI, which could be years or months (but very very critical months!). Plus whatever you add to the hype, plus worlds where you somehow make $10 billion from this are also worlds where you’ve had an inordinate impact, which makes me more suspicious the $10 billion company world is the one where someone decided to just make the company another AGI lab.
the relative value of serial time to money (which is exchangeable with parallel time). If you agree with the first statement, would you trade $10 billion dollars for 3 weeks of serial time at the current margin?
Definitely not! Alignment is currently talent and time constrained, and very much not funding constrained. I don’t even know what we’d buy that’d be worth $10 billion. Maybe some people have some good ideas. Perhaps we could buy lots of compute? But we can already buy lots of compute. I don’t know why we aren’t, but I doubt its because we can’t afford it.
Maybe I’d trade a day for $10 billion? I don’t think I’d trade 2 days for $20 billion though. Maybe I’m just not imaginative enough. Any ideas yourself?
If you would not trade $10 billion for 3 weeks that could be because:
I’m more optimistic about empirical research / think the time iterating at the end when we have the systems is significantly more important than the time now when we can only try to reason about them.
you think money will be much less useful than I expect it to be
I wouldn’t trade $10 billion, but I think empirical research is good. It just seems like we can already afford a bunch of the stuff we want, and I expect we will continue to get lots of money without needing to sacrifice 3 weeks.
I also think people are generally bad consequentialists on questions like these. There is an obvious loss, and a speculative gain. The speculative gain looks very shiny because you make lots of money and end up doing something cool. The obvious loss does not seem very important because its not immediately world destroying, and somewhat boring.
In general, it is a bad idea to trade increased probability that the world ends for money if your goal is to decrease probability that the world ends. People are usually bad at this kind of consequentialism, and this definitely strikes my ‘galaxybrain take’ detector.
And to the “but we’ll do it safer than the others” or “we’ll use our increased capabilities for alignment!” responses, I refer you to Nate’s excellent post rebutting that line of thought.
I interpreted this post as the author saying that they thought general AI capabilities would be barely advanced by this kind of thing, if they were advanced by it at all. The author doesn’t seem to suggest building an AGI startup, but rather some kind of AI application startup.
I’m curious if you think your reasoning applies to anything with a small chance of accelerating timelines by a small amount, or if you instead disagree with the object-level claim that such a startup would only have a small chance of accelerating timelines by a small amount.
I think it has a large chance of accelerating timelines by a small amount, and a small chance of accelerating timelines by a large amount. You can definitely increase capabilities, even if they’re not doing research directly into increasing the size of our language models. Figuring out how you milk language models for all the capabilities they have, the limits of such milking, and making highly capable APIs easy for language models to use are all things which shorten timelines. You go from needing a super duper AGI to take over the world to a barely super-human AGI, if all it can do is output text.
Adjacently, contributing to the AGI hype shortens timelines too.
I also think the above assumes monetary or prestige pressures won’t cause organizational value drift. I think its quite likely whoever starts this will see pressure from funders, staff, and others to turn it into an AGI firm. You need good reason to believe your firm is going to not cave in, and I see nothing addressing this concern in the original post.
Also, from what I’ve heard, you cannot in fact use ungodly amounts of money to move talent. Generally, if top researchers were swayable that way, they’d be working in industry. Mostly, they just like working on their research, and don’t care much about how much they’re paid.
Thanks for the referral. I agree that the distinction between serial time and parallel time is important and that serial time is more valuable. I’m not sure if it is astronomically more valuable though. There are two points we could have differing views on:
- the amount of expected serial time a successful (let’s say $10 billion dollar) AI startup is likely to counterfactually burn. In the post I claimed that this seems unlikely to be more than a few weeks. Would you agree with this?
- the relative value of serial time to money (which is exchangeable with parallel time). If you agree with the first statement, would you trade $10 billion dollars for 3 weeks of serial time at the current margin?
If you would not trade $10 billion for 3 weeks that could be because:
- I’m more optimistic about empirical research / think the time iterating later when we have the systems is significantly more important than the time now when we can only try to reason about them.
- you think money will be much less useful than I expect it to be
I would be interested in pinning down where I disagree with you (and others who probably disagree with the post for similar reasons).
No, see my comment above. Its the difference between a super duper AGI and only a super-human AGI, which could be years or months (but very very critical months!). Plus whatever you add to the hype, plus worlds where you somehow make $10 billion from this are also worlds where you’ve had an inordinate impact, which makes me more suspicious the $10 billion company world is the one where someone decided to just make the company another AGI lab.
Definitely not! Alignment is currently talent and time constrained, and very much not funding constrained. I don’t even know what we’d buy that’d be worth $10 billion. Maybe some people have some good ideas. Perhaps we could buy lots of compute? But we can already buy lots of compute. I don’t know why we aren’t, but I doubt its because we can’t afford it.
Maybe I’d trade a day for $10 billion? I don’t think I’d trade 2 days for $20 billion though. Maybe I’m just not imaginative enough. Any ideas yourself?
Didn’t see the second part there.
I wouldn’t trade $10 billion, but I think empirical research is good. It just seems like we can already afford a bunch of the stuff we want, and I expect we will continue to get lots of money without needing to sacrifice 3 weeks.
I also think people are generally bad consequentialists on questions like these. There is an obvious loss, and a speculative gain. The speculative gain looks very shiny because you make lots of money and end up doing something cool. The obvious loss does not seem very important because its not immediately world destroying, and somewhat boring.