@EliezerYudkowsky has suggested nanobots and I could think of some other possibilities but I think they’re infohazards so I’m not going to share them.
More broadly, my expectation is that a superintelligent AI would be able to do anything that a large group of intelligent and motivated humans could do, and that includes causing human extinction.
Nanobots are a terrible method for world destruction, given that they have not been invented yet. Speaking as a computational physicist, there are some things you simply cannot do accurately without experimentation, and I am certain that building nanobot factories is one of them.
I think if you actually want to convince people that AI x-risk is a threat, you unavoidably have to provide a realistic scenario of takeover. I don’t understand why doing so would be an “infohazard”, unless you think that a human could pull off your plan?
It’s very important if you believe the AI will have limitations or will care even a little bit about efficiency. Developing an entirely new field of engineering from scratch is a highly difficult task that likely requires significant amounts of experimentation and resources to get right. I’m not sure if nanomachines as envisaged by Drexler are even possible, but even if they are, it’s definitely impossible to design them well from first principles computation alone.
Compare that to something like designing a powerful virus: a lot of the work to get there has already been done by nature, you have significant amounts of experiments and data available on viruses and how they spread, etc. This is a path that, while still incredibly difficult, is clearly far easier than non-existent nanomachines.
I’m not talking about tinkering in someone’s backyard, making nanomachines feasible would require ridiculous amounts of funding and resources over many many years. It’s an extremely risky plan that provides signficant amount of risk of exposure.
Why would an AI choose this plan, instead of something with a much lower footprint like bio-weapons?
If you can convince me of the “many many years” claim, that would be an update. Other than that you are just saying things I already know and believe.
I never claimed that nanotech would be the best plan, nor that it would be Yudkowky’s bathtub-nanotech scenario instead of a scenario involving huge amounts of experimentation. I was just reacting to your terrible leaps of logic, e.g. “nanobots are a terrible method for world destruction given that they have not been invented yet” and “making nanobots requires experimentation and resources therefore AIs won’t do it.” (I agree that if it takes many many years, there will surely be a faster method than nanobots, but you haven’t really argued for that.)
I’d love to see some sort of quantitative estimate from you of how long it would take modern civilization to build nanotech if it really tried. Like, suppose nanotech became the new Hot Thing starting now and all the genius engineers currently at SpaceX and various other places united to make nanotech startups, funded by huge amounts of government funding and VC investment, etc. And suppose the world otherwise remains fairly static, so e.g. climate change doesn’t kill us, AGI doesn’t happen, etc. How many years until we have the sorts of things Drexler described? (Assume that they are possible)
“nanobots are a terrible method for world destruction given that they have not been invented yet” and “making nanobots requires experimentation and resources therefore AIs won’t do it.”
These are both statements I still believe are true. None of them are “terrible leaps of logic”, as I have patiently explained the logic behind them with arguments. I do not appreciate the lack of charity you have displayed here.
I’d love to see some sort of quantitative estimate from you of how long it would take modern civilization to build nanotech if it really tried. Like, suppose nanotech became the new Hot Thing starting now and all the genius engineers currently at SpaceX and various other places united to make nanotech startups, funded by huge amounts of government funding and VC investment, etc. And suppose the world otherwise remains fairly static, so e.g. climate change doesn’t kill us, AGI doesn’t happen, etc. How many years until we have the sorts of things Drexler described? (Assume that they are possible)
Well, I think theres a pretty decent chance that they are impossible. See this post for several reasons why. If they are possible, I would suspect it would take decades at the least to make something that is useful for anyone, and also that the results would still fail to live up to the nigh-magical expectations set by science fiction scenarios. The most likely scenario involves making a toy nanobot system in a lab somewhere that is stupidly expensive to make and doesn’t work that well, which eventually finds some niche applications in medicine or something.
Re: uncharitability: I think I was about as uncharitable as you were. That said, I do apologize—I should hold myself to a higher standard.
I agree they might be impossible. (If it only finds some niche application in medicine, that means it’s impossible, btw. Anything remotely similar to what Drexler described would be much more revolutionary than that.)
If they are possible though, and it takes (say) 50 years for ordinary human scientists to figure it out starting now… then it’s quite plausible to me that it could take 2 OOMs less time than that, or possibly even 4 OOMs, for superintelligent AI scientists to figure it out starting whenever superintelligent AI scientists appear (assuming they have access to proper experimental facilities. I am very uncertain about how large such facilities would need to be.) 2 OOMs less time would be 6 months; 4 OOMs would be Yudkowsky’s bathtub nanotech scenario (except not necessarily in a single bathtub, presumably it’s much more likely to be feasible if they have access to lots of laboratories). I also think it’s plausible that even for a superintelligence it would take at least 5 years (only 1 OOM speedup over humans). (again, conditional on it being possible at all + taking about 50 years for ordinary human scientists) A crux for me here would be if you could show that deciding what experiments to run and interpreting the results are both pretty easy for ordinary human scientists, and that the bottleneck is basically just getting the funding and time to run all the experiments.
To be clear I’m pretty uncertain about all this. I’m prompting you with stuff like this to try to elicit your expertise, and get you to give arguments or intuition pumps that might address my cruxes.
Yes, the plans that I have in mind could also be hypothetically executed by humans and I don’t think it’s a good idea to spread those ideas. BTW I am not personally especially convinced by the nanobot argument, either.
Are you able to use your imagination to think of ways that a well-resourced and motivated group of humans could cause human extinction? If so, is there a reason to think that an AI wouldn’t be able to execute the same plan?
@EliezerYudkowsky has suggested nanobots and I could think of some other possibilities but I think they’re infohazards so I’m not going to share them.
More broadly, my expectation is that a superintelligent AI would be able to do anything that a large group of intelligent and motivated humans could do, and that includes causing human extinction.
Nanobots are a terrible method for world destruction, given that they have not been invented yet. Speaking as a computational physicist, there are some things you simply cannot do accurately without experimentation, and I am certain that building nanobot factories is one of them.
I think if you actually want to convince people that AI x-risk is a threat, you unavoidably have to provide a realistic scenario of takeover. I don’t understand why doing so would be an “infohazard”, unless you think that a human could pull off your plan?
A superintelligent AI is able to invent new things. Whether a thing has been invented or not previously is not that important.
It’s very important if you believe the AI will have limitations or will care even a little bit about efficiency. Developing an entirely new field of engineering from scratch is a highly difficult task that likely requires significant amounts of experimentation and resources to get right. I’m not sure if nanomachines as envisaged by Drexler are even possible, but even if they are, it’s definitely impossible to design them well from first principles computation alone.
Compare that to something like designing a powerful virus: a lot of the work to get there has already been done by nature, you have significant amounts of experiments and data available on viruses and how they spread, etc. This is a path that, while still incredibly difficult, is clearly far easier than non-existent nanomachines.
A superintelligent AI will be able to do significant amounts of experimentation and acquire significant amounts of resources.
I’m not talking about tinkering in someone’s backyard, making nanomachines feasible would require ridiculous amounts of funding and resources over many many years. It’s an extremely risky plan that provides signficant amount of risk of exposure.
Why would an AI choose this plan, instead of something with a much lower footprint like bio-weapons?
If you can convince me of the “many many years” claim, that would be an update. Other than that you are just saying things I already know and believe.
I never claimed that nanotech would be the best plan, nor that it would be Yudkowky’s bathtub-nanotech scenario instead of a scenario involving huge amounts of experimentation. I was just reacting to your terrible leaps of logic, e.g. “nanobots are a terrible method for world destruction given that they have not been invented yet” and “making nanobots requires experimentation and resources therefore AIs won’t do it.” (I agree that if it takes many many years, there will surely be a faster method than nanobots, but you haven’t really argued for that.)
I’d love to see some sort of quantitative estimate from you of how long it would take modern civilization to build nanotech if it really tried. Like, suppose nanotech became the new Hot Thing starting now and all the genius engineers currently at SpaceX and various other places united to make nanotech startups, funded by huge amounts of government funding and VC investment, etc. And suppose the world otherwise remains fairly static, so e.g. climate change doesn’t kill us, AGI doesn’t happen, etc. How many years until we have the sorts of things Drexler described? (Assume that they are possible)
These are both statements I still believe are true. None of them are “terrible leaps of logic”, as I have patiently explained the logic behind them with arguments. I do not appreciate the lack of charity you have displayed here.
Well, I think theres a pretty decent chance that they are impossible. See this post for several reasons why. If they are possible, I would suspect it would take decades at the least to make something that is useful for anyone, and also that the results would still fail to live up to the nigh-magical expectations set by science fiction scenarios. The most likely scenario involves making a toy nanobot system in a lab somewhere that is stupidly expensive to make and doesn’t work that well, which eventually finds some niche applications in medicine or something.
Re: uncharitability: I think I was about as uncharitable as you were. That said, I do apologize—I should hold myself to a higher standard.
I agree they might be impossible. (If it only finds some niche application in medicine, that means it’s impossible, btw. Anything remotely similar to what Drexler described would be much more revolutionary than that.)
If they are possible though, and it takes (say) 50 years for ordinary human scientists to figure it out starting now… then it’s quite plausible to me that it could take 2 OOMs less time than that, or possibly even 4 OOMs, for superintelligent AI scientists to figure it out starting whenever superintelligent AI scientists appear (assuming they have access to proper experimental facilities. I am very uncertain about how large such facilities would need to be.) 2 OOMs less time would be 6 months; 4 OOMs would be Yudkowsky’s bathtub nanotech scenario (except not necessarily in a single bathtub, presumably it’s much more likely to be feasible if they have access to lots of laboratories). I also think it’s plausible that even for a superintelligence it would take at least 5 years (only 1 OOM speedup over humans). (again, conditional on it being possible at all + taking about 50 years for ordinary human scientists) A crux for me here would be if you could show that deciding what experiments to run and interpreting the results are both pretty easy for ordinary human scientists, and that the bottleneck is basically just getting the funding and time to run all the experiments.
To be clear I’m pretty uncertain about all this. I’m prompting you with stuff like this to try to elicit your expertise, and get you to give arguments or intuition pumps that might address my cruxes.
Yes, the plans that I have in mind could also be hypothetically executed by humans and I don’t think it’s a good idea to spread those ideas. BTW I am not personally especially convinced by the nanobot argument, either.
Are you able to use your imagination to think of ways that a well-resourced and motivated group of humans could cause human extinction? If so, is there a reason to think that an AI wouldn’t be able to execute the same plan?