I broadly endorse this reply and have mostly shifted to trying to talk about “covalently bonded” bacteria, since using the term “diamondoid” (tightly covalently bonded CHON) causes people to panic about the lack of currently known mechanosynthesis pathways for tetrahedral carbon lattices.
I broadly endorse this reply and have mostly shifted to trying to talk about “covalently bonded” bacteria
This terminology is actually significantly worse, because it makes it almost impossible for anyone to follow up on your claims. Covalent bonds are the most common type of bond in organic chemistry, and thus all existing bacteria have them in ridiculous abundance. So claiming the new technology will be “covalently bonded” does not distinguish it from existing bacteria in the slightest.
To correct some other weird definitions you made in your very short reply: “tetrahedral carbon lattice” is the literal exact same thing as diamond. The scientific definition of diamondoid is also not “tightly covalently bonded CHON”, it specifically refers to hydrocarbon variants of adamantane, which are not tetrahedral (I discussed this in the post). Also, the new technology probably would not fit the definition of “bacteria”, except in a metaphorical sense.
Now, I’m assuming you mean something like “in places where existing bacteria uses weak van der waals forces to stick together, the new tech will use stronger covalent bonds instead”. If you have specific research you are referring to, I would be interested in reading it, because, again, you have made googling the subject impossible.
My problem here would be that in a lot of cases, you actually want the forces to be weak. If you want to assemble and reassemble things, and stick them and break them, having too strong forces will make life significantly harder (this was the subject of the theoretical study I looked at). There is a reason bricklayers don’t coat their gloves in superglue when working with bricks.
As for the “tetrahedral carbon” if you are aware of other dedicated research efforts for mechanosynthesis that I have missed in my post, I would be genuinely interested in reading up on them. I did my best to look, and did my best to highlight the ones I could find in my extensively researched article which I’m unsure if you actually read.
What if he just said “Some sort of super-powerful nanofactory-like thing?”
He’s not citing some existing literature that shows how to do it, but rather citing some existing literature which should make it plausible to a reasonable judge that a million superintelligences working for a year could figure out how to do it. (If you dispute the plausibility of this, what’s your argument? We have an unfinished exchange on this point elsewhere in this comment section. Seems you agree that a galaxy full of superintelligences could do it; I feel like it’s pretty plausible that if a galaxy of superintelligences could do it, a mere million also could do it.)
What if he just said “Some sort of super-powerful nanofactory-like thing?”
I would vastly prefer this phrasing, because it would be an accurate relaying of his beliefs, and would not involve the use of scientific terms that are at best misleading and at worst active misinformation.
As for the “millions of superintelligences”, one of my main cruxes is that I do not think we will have millions of superintelligences in my lifetime. We may have lots of AGI, but I do not believe that AGI=superintelligence. Also, I think that if a few superintelligences come into existence they may prevent others from being built out of self-preservation. These points are probably out of scope here though.
I don’t think a million superintelligences could invent nanotech in a year, with only the avalaible resources on earth. Unlike the galaxy, there is limited computational power available on earth, and limited everything else as well. I do not think the sheer scale of experimentation required could be assembled in a year, without having already invented nanotech. The galaxy situation is fundamentally misleading.
Lastly, I think even if nanotech is invented, it will probably end up being disappointing or limited in some way. This tends to be the case with all technologies: Did anyone predict that when we could build an AI that could easily pass a simple turing test, but be unable to multiply large numbers together? Hypothetical technologies get to be perfect in our minds, but as something actually gets built, it accumulates shortcomings and weaknesses from the inevitable brushes with engineering.
Cool. Seems you and I are mostly agreed on terminology then.
Yeah we definitely disagree about that crux. You’ll see. Happy to talk about it more sometime if you like.
Re: galaxy vs. earth: The difference is one of degree, not kind. In both cases we have a finite amount of resources and a finite amount of time with which to do experiments. The proper way to handle this, I think, is to smear out our uncertainty over many orders of magnitude. E.g. the first OOM gets 5% of our probability mass, the second OOM gets 5% of the remaining probability mass, and so forth. Then we look at how many OOMs of extra research and testing (compared to what humans have done) a million ASIs would be able to do in a year, and compare it to how many OOMs extra (beyond that level) a galaxy worth of ASI would be able to do in many years. And crunch the numbers.
It really seems to me like the galaxy thing is just going to mislead, rather than elucidate. I can make my judgements about a system where one planet is converted into computronium, one planets contains a store of every available element, one planet is tiled completely with experimental labs doing automated experiments, etc. But the results of that hypothetical won’t scale down to what we actually care about. For example, it wouldn’t account for the infrastructure that needs to be built to assemble any of those components in bulk.
If someone wants to try their hand at modelling a more earthly scenario, I’d be happy to offer my insights. Remember, this development of nanotech has to predate the AI taking over the world, or else the whole exercise is pointless. You could look at something like “AI blackmails the dictator of a small country into starting a research program” as a starting point.
Personally, I don’t think there is very much you can be certain about, beyond: “this problem is extremely fucking hard”, and “humans aren’t cracking this anytime soon”. I think building the physical infrastructure required to properly do the research in bulk could easily take more than a year on it’s own.
I agree with the claims “this problem is extremely fucking hard” and “humans aren’t cracking this anytime soon” and I suspect Yudkowsky does too these days.
I disagree that nanotech has to predate taking over the world; that wasn’t an assumption I was making or a conclusion I was arguing for at any rate. I agree it is less likely that ASIs will make nanotech before takeover than that they will make nanotech while still on earth.
I like your suggestion to model a more earthly scenario but I lack the energy and interest to do so right now.
My closing statement is that I think your kind of reasoning would have been consistently wrong had it been used in the past—e.g. in 1600 you would have declared so many things to be impossible on the grounds that you didn’t see a way for the natural philosophers and engineers of your time to build them. Things like automobiles, flying machines, moving pictures, thinking machines, etc. It was indeed super difficult to build those things, it turns out—‘impossible’ relative to the R&D capabilities of 1600 -- but R&D capabilities improved by many OOMs, and the impossible became possible.
I disagree that nanotech has to predate taking over the world; that wasn’t an assumption I was making or a conclusion I was arguing for at any rate. I agree it is less likely that ASIs will make nanotech before takeover than that they will make nanotech while still on earth.
Sorry, to be clear, I wasn’t actually making a prediction as to whether nanotech predates AI takeover. My point is that that these discussions are in the context of the question “can nanotech be used to defeat humanity”. If AI can only invent nanotech after defeating humanity, that’s interesting but has no bearing on the question.
I also lack the energy or interest to do the modelling, so we’ll have to leave it there.
My closing statement is that I think your kind of reasoning would have been consistently wrong had it been used in the past—e.g. in 1600 you would have declared so many things to be impossible on the grounds that you didn’t see a way for the natural philosophers and engineers of your time to build them. Things like automobiles, flying machines, moving pictures, thinking machines, etc. It was indeed super difficult to build those things, it turns out—‘impossible’ relative to the R&D capabilities of 1600 -- but R&D capabilities improved by many OOMs, and the impossible became possible.
My closing rebuttal: I have never stated that I am certain that nanotech is impossible. I have only stated that it could be impossible, impractical, or disappointing, and that the timelines for development are large, and would remain so even with the advent of AGI.
If I had stated in 1600 that flying machines, moving pictures, thinking machines, etc were at least 100 years off, I would have been entirely correct and accurate. And for every great technological change that turned out to be true and transformative, there are a hundred great ideas that turned out to be prohibitively expensive, or impractical, or just plain not workable. And as for the ones that did work out, and did transform the world: it almost always took a long time to build them, once we had the ability to. And even then they started out shitty as hell, and took a long, long time to become as flawless as they are today.
I’m not saying new tech can’t change the world, I’m just saying it can’t do it instantly.
(I forgot to mention an important part of my argument, oops—You wouldn’t have said “at least 100 years off” you would have said “at least 5000 years off.” Because you are anchoring to recent-past rates of progress rather than looking at how rates of progress increase over time and extrapolating. (This is just an analogy / data point, not the key part of my argument, but look at GWP growth rates as a proxy for tech progress rates: According to this GWP doubling time was something like 600 years back then, whereas it’s more like 20 years now. So 1.5 OOMs faster.) Saying “at least a hundred years off” in 1600 would be like saying “at least 3 years off” today. Which I think is quite reasonable.)
That argument does make more sense, although it still doesn’t apply to me, as I would never confidently state a 5000 year forecast due to the inherent uncertainty of long term predictions. (My estimates for nanotech are also high uncertainty, for the record).
I broadly endorse this reply and have mostly shifted to trying to talk about “covalently bonded” bacteria, since using the term “diamondoid” (tightly covalently bonded CHON) causes people to panic about the lack of currently known mechanosynthesis pathways for tetrahedral carbon lattices.
This terminology is actually significantly worse, because it makes it almost impossible for anyone to follow up on your claims. Covalent bonds are the most common type of bond in organic chemistry, and thus all existing bacteria have them in ridiculous abundance. So claiming the new technology will be “covalently bonded” does not distinguish it from existing bacteria in the slightest.
To correct some other weird definitions you made in your very short reply: “tetrahedral carbon lattice” is the literal exact same thing as diamond. The scientific definition of diamondoid is also not “tightly covalently bonded CHON”, it specifically refers to hydrocarbon variants of adamantane, which are not tetrahedral (I discussed this in the post). Also, the new technology probably would not fit the definition of “bacteria”, except in a metaphorical sense.
Now, I’m assuming you mean something like “in places where existing bacteria uses weak van der waals forces to stick together, the new tech will use stronger covalent bonds instead”. If you have specific research you are referring to, I would be interested in reading it, because, again, you have made googling the subject impossible.
My problem here would be that in a lot of cases, you actually want the forces to be weak. If you want to assemble and reassemble things, and stick them and break them, having too strong forces will make life significantly harder (this was the subject of the theoretical study I looked at). There is a reason bricklayers don’t coat their gloves in superglue when working with bricks.
As for the “tetrahedral carbon” if you are aware of other dedicated research efforts for mechanosynthesis that I have missed in my post, I would be genuinely interested in reading up on them. I did my best to look, and did my best to highlight the ones I could find in my extensively researched article which I’m unsure if you actually read.
What if he just said “Some sort of super-powerful nanofactory-like thing?”
He’s not citing some existing literature that shows how to do it, but rather citing some existing literature which should make it plausible to a reasonable judge that a million superintelligences working for a year could figure out how to do it. (If you dispute the plausibility of this, what’s your argument? We have an unfinished exchange on this point elsewhere in this comment section. Seems you agree that a galaxy full of superintelligences could do it; I feel like it’s pretty plausible that if a galaxy of superintelligences could do it, a mere million also could do it.)
I would vastly prefer this phrasing, because it would be an accurate relaying of his beliefs, and would not involve the use of scientific terms that are at best misleading and at worst active misinformation.
As for the “millions of superintelligences”, one of my main cruxes is that I do not think we will have millions of superintelligences in my lifetime. We may have lots of AGI, but I do not believe that AGI=superintelligence. Also, I think that if a few superintelligences come into existence they may prevent others from being built out of self-preservation. These points are probably out of scope here though.
I don’t think a million superintelligences could invent nanotech in a year, with only the avalaible resources on earth. Unlike the galaxy, there is limited computational power available on earth, and limited everything else as well. I do not think the sheer scale of experimentation required could be assembled in a year, without having already invented nanotech. The galaxy situation is fundamentally misleading.
Lastly, I think even if nanotech is invented, it will probably end up being disappointing or limited in some way. This tends to be the case with all technologies: Did anyone predict that when we could build an AI that could easily pass a simple turing test, but be unable to multiply large numbers together? Hypothetical technologies get to be perfect in our minds, but as something actually gets built, it accumulates shortcomings and weaknesses from the inevitable brushes with engineering.
Cool. Seems you and I are mostly agreed on terminology then.
Yeah we definitely disagree about that crux. You’ll see. Happy to talk about it more sometime if you like.
Re: galaxy vs. earth: The difference is one of degree, not kind. In both cases we have a finite amount of resources and a finite amount of time with which to do experiments. The proper way to handle this, I think, is to smear out our uncertainty over many orders of magnitude. E.g. the first OOM gets 5% of our probability mass, the second OOM gets 5% of the remaining probability mass, and so forth. Then we look at how many OOMs of extra research and testing (compared to what humans have done) a million ASIs would be able to do in a year, and compare it to how many OOMs extra (beyond that level) a galaxy worth of ASI would be able to do in many years. And crunch the numbers.
It really seems to me like the galaxy thing is just going to mislead, rather than elucidate. I can make my judgements about a system where one planet is converted into computronium, one planets contains a store of every available element, one planet is tiled completely with experimental labs doing automated experiments, etc. But the results of that hypothetical won’t scale down to what we actually care about. For example, it wouldn’t account for the infrastructure that needs to be built to assemble any of those components in bulk.
If someone wants to try their hand at modelling a more earthly scenario, I’d be happy to offer my insights. Remember, this development of nanotech has to predate the AI taking over the world, or else the whole exercise is pointless. You could look at something like “AI blackmails the dictator of a small country into starting a research program” as a starting point.
Personally, I don’t think there is very much you can be certain about, beyond: “this problem is extremely fucking hard”, and “humans aren’t cracking this anytime soon”. I think building the physical infrastructure required to properly do the research in bulk could easily take more than a year on it’s own.
I agree with the claims “this problem is extremely fucking hard” and “humans aren’t cracking this anytime soon” and I suspect Yudkowsky does too these days.
I disagree that nanotech has to predate taking over the world; that wasn’t an assumption I was making or a conclusion I was arguing for at any rate. I agree it is less likely that ASIs will make nanotech before takeover than that they will make nanotech while still on earth.
I like your suggestion to model a more earthly scenario but I lack the energy and interest to do so right now.
My closing statement is that I think your kind of reasoning would have been consistently wrong had it been used in the past—e.g. in 1600 you would have declared so many things to be impossible on the grounds that you didn’t see a way for the natural philosophers and engineers of your time to build them. Things like automobiles, flying machines, moving pictures, thinking machines, etc. It was indeed super difficult to build those things, it turns out—‘impossible’ relative to the R&D capabilities of 1600 -- but R&D capabilities improved by many OOMs, and the impossible became possible.
Sorry, to be clear, I wasn’t actually making a prediction as to whether nanotech predates AI takeover. My point is that that these discussions are in the context of the question “can nanotech be used to defeat humanity”. If AI can only invent nanotech after defeating humanity, that’s interesting but has no bearing on the question.
I also lack the energy or interest to do the modelling, so we’ll have to leave it there.
My closing rebuttal: I have never stated that I am certain that nanotech is impossible. I have only stated that it could be impossible, impractical, or disappointing, and that the timelines for development are large, and would remain so even with the advent of AGI.
If I had stated in 1600 that flying machines, moving pictures, thinking machines, etc were at least 100 years off, I would have been entirely correct and accurate. And for every great technological change that turned out to be true and transformative, there are a hundred great ideas that turned out to be prohibitively expensive, or impractical, or just plain not workable. And as for the ones that did work out, and did transform the world: it almost always took a long time to build them, once we had the ability to. And even then they started out shitty as hell, and took a long, long time to become as flawless as they are today.
I’m not saying new tech can’t change the world, I’m just saying it can’t do it instantly.
Thanks for discussing with me!
(I forgot to mention an important part of my argument, oops—You wouldn’t have said “at least 100 years off” you would have said “at least 5000 years off.” Because you are anchoring to recent-past rates of progress rather than looking at how rates of progress increase over time and extrapolating. (This is just an analogy / data point, not the key part of my argument, but look at GWP growth rates as a proxy for tech progress rates: According to this GWP doubling time was something like 600 years back then, whereas it’s more like 20 years now. So 1.5 OOMs faster.) Saying “at least a hundred years off” in 1600 would be like saying “at least 3 years off” today. Which I think is quite reasonable.)
That argument does make more sense, although it still doesn’t apply to me, as I would never confidently state a 5000 year forecast due to the inherent uncertainty of long term predictions. (My estimates for nanotech are also high uncertainty, for the record).
no worries, I enjoyed the debate!