tldr; wrote some responses to sections, don’t think I have an overall point. I think this line of argumentation deserves to be taken seriously but think this post is maybe trying to do too much at once. The main argument is simply cluelessness + short term positive EV.
In virtually every other area of human decision-making, people generally accept without much argument that the very-long-term consequences of our actions are extremely difficult to predict.
I’m a little confused what your argumentative technique is here. Is that fact that most humans do something the core component here? Wouldn’t this immediately disqualify much of what EAs work on? Or is this just a persuasive technique, and you mean ~ “most humans think this for reason x. I also think this for reason x, though the fact most humans think it matters little to me.” For me, most humans do x is not an especially convincing argument of something.
I don’t want to get bogged down on cluelessness because there are many lengthy discussions elsewhere but I’ll say that cluelessness depends on the question. If you told me what the rainforest looked like and then asked me to guess the animals I wouldn’t have a chance. If you asked me to guess if they ate food and drank water I think I would do decent. Or a more on the nose example. If you took me back to 5 million years ago and asked me to guess what would happen to the chimps if humans came to exist, I wouldn’t be able to predict much specifics, but I might be able to predict (1) humans would become the top dog, and with less certainty (2) chimp population would go down and with even less certainty (3) chimps will go extinct. That’s why the horse model gets so much play, people have some level of belief that there are certain outcomes that might be less chaotic if modeled correctly.
To wrap up I think your first 4 paragraphs could be shortened to your unique views on cluelessness (specifically wrt ai?) + discount rates/whatever other unique philosophical axioms you might hold.
Understood in this way, AI does not actually pose a risk of astronomical catastrophe in Bostrom’s sense.
To be clear, neither does the asteroid. Aliens might exist and our survival similarly presents a risk of replacement for all the alien civs that won’t have time to biologically evolve as (humans or ai from earth) speed through the lightcone. Also even if no aliens, we have no idea if conditional on humans being grabby, utility is net positive or negative. There isn’t even agreement on this forum or in the world on if there is such a thing as a negative life or not. Don’t think i’m arguing against you here but feels like you are being a little loose here (don’t want to be too pedantic as I can totally understand if you are writing for a more general audience).
Now, you might still reasonably be very concerned about such a replacement catastrophe. I myself share that concern and take the possibility seriously. But it is crucial to keep the structure of the original argument clearly in mind. … Even if you accept that killing eight billion people would be an extraordinarily terrible outcome, it does not automatically follow that this harm carries the same moral weight as a catastrophe that permanently eliminates the possibility of 10^23 future lives.
Well I have my own “values”. Just because I die doesn’t mean these disappear. I’d prefer that those 10^23 lives aren’t horrifically tortured for instance.
Though I say this with extremely weak confidence, I feel like in the case where a “single agent/hivemind” misaligned ai immediately wipes us all out, I’m thinking they probably are not going to convert resources into utility as efficiently as me (by my current values), and thus this might be viewed as an s-risk. I’m guessing you might say that we can’t possibly predict that, but then can we even predict if those 10^23 lives will be positive or negative? if not I guess i’m not sure why you brought any of this up anyway. Bostrom’s whole argument predicates on the assumption that earth descended life is + ev, which predicates on not being clueless or having a very kumbaya pronatal moral philosophy.
So I guess even better for you, from my POV you don’t even need to counter argue this.
Virtually every proposed mechanism by which AI systems might cause human extinction relies on the assumption that these AI systems would be extraordinarily capable, productive, or technologically sophisticated.
I might not be especially up to date here. Can’t it like cause a nuclear fallout etc? totalitarian lock in? the matrix? Extreme wealth and power disparity? is there agreement that the only scenarios in which our potential is permanently curtailed the terminator flavors?
The reason is that a decade of delayed progress would mean that nearly a billion people will die from diseases and age-related decline who might otherwise have been saved by the rapid medical advances that AI could enable. Those billion people would have gone on to live much longer, healthier, and more prosperous lives.
You might need to flesh this out a bit more for me because I don’t think it’s as true as you said. Is the claim here that AI will (1) invent new medicine or (2) replace doctors or (3) improve US healthcare policy?
(1) Drug development pipelines are excruciatingly long and mostly not because of a lack of hypotheses. For instance, https://pmc.ncbi.nlm.nih.gov/articles/PMC10786682/ GLP-1 have been in the pipelines for half a century (though debatably with better ai some of the nausea stuff could have been figured out quicker). IL-23 connection to IBD/Crohns was basically known ~2000 as it was one of the first/most significant single nucleotide mutations picked up with GWAS phenotype/genotype studies. Yet Skyrizi only hit the market a few years ago. Assuming ai could instantly just invent the drugs, IIRC it’s a minimum of like 7 years to get approval. That’s absolute minimum. And likely even super intelligent AI is gonna need physical labs, iteration, make mistakes, etc.
Assuming sufficient AGI in 2030 for this threshold, we are looking at early 2040s before we start to see significant impact on the drugs we use, although it’s possible AI will usher a new era of repurposes drug cocktails via extremely good lit review (although IMO the current tools might already be enough to see huge benefits here!).
(2) Doctors, while overpaid, still only make up like 10-15% of healthcare costs in the US. I do think ai will end up being better than them, although whether people will quickly accept this idk. So you can get some nice savings there, but again that’s assuming you just break the massive lobbying power they have. And beyond the costs, tons of the most important health stuff is already widely known among the public. Stuff like don’t smoke cigarettes, don’t drink alcohol, don’t be fat, don’t be lonely. People still fail to do this stuff. Not an information problem. Further doctors often know when they are overprescribing useless stuff, often just an incentives problem. No good reason to think AI will break this trend unless you are envisioning a completely decentralized or single payer system that uses all ai doctors, both are at least partially political issues not intelligence. And if we are talking solid basic primary care for the developing world, I just question how smart the ai needs to be. I’d assume a 130 iq llm with perfect vision and full knowledge of medical lit would be more than sufficient, and that seems like it will be the next major gemini release?
(3) will leave this for now.
Kinda got sidetracked here and will leave this comment here for now because so long, but I guess takeaway from this section: You can’t claim cluelessness on the harms and then assume the benefits are guaranteed.
tldr; wrote some responses to sections, don’t think I have an overall point. I think this line of argumentation deserves to be taken seriously but think this post is maybe trying to do too much at once. The main argument is simply cluelessness + short term positive EV.
I’m a little confused what your argumentative technique is here. Is that fact that most humans do something the core component here? Wouldn’t this immediately disqualify much of what EAs work on? Or is this just a persuasive technique, and you mean ~ “most humans think this for reason x. I also think this for reason x, though the fact most humans think it matters little to me.”
For me, most humans do x is not an especially convincing argument of something.
I don’t want to get bogged down on cluelessness because there are many lengthy discussions elsewhere but I’ll say that cluelessness depends on the question. If you told me what the rainforest looked like and then asked me to guess the animals I wouldn’t have a chance. If you asked me to guess if they ate food and drank water I think I would do decent. Or a more on the nose example. If you took me back to 5 million years ago and asked me to guess what would happen to the chimps if humans came to exist, I wouldn’t be able to predict much specifics, but I might be able to predict (1) humans would become the top dog, and with less certainty (2) chimp population would go down and with even less certainty (3) chimps will go extinct. That’s why the horse model gets so much play, people have some level of belief that there are certain outcomes that might be less chaotic if modeled correctly.
To wrap up I think your first 4 paragraphs could be shortened to your unique views on cluelessness (specifically wrt ai?) + discount rates/whatever other unique philosophical axioms you might hold.
To be clear, neither does the asteroid. Aliens might exist and our survival similarly presents a risk of replacement for all the alien civs that won’t have time to biologically evolve as (humans or ai from earth) speed through the lightcone. Also even if no aliens, we have no idea if conditional on humans being grabby, utility is net positive or negative. There isn’t even agreement on this forum or in the world on if there is such a thing as a negative life or not. Don’t think i’m arguing against you here but feels like you are being a little loose here (don’t want to be too pedantic as I can totally understand if you are writing for a more general audience).
Well I have my own “values”. Just because I die doesn’t mean these disappear. I’d prefer that those 10^23 lives aren’t horrifically tortured for instance.
Though I say this with extremely weak confidence, I feel like in the case where a “single agent/hivemind” misaligned ai immediately wipes us all out, I’m thinking they probably are not going to convert resources into utility as efficiently as me (by my current values), and thus this might be viewed as an s-risk. I’m guessing you might say that we can’t possibly predict that, but then can we even predict if those 10^23 lives will be positive or negative? if not I guess i’m not sure why you brought any of this up anyway. Bostrom’s whole argument predicates on the assumption that earth descended life is + ev, which predicates on not being clueless or having a very kumbaya pronatal moral philosophy.
So I guess even better for you, from my POV you don’t even need to counter argue this.
I might not be especially up to date here. Can’t it like cause a nuclear fallout etc? totalitarian lock in? the matrix? Extreme wealth and power disparity? is there agreement that the only scenarios in which our potential is permanently curtailed the terminator flavors?
You might need to flesh this out a bit more for me because I don’t think it’s as true as you said. Is the claim here that AI will (1) invent new medicine or (2) replace doctors or (3) improve US healthcare policy?
(1) Drug development pipelines are excruciatingly long and mostly not because of a lack of hypotheses. For instance, https://pmc.ncbi.nlm.nih.gov/articles/PMC10786682/ GLP-1 have been in the pipelines for half a century (though debatably with better ai some of the nausea stuff could have been figured out quicker). IL-23 connection to IBD/Crohns was basically known ~2000 as it was one of the first/most significant single nucleotide mutations picked up with GWAS phenotype/genotype studies. Yet Skyrizi only hit the market a few years ago. Assuming ai could instantly just invent the drugs, IIRC it’s a minimum of like 7 years to get approval. That’s absolute minimum. And likely even super intelligent AI is gonna need physical labs, iteration, make mistakes, etc.
Assuming sufficient AGI in 2030 for this threshold, we are looking at early 2040s before we start to see significant impact on the drugs we use, although it’s possible AI will usher a new era of repurposes drug cocktails via extremely good lit review (although IMO the current tools might already be enough to see huge benefits here!).
(2) Doctors, while overpaid, still only make up like 10-15% of healthcare costs in the US. I do think ai will end up being better than them, although whether people will quickly accept this idk. So you can get some nice savings there, but again that’s assuming you just break the massive lobbying power they have. And beyond the costs, tons of the most important health stuff is already widely known among the public. Stuff like don’t smoke cigarettes, don’t drink alcohol, don’t be fat, don’t be lonely. People still fail to do this stuff. Not an information problem. Further doctors often know when they are overprescribing useless stuff, often just an incentives problem. No good reason to think AI will break this trend unless you are envisioning a completely decentralized or single payer system that uses all ai doctors, both are at least partially political issues not intelligence. And if we are talking solid basic primary care for the developing world, I just question how smart the ai needs to be. I’d assume a 130 iq llm with perfect vision and full knowledge of medical lit would be more than sufficient, and that seems like it will be the next major gemini release?
(3) will leave this for now.
Kinda got sidetracked here and will leave this comment here for now because so long, but I guess takeaway from this section: You can’t claim cluelessness on the harms and then assume the benefits are guaranteed.