I was trying to hint at prima facie plausible ways in which the present generation can increase the value of the long-term future by more than one part in billions, rather than “assume” that this is the case, though of course I never gave anything resembling a rigorous argument.
As I understand, the argument originally given was that there was a tiny effect of pushing for AI acceleration, which seems outweighed by unnamed and gigantic “indirect” effects in the long-run from alternative strategies of improving the long-run future. I responded by trying to get more clarity on what these gigantic indirect effects actually are, how we can predictably bring them about, and why we would think it’s plausible that we could bring them about in the first place. From my perspective, the shape of this argument looks something like:
Your action X has this tiny positive near-term effect (ETA: or a tiny direct effect)
My action Y has this large positive long-term effect (ETA: or a large indirect effect)
Therefore, Y is better than X.
Do you see the flaw here? Well, both X and Y could have long-term effects! So, it’s not sufficient to compare the short-term effect of X to the long-term effect of Y. You need to compare both effects, on both time horizons. As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).
More generally, I think you’re probably trying to point to some concept you think is obvious and clear here, and I’m not seeing it, which is why I’m asking you to be more precise and rigorous about what you’re actually claiming.
I’m not sure I understand the point you make in the second paragraph. What would be the predictable long-term effect of hastening the arrival of AGI in the short-term?
In my original comment I pointed towards a mechanism. Here’s a more precise characterization of the argument:
Total utilitarianism generally supports, all else being equal, larger population sizes with low per capita incomes over small population sizes with high per capita incomes.
To the extent that our actions do not “wash out”, it seems reasonable to assume that pushing for large population sizes now would make it more likely in the long-run that we get large population sizes with low per-capita incomes compared to a small population size with high per capita incomes. (Keep in mind here that I’m not making any claim about the total level of resources.)
To respond to this argument you could say that in fact our actions do “wash out” here, so as to make the effect of pushing for larger population sizes rather small in the long run. But in response to that argument, I claim that this objection can be reversed and applied to almost any alternative strategy for improving the future that you might think is actually better. (In other words, I actually need to see your reasons for why there’s an asymmetry here; and I don’t currently see these reasons.)
Alternatively, you could just say that total utilitarianism is unreasonable and a bad ethical theory, but my original comment was about analyzing the claim about accelerating AI from the perspective of total utilitarianism, which, as a theory, seems to be relatively popular among EAs. So I’d prefer to keep this discussion grounded within that context.
Yes, I agree that we should consider the long-term effects of each intervention when comparing them. I focused on the short-term effects of hastening AI progress because it is those effects that are normally cited as the relevant justification in EA/utilitarian discussions of that intervention. For instance, those are the effects that Bostrom considers in ‘Astronomical waste’. Conceivably, there is a separate argument that appeals to the beneficial long-term effects of AI capability acceleration. I haven’t considered this argument because I haven’t seen many people make it, so I assume that accelerationist types tend to believe that the short-term effects dominate.
I think Bostrom’s argument merely compares a pure x-risk (such as a huge asteroid hurtling towards Earth) relative to technological acceleration, and then concludes that reducing the probability of a pure x-risk is more important because the x-risk threatens the eventual colonization of the universe. I agree with this argument in the case of a pure x-risk, but as I noted in my original comment, I don’t think that AI risk is a pure x-risk.
If, by contrast, all we’re doing by doing AI safety research is influencing something like “the values of the agents in society in the future” (and not actually influencing the probability of eventual colonization), then this action seems to plausibly just wash out in the long-term. In this case, it seems very appropriate to compare the short-term effects of AI safety to the short-term effects of acceleration.
Let me put it another way. We can think about two (potentially competing) strategies for making the future better, along with their relevant short and possible long-term effects:
Doing AI safety research
Short-term effects: makes it more likely that AIs are kind to current or near-future humans
Possible long-term effect: makes it more likely that AIs in the very long-run will share the values of the human species, relative to some unaligned alternative
Accelerating AI
Short-term effect: helps current humans by hastening the arrival of advanced technology
Possible long-term effect: makes it more likely that we have a large population size at low per capita incomes, relative to a low population size with high per capita income
My opinion is that both of these long-term effects are very speculative, so it’s generally better to focus on a heuristic of doing what’s better in the short-term, while keeping the long-term consequences in mind. And when I do that, I do not come to a strong conclusion that AI safety research “beats” AI acceleration, from a total utilitarian perspective.
Your action X has this tiny positive near-term effect.
My action Y has this large positive long-term effect.
Therefore, Y is better than X.
To be clear, this wasn’t the structure of my original argument (though it might be Pablo’s). My argument was more like “you seem to be implying that action X is good because of its direct effect (literal first order acceleration), but actually the direct effect is small when considered in a particular perspective (longtermism), so for the that perspective we need to consideer indirect effects and the analysis for that looks pretty different”.
Note that I wasn’t trying really trying argue much about the sign of the indirect effect, though people have indeed discussed this in some detail in various contexts.
I agree your original argument was slightly different than the form I stated. I was speaking too loosely, and conflated what I thought Pablo might be thinking with what you stated originally.
I think the important claim from my comment is “As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).”
I think the important claim from my comment is “As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).”
Explicitly confirming that this seems right to me.
As I understand, the argument originally given was that there was a tiny effect of pushing for AI acceleration, which seems outweighed by unnamed and gigantic “indirect” effects in the long-run from alternative strategies of improving the long-run future. I responded by trying to get more clarity on what these gigantic indirect effects actually are, how we can predictably bring them about, and why we would think it’s plausible that we could bring them about in the first place. From my perspective, the shape of this argument looks something like:
Your action X has this tiny positive near-term effect (ETA: or a tiny direct effect)
My action Y has this large positive long-term effect (ETA: or a large indirect effect)
Therefore, Y is better than X.
Do you see the flaw here? Well, both X and Y could have long-term effects! So, it’s not sufficient to compare the short-term effect of X to the long-term effect of Y. You need to compare both effects, on both time horizons. As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).
More generally, I think you’re probably trying to point to some concept you think is obvious and clear here, and I’m not seeing it, which is why I’m asking you to be more precise and rigorous about what you’re actually claiming.
In my original comment I pointed towards a mechanism. Here’s a more precise characterization of the argument:
Total utilitarianism generally supports, all else being equal, larger population sizes with low per capita incomes over small population sizes with high per capita incomes.
To the extent that our actions do not “wash out”, it seems reasonable to assume that pushing for large population sizes now would make it more likely in the long-run that we get large population sizes with low per-capita incomes compared to a small population size with high per capita incomes. (Keep in mind here that I’m not making any claim about the total level of resources.)
To respond to this argument you could say that in fact our actions do “wash out” here, so as to make the effect of pushing for larger population sizes rather small in the long run. But in response to that argument, I claim that this objection can be reversed and applied to almost any alternative strategy for improving the future that you might think is actually better. (In other words, I actually need to see your reasons for why there’s an asymmetry here; and I don’t currently see these reasons.)
Alternatively, you could just say that total utilitarianism is unreasonable and a bad ethical theory, but my original comment was about analyzing the claim about accelerating AI from the perspective of total utilitarianism, which, as a theory, seems to be relatively popular among EAs. So I’d prefer to keep this discussion grounded within that context.
Thanks for the clarification.
Yes, I agree that we should consider the long-term effects of each intervention when comparing them. I focused on the short-term effects of hastening AI progress because it is those effects that are normally cited as the relevant justification in EA/utilitarian discussions of that intervention. For instance, those are the effects that Bostrom considers in ‘Astronomical waste’. Conceivably, there is a separate argument that appeals to the beneficial long-term effects of AI capability acceleration. I haven’t considered this argument because I haven’t seen many people make it, so I assume that accelerationist types tend to believe that the short-term effects dominate.
I think Bostrom’s argument merely compares a pure x-risk (such as a huge asteroid hurtling towards Earth) relative to technological acceleration, and then concludes that reducing the probability of a pure x-risk is more important because the x-risk threatens the eventual colonization of the universe. I agree with this argument in the case of a pure x-risk, but as I noted in my original comment, I don’t think that AI risk is a pure x-risk.
If, by contrast, all we’re doing by doing AI safety research is influencing something like “the values of the agents in society in the future” (and not actually influencing the probability of eventual colonization), then this action seems to plausibly just wash out in the long-term. In this case, it seems very appropriate to compare the short-term effects of AI safety to the short-term effects of acceleration.
Let me put it another way. We can think about two (potentially competing) strategies for making the future better, along with their relevant short and possible long-term effects:
Doing AI safety research
Short-term effects: makes it more likely that AIs are kind to current or near-future humans
Possible long-term effect: makes it more likely that AIs in the very long-run will share the values of the human species, relative to some unaligned alternative
Accelerating AI
Short-term effect: helps current humans by hastening the arrival of advanced technology
Possible long-term effect: makes it more likely that we have a large population size at low per capita incomes, relative to a low population size with high per capita income
My opinion is that both of these long-term effects are very speculative, so it’s generally better to focus on a heuristic of doing what’s better in the short-term, while keeping the long-term consequences in mind. And when I do that, I do not come to a strong conclusion that AI safety research “beats” AI acceleration, from a total utilitarian perspective.
To be clear, this wasn’t the structure of my original argument (though it might be Pablo’s). My argument was more like “you seem to be implying that action X is good because of its direct effect (literal first order acceleration), but actually the direct effect is small when considered in a particular perspective (longtermism), so for the that perspective we need to consideer indirect effects and the analysis for that looks pretty different”.
Note that I wasn’t trying really trying argue much about the sign of the indirect effect, though people have indeed discussed this in some detail in various contexts.
I agree your original argument was slightly different than the form I stated. I was speaking too loosely, and conflated what I thought Pablo might be thinking with what you stated originally.
I think the important claim from my comment is “As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).”
Explicitly confirming that this seems right to me.