Under purely longtermist views, accelerating AI by 1 year increases available cosmic resources by 1 part in 10 billion. This is tiny.
Tiny compared to what? Are you assuming we can take some other action whose consequences don’t wash out over the long-term, e.g. because of a value lock-in? In general, these assumptions just seem quite weak and underspecified to me.
What exactly is the alternative action that has vastly greater value in expectation, and why does it have greater value? If what you mean is that we can try to reduce the risk of extinction instead, keep in mind that my first bullet point preempted pretty much this exact argument:
Unlike existential risk from other sources (e.g. an asteroid) AI x-risk is unique because humans would be replaced by other beings, rather than completely dying out. This means you can’t simply apply a naive argument that AI threatens total extinction of value to make the case that AI safety is astronomically important, in the sense that you can for other x-risks. You generally need additional assumptions.
What exactly is the alternative action that has vastly greater value in expectation, and why does it have greater value?
Ensuring human control throughout the singularity rather than having AIs get control very obviously has relatively massive effects. Of course, we can debate the sign here, I’m just making a claim about the magnitude.
I’m not talking about extinction of all smart beings on earth (AIs and humans), which seems like a small fraction of existential risk.
(Separately, the badness of such extinction seems maybe somewhat overrated because pretty likely intelligent life will just re-evolve in the next 300 million years. Intelligent life doesn’t seem that contingent. Also aliens.)
I think it remains the case that the value of accelerating AI progress is tiny relative to other apparently available interventions, such as ensuring that AIs are sentient or improving their expected well-being conditional on their being sentient. The case for focusing on how a transformative technology unfolds, rather than on when it unfolds,[1] seems robust to a relatively wide range of technologies and assumptions. Still, this seems worth further investigation.
Indeed, it seems that when the transformation unfolds is primarily important because of how it unfolds, insofar as the quality of a transformation is partly determined by its timing.
I’m claiming that it is not actually clear that we can take actions that don’t merely wash out over the long-term. In this case, you cannot simply assume that we can meaningfully and predictably affect how valuable the long-term future will be in, for example, billions of years. I agree that, yes, if you assume we can meaningfully affect the very long-run, then all actions that merely have short-term effects will have “tiny” impacts by comparison. But the assumption that we can meaningfully and predictably affect the long-run is precisely the thing that needs to be argued. I think it’s important for EAs to try to be more rigorous about their empirical claims here.
Moreover, actions that have short-term effects can generally be assumed to have longer term effects if our actions propagate. For example, support for larger population sizes now would presumably increase the probability that larger population sizes exist in the very long run, compared to the alternative of smaller population sizes with high per capita incomes. It seems arbitrary to assume this effect will be negligible but then also assume other competing effects won’t be negligible. I don’t see any strong arguments for this position.
I was trying to hint at prima facie plausible ways in which the present generation can increase the value of the long-term future by more than one part in billions, rather than “assume” that this is the case, though of course I never gave anything resembling a rigorous argument.
I do agree that the “washing out” hypothesis is a reasonable default and that one needs a positive reason for expecting our present actions to persist into the long-term. One seemingly plausible mechanism is influencing how a transformative technology unfolds: it seems that the first generation that creates AGI has significantly more influence on how much artificial sentience there is in the universe a trillion years from now than, say, the millionth generation. Do you disagree with this claim?
I’m not sure I understand the point you make in the second paragraph. What would be the predictable long-term effects of hastening the arrival of AGI in the short-term?
I was trying to hint at prima facie plausible ways in which the present generation can increase the value of the long-term future by more than one part in billions, rather than “assume” that this is the case, though of course I never gave anything resembling a rigorous argument.
As I understand, the argument originally given was that there was a tiny effect of pushing for AI acceleration, which seems outweighed by unnamed and gigantic “indirect” effects in the long-run from alternative strategies of improving the long-run future. I responded by trying to get more clarity on what these gigantic indirect effects actually are, how we can predictably bring them about, and why we would think it’s plausible that we could bring them about in the first place. From my perspective, the shape of this argument looks something like:
Your action X has this tiny positive near-term effect (ETA: or a tiny direct effect)
My action Y has this large positive long-term effect (ETA: or a large indirect effect)
Therefore, Y is better than X.
Do you see the flaw here? Well, both X and Y could have long-term effects! So, it’s not sufficient to compare the short-term effect of X to the long-term effect of Y. You need to compare both effects, on both time horizons. As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).
More generally, I think you’re probably trying to point to some concept you think is obvious and clear here, and I’m not seeing it, which is why I’m asking you to be more precise and rigorous about what you’re actually claiming.
I’m not sure I understand the point you make in the second paragraph. What would be the predictable long-term effect of hastening the arrival of AGI in the short-term?
In my original comment I pointed towards a mechanism. Here’s a more precise characterization of the argument:
Total utilitarianism generally supports, all else being equal, larger population sizes with low per capita incomes over small population sizes with high per capita incomes.
To the extent that our actions do not “wash out”, it seems reasonable to assume that pushing for large population sizes now would make it more likely in the long-run that we get large population sizes with low per-capita incomes compared to a small population size with high per capita incomes. (Keep in mind here that I’m not making any claim about the total level of resources.)
To respond to this argument you could say that in fact our actions do “wash out” here, so as to make the effect of pushing for larger population sizes rather small in the long run. But in response to that argument, I claim that this objection can be reversed and applied to almost any alternative strategy for improving the future that you might think is actually better. (In other words, I actually need to see your reasons for why there’s an asymmetry here; and I don’t currently see these reasons.)
Alternatively, you could just say that total utilitarianism is unreasonable and a bad ethical theory, but my original comment was about analyzing the claim about accelerating AI from the perspective of total utilitarianism, which, as a theory, seems to be relatively popular among EAs. So I’d prefer to keep this discussion grounded within that context.
Yes, I agree that we should consider the long-term effects of each intervention when comparing them. I focused on the short-term effects of hastening AI progress because it is those effects that are normally cited as the relevant justification in EA/utilitarian discussions of that intervention. For instance, those are the effects that Bostrom considers in ‘Astronomical waste’. Conceivably, there is a separate argument that appeals to the beneficial long-term effects of AI capability acceleration. I haven’t considered this argument because I haven’t seen many people make it, so I assume that accelerationist types tend to believe that the short-term effects dominate.
I think Bostrom’s argument merely compares a pure x-risk (such as a huge asteroid hurtling towards Earth) relative to technological acceleration, and then concludes that reducing the probability of a pure x-risk is more important because the x-risk threatens the eventual colonization of the universe. I agree with this argument in the case of a pure x-risk, but as I noted in my original comment, I don’t think that AI risk is a pure x-risk.
If, by contrast, all we’re doing by doing AI safety research is influencing something like “the values of the agents in society in the future” (and not actually influencing the probability of eventual colonization), then this action seems to plausibly just wash out in the long-term. In this case, it seems very appropriate to compare the short-term effects of AI safety to the short-term effects of acceleration.
Let me put it another way. We can think about two (potentially competing) strategies for making the future better, along with their relevant short and possible long-term effects:
Doing AI safety research
Short-term effects: makes it more likely that AIs are kind to current or near-future humans
Possible long-term effect: makes it more likely that AIs in the very long-run will share the values of the human species, relative to some unaligned alternative
Accelerating AI
Short-term effect: helps current humans by hastening the arrival of advanced technology
Possible long-term effect: makes it more likely that we have a large population size at low per capita incomes, relative to a low population size with high per capita income
My opinion is that both of these long-term effects are very speculative, so it’s generally better to focus on a heuristic of doing what’s better in the short-term, while keeping the long-term consequences in mind. And when I do that, I do not come to a strong conclusion that AI safety research “beats” AI acceleration, from a total utilitarian perspective.
Your action X has this tiny positive near-term effect.
My action Y has this large positive long-term effect.
Therefore, Y is better than X.
To be clear, this wasn’t the structure of my original argument (though it might be Pablo’s). My argument was more like “you seem to be implying that action X is good because of its direct effect (literal first order acceleration), but actually the direct effect is small when considered in a particular perspective (longtermism), so for the that perspective we need to consideer indirect effects and the analysis for that looks pretty different”.
Note that I wasn’t trying really trying argue much about the sign of the indirect effect, though people have indeed discussed this in some detail in various contexts.
I agree your original argument was slightly different than the form I stated. I was speaking too loosely, and conflated what I thought Pablo might be thinking with what you stated originally.
I think the important claim from my comment is “As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).”
I think the important claim from my comment is “As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).”
Explicitly confirming that this seems right to me.
Moreover, actions that have short-term effects can generally be assumed to have longer term effects if our actions propagate.
I don’t disagree with this. I was just claiming that the “indirect” effects dominate (by indirect, I just mean effects other than shifting the future closer in time).
There is still the question of indirect/direct effects.
I was just claiming that the “indirect” effects dominate (by indirect, I just mean effects other than shifting the future closer in time).
I understand that. I wanted to know why you thought that. I’m asking for clarity. I don’t currently understand your reasons. See this recent comment of mine for more info.
Tiny compared to what? Are you assuming we can take some other action whose consequences don’t wash out over the long-term, e.g. because of a value lock-in? In general, these assumptions just seem quite weak and underspecified to me.
What exactly is the alternative action that has vastly greater value in expectation, and why does it have greater value? If what you mean is that we can try to reduce the risk of extinction instead, keep in mind that my first bullet point preempted pretty much this exact argument:
Ensuring human control throughout the singularity rather than having AIs get control very obviously has relatively massive effects. Of course, we can debate the sign here, I’m just making a claim about the magnitude.
I’m not talking about extinction of all smart beings on earth (AIs and humans), which seems like a small fraction of existential risk.
(Separately, the badness of such extinction seems maybe somewhat overrated because pretty likely intelligent life will just re-evolve in the next 300 million years. Intelligent life doesn’t seem that contingent. Also aliens.)
For what it’s worth, I think my reply to Pablo here responds to your comment fairly adequately too.
I think it remains the case that the value of accelerating AI progress is tiny relative to other apparently available interventions, such as ensuring that AIs are sentient or improving their expected well-being conditional on their being sentient. The case for focusing on how a transformative technology unfolds, rather than on when it unfolds,[1] seems robust to a relatively wide range of technologies and assumptions. Still, this seems worth further investigation.
Indeed, it seems that when the transformation unfolds is primarily important because of how it unfolds, insofar as the quality of a transformation is partly determined by its timing.
I’m claiming that it is not actually clear that we can take actions that don’t merely wash out over the long-term. In this case, you cannot simply assume that we can meaningfully and predictably affect how valuable the long-term future will be in, for example, billions of years. I agree that, yes, if you assume we can meaningfully affect the very long-run, then all actions that merely have short-term effects will have “tiny” impacts by comparison. But the assumption that we can meaningfully and predictably affect the long-run is precisely the thing that needs to be argued. I think it’s important for EAs to try to be more rigorous about their empirical claims here.
Moreover, actions that have short-term effects can generally be assumed to have longer term effects if our actions propagate. For example, support for larger population sizes now would presumably increase the probability that larger population sizes exist in the very long run, compared to the alternative of smaller population sizes with high per capita incomes. It seems arbitrary to assume this effect will be negligible but then also assume other competing effects won’t be negligible. I don’t see any strong arguments for this position.
I was trying to hint at prima facie plausible ways in which the present generation can increase the value of the long-term future by more than one part in billions, rather than “assume” that this is the case, though of course I never gave anything resembling a rigorous argument.
I do agree that the “washing out” hypothesis is a reasonable default and that one needs a positive reason for expecting our present actions to persist into the long-term. One seemingly plausible mechanism is influencing how a transformative technology unfolds: it seems that the first generation that creates AGI has significantly more influence on how much artificial sentience there is in the universe a trillion years from now than, say, the millionth generation. Do you disagree with this claim?
I’m not sure I understand the point you make in the second paragraph. What would be the predictable long-term effects of hastening the arrival of AGI in the short-term?
As I understand, the argument originally given was that there was a tiny effect of pushing for AI acceleration, which seems outweighed by unnamed and gigantic “indirect” effects in the long-run from alternative strategies of improving the long-run future. I responded by trying to get more clarity on what these gigantic indirect effects actually are, how we can predictably bring them about, and why we would think it’s plausible that we could bring them about in the first place. From my perspective, the shape of this argument looks something like:
Your action X has this tiny positive near-term effect (ETA: or a tiny direct effect)
My action Y has this large positive long-term effect (ETA: or a large indirect effect)
Therefore, Y is better than X.
Do you see the flaw here? Well, both X and Y could have long-term effects! So, it’s not sufficient to compare the short-term effect of X to the long-term effect of Y. You need to compare both effects, on both time horizons. As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).
More generally, I think you’re probably trying to point to some concept you think is obvious and clear here, and I’m not seeing it, which is why I’m asking you to be more precise and rigorous about what you’re actually claiming.
In my original comment I pointed towards a mechanism. Here’s a more precise characterization of the argument:
Total utilitarianism generally supports, all else being equal, larger population sizes with low per capita incomes over small population sizes with high per capita incomes.
To the extent that our actions do not “wash out”, it seems reasonable to assume that pushing for large population sizes now would make it more likely in the long-run that we get large population sizes with low per-capita incomes compared to a small population size with high per capita incomes. (Keep in mind here that I’m not making any claim about the total level of resources.)
To respond to this argument you could say that in fact our actions do “wash out” here, so as to make the effect of pushing for larger population sizes rather small in the long run. But in response to that argument, I claim that this objection can be reversed and applied to almost any alternative strategy for improving the future that you might think is actually better. (In other words, I actually need to see your reasons for why there’s an asymmetry here; and I don’t currently see these reasons.)
Alternatively, you could just say that total utilitarianism is unreasonable and a bad ethical theory, but my original comment was about analyzing the claim about accelerating AI from the perspective of total utilitarianism, which, as a theory, seems to be relatively popular among EAs. So I’d prefer to keep this discussion grounded within that context.
Thanks for the clarification.
Yes, I agree that we should consider the long-term effects of each intervention when comparing them. I focused on the short-term effects of hastening AI progress because it is those effects that are normally cited as the relevant justification in EA/utilitarian discussions of that intervention. For instance, those are the effects that Bostrom considers in ‘Astronomical waste’. Conceivably, there is a separate argument that appeals to the beneficial long-term effects of AI capability acceleration. I haven’t considered this argument because I haven’t seen many people make it, so I assume that accelerationist types tend to believe that the short-term effects dominate.
I think Bostrom’s argument merely compares a pure x-risk (such as a huge asteroid hurtling towards Earth) relative to technological acceleration, and then concludes that reducing the probability of a pure x-risk is more important because the x-risk threatens the eventual colonization of the universe. I agree with this argument in the case of a pure x-risk, but as I noted in my original comment, I don’t think that AI risk is a pure x-risk.
If, by contrast, all we’re doing by doing AI safety research is influencing something like “the values of the agents in society in the future” (and not actually influencing the probability of eventual colonization), then this action seems to plausibly just wash out in the long-term. In this case, it seems very appropriate to compare the short-term effects of AI safety to the short-term effects of acceleration.
Let me put it another way. We can think about two (potentially competing) strategies for making the future better, along with their relevant short and possible long-term effects:
Doing AI safety research
Short-term effects: makes it more likely that AIs are kind to current or near-future humans
Possible long-term effect: makes it more likely that AIs in the very long-run will share the values of the human species, relative to some unaligned alternative
Accelerating AI
Short-term effect: helps current humans by hastening the arrival of advanced technology
Possible long-term effect: makes it more likely that we have a large population size at low per capita incomes, relative to a low population size with high per capita income
My opinion is that both of these long-term effects are very speculative, so it’s generally better to focus on a heuristic of doing what’s better in the short-term, while keeping the long-term consequences in mind. And when I do that, I do not come to a strong conclusion that AI safety research “beats” AI acceleration, from a total utilitarian perspective.
To be clear, this wasn’t the structure of my original argument (though it might be Pablo’s). My argument was more like “you seem to be implying that action X is good because of its direct effect (literal first order acceleration), but actually the direct effect is small when considered in a particular perspective (longtermism), so for the that perspective we need to consideer indirect effects and the analysis for that looks pretty different”.
Note that I wasn’t trying really trying argue much about the sign of the indirect effect, though people have indeed discussed this in some detail in various contexts.
I agree your original argument was slightly different than the form I stated. I was speaking too loosely, and conflated what I thought Pablo might be thinking with what you stated originally.
I think the important claim from my comment is “As far as I can tell, I haven’t seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don’t see what the exact argument is).”
Explicitly confirming that this seems right to me.
I don’t disagree with this. I was just claiming that the “indirect” effects dominate (by indirect, I just mean effects other than shifting the future closer in time).
There is still the question of indirect/direct effects.
I understand that. I wanted to know why you thought that. I’m asking for clarity. I don’t currently understand your reasons. See this recent comment of mine for more info.
(I don’t think I’m going to engage further here, sorry.)