Anyway, I’m a huge fan of 95% of EA’s work, but really think it has gone down the wrong path with longtermism. Sorry for the sass—much love to all :)
It’s all good! Seriously, I really appreciate the engagement from you and Vaden: it’s obvious that you both care a lot and are offering the criticism precisely because of that. I currently think you’re mistaken about some of the substance, but this kind of dialogue is the type of thing which can help to keep EA intellectually healthy.
I’m confused about the claim
>I don’t think they’re saying (and I certainly don’t think) that we can ignore the effects of our actions over the next century; rather I think those effects matter much more for their instrumental value than intrinsic value.
This seems in direct opposition to what the authors say (and what Vaden quoted above), that
>The idea, then, is that for the purposes of evaluating actions, we can in the first instance often simply ignore all the effects contained in the first 100 (or even 1000) years
I understand that they may not feel this way, but it is what they argued for and is, consequently, the idea that deserves to be criticized.
So my interpretation had been that they were using a technical sense of “evaluating actions”, meaning something like “if we had access to full information about consequences, how would we decide which ones were actually good”.
However, on a close read I see that they’re talking about ex ante effects. This makes me think that this is at least confusingly explained, and perhaps confused. It now seems most probable to me that they mean something like “we can ignore the effects of the actions contained in the first 100 years, except insofar as those feed into our understanding of the longer-run effects”. But the “except insofar …” clause would be concealing a lot, since 100 years is so long that almost all of our understanding of the longer-run effects must go via guesses about the long-term goodness of the shorter-run effects.
[As an aside, I’ve been planning to write a post about some related issues; maybe I’ll move it up my priority stack.]
The “immeasurability” of the future that Vaden has highlighted has nothing to do with the literal finiteness of the timeline of the universe. It has to do, rather, with the set of all possible futures (which is provably infinite). This set is immeasurable in the mathematical sense of lacking sufficient structure to be operated upon with a well-defined probability measure. Let me turn the question around on you: Suppose we knew that the time-horizon of the universe was finite, can you write out the sample space, $\sigma$-algebra, and measure which allows us to compute over possible futures?
I like the question; I think this may be getting at something deep, and I want to think more about it.
Nonetheless, my first response was: while I can’t write this down, if we helped ourselves to some cast-iron guarantees about the size and future lifespan of the universe (and made some assumptions about quantization) then we’d know that the set of possible futures was smaller than a particular finite number (since there would only be a finite number of time steps and a finite number of ways of arranging all particles at each time step). Then even if I can’t write it down, in principle someone could write it down, and the mathematical worries about undefined expectations go away.
The reason I want to think more about it is that I think there’s something interesting about the interplay between objective and subjective probabilities here. How much should it help me as a boundedly rational actor to know that in theory a fully rational actor could put a measure on things, if it’s practically immeasurable for me?
Considering that the Open Philanthropy Project has poured millions into AI Safety, that its listed as a top cause by 80K, and that EA’s far-future-fund makes payouts to AI safety work, if Shivani’s reasoning isn’t to be taken seriously then now is probably a good time to make that abundantly clear. Apologies for the harshness in tone here, but for an august institute like GPI to make normative suggestions in its research and then expect no one to act on them is irresponsible.
Sorry, I made an error here in just reading Vaden’s quotation of Shivani’s reasoning rather than looking at it in full context.
In the construction of the argument in the paper Shivani is explicitly trying to compare the long-term effects of action A to the short-term effects of action B (which was selected to have particularly good short-term effects). The paper argues that there are several cases where the former is larger than the latter. It doesn’t follow that A is overall better than B, because the long-term effects of B are unexamined.
The comparison of of AMF to AI safety that was quoted felt like a toy example to me because it obviously wasn’t trying to be a full comparison between the two, but was rather being used to illustrate a particular point. (I think maybe the word “toy” is not quite right.)
In any case I consider it a minor fault of the paper that one could read just the section quoted and reasonably come away with the impression that comparing the short-term number of lives saved by AMF with the long-term number of lives expected to be saved by investing in AI safety was the right way to compare between those two opportunities. (Indeed one could come away with the impression that the AMF price to save a life was the long-run price, but in the structure of the argument being used they need it to be just the short-term price.)
Note that I do think AI safety is very important, and I endorse the actions of the various organisations you mention. But I don’t think that comparing some long-term expectation on one side with a short-term expectation on the other is the right argument for justifying this (particularly versions which make the ratio-of-goodness scale directly with estimates of the size of the future), and that was the part I was objecting to. (I think this argument is sometimes seen in earnest “in the wild”, and arguably on account of that the paper should take extra steps to make it clear that it is not the argument being made.)
It’s all good! Seriously, I really appreciate the engagement from you and Vaden: it’s obvious that you both care a lot and are offering the criticism precisely because of that. I currently think you’re mistaken about some of the substance, but this kind of dialogue is the type of thing which can help to keep EA intellectually healthy.
So my interpretation had been that they were using a technical sense of “evaluating actions”, meaning something like “if we had access to full information about consequences, how would we decide which ones were actually good”.
However, on a close read I see that they’re talking about ex ante effects. This makes me think that this is at least confusingly explained, and perhaps confused. It now seems most probable to me that they mean something like “we can ignore the effects of the actions contained in the first 100 years, except insofar as those feed into our understanding of the longer-run effects”. But the “except insofar …” clause would be concealing a lot, since 100 years is so long that almost all of our understanding of the longer-run effects must go via guesses about the long-term goodness of the shorter-run effects.
[As an aside, I’ve been planning to write a post about some related issues; maybe I’ll move it up my priority stack.]
I like the question; I think this may be getting at something deep, and I want to think more about it.
Nonetheless, my first response was: while I can’t write this down, if we helped ourselves to some cast-iron guarantees about the size and future lifespan of the universe (and made some assumptions about quantization) then we’d know that the set of possible futures was smaller than a particular finite number (since there would only be a finite number of time steps and a finite number of ways of arranging all particles at each time step). Then even if I can’t write it down, in principle someone could write it down, and the mathematical worries about undefined expectations go away.
The reason I want to think more about it is that I think there’s something interesting about the interplay between objective and subjective probabilities here. How much should it help me as a boundedly rational actor to know that in theory a fully rational actor could put a measure on things, if it’s practically immeasurable for me?
Sorry, I made an error here in just reading Vaden’s quotation of Shivani’s reasoning rather than looking at it in full context.
In the construction of the argument in the paper Shivani is explicitly trying to compare the long-term effects of action A to the short-term effects of action B (which was selected to have particularly good short-term effects). The paper argues that there are several cases where the former is larger than the latter. It doesn’t follow that A is overall better than B, because the long-term effects of B are unexamined.
The comparison of of AMF to AI safety that was quoted felt like a toy example to me because it obviously wasn’t trying to be a full comparison between the two, but was rather being used to illustrate a particular point. (I think maybe the word “toy” is not quite right.)
In any case I consider it a minor fault of the paper that one could read just the section quoted and reasonably come away with the impression that comparing the short-term number of lives saved by AMF with the long-term number of lives expected to be saved by investing in AI safety was the right way to compare between those two opportunities. (Indeed one could come away with the impression that the AMF price to save a life was the long-run price, but in the structure of the argument being used they need it to be just the short-term price.)
Note that I do think AI safety is very important, and I endorse the actions of the various organisations you mention. But I don’t think that comparing some long-term expectation on one side with a short-term expectation on the other is the right argument for justifying this (particularly versions which make the ratio-of-goodness scale directly with estimates of the size of the future), and that was the part I was objecting to. (I think this argument is sometimes seen in earnest “in the wild”, and arguably on account of that the paper should take extra steps to make it clear that it is not the argument being made.)