On addressing cluelessness—for the most part I agree with the authors’ views, which includes the view that there needs to be further research in this area.
I do find it odd however that they attempt to counter the worry of ‘simple cluelessness’ but not that of ‘complex cluelessness’ i.e. to counter the possibility that there could be semi-foreseeable unintended consequences of longtermist interventions that make us ultimately uncertain on the sign of the expected-value assessment of these interventions. Maybe they see this as obviously not an issue...but I would have appreciated some thoughts on this.
I think complex cluelessness is essentially covered by the other subsections in the Cluelessness section. It’s an issue of assigning numbers arbitrarily to the point that what you should do depends on your arbitrary beliefs. I don’t think they succeed in addressing the issue, though, since they don’t sufficiently discuss and address ways each of their proposed interventions could backfire despite our best intentions (they do discuss some in section 4, though). The bar is pretty high to satisfy any “reasonable” person.
Thanks, I really haven’t given sufficient thought to the cluelessness section which seems the most novel and tricky. Fanaticism is probably just as important, if not more so, but is also easier to get one’s head around.
I agree with you in your other comment though that the following seems to imply that the authors are not “complexly clueless” about AI safety:
For example, we don’t think any reasonable representor even contains a probability function according to which efforts to mitigate AI risk save only 0.001 lives per $100 in expectation.
I mean I guess it is probably the case that if you’re saying it’s unreasonable for a probability function associated with very small positive expected value to be contained in your representor, you’ll also say a probability function associated with negative expected value also isn’t contained in it. This does seem to me to be a slightly extreme view.
Ya, maybe your representor should be a convex set, so that for any two functions in it, you can take any probabilistic mixture of them, and that would also be in your representor. This way, if you have one with expected value x and another with expected value y, you should have functions with each possible expected value between. So, if you have positive and negative EVs in your representor, you would also have 0 EV.
Do you mean negative EV is slightly extreme or ruling out negative EV is slightly extreme?
I think neglecting to look into and address ways something could be negative (e.g. a probability difference, EV) often leads us to unjustifiably assuming a positive lower bound, and I think this is an easy mistake to make or miss. Combining a positive lower bound with astronomical stakes would make the argument appear very compelling.
Thanks for this post Jack, I found it really useful as I haven’t got round yet to reading the updated paper. This break down in the cluelessness section was a new arrangement to me. Does anyone know if this break down has been used elsewhere? If not this seems like useful progress in better defining the cluelessness objections to longtermism.
Thanks Robert. I’ve never seen this breakdown of cluelessness and it could be a useful way for further research to define the issue.
The Global Priorities Institute raised the modelling of cluelessness in their research agenda and I’m looking forward to further work on this. If interested, see below for the two research questions related to cluelessness in the GPI research agenda. I have a feeling that there is still quite a bit of research that could be conducted in this area.
------------------
Forecasting the long-term effects of our actions often requires us to make difficult comparisons between complex and messy bodies of competing evidence, a situation Greaves (2016) calls “complex cluelessness”. We must also reckon with our own incomplete awareness, that is, the likelihood that the long-run future will be shaped by events we’ve never considered and perhaps can’t fully imagine. What is the appropriate response to this sort of epistemic situation? For instance, does rationality require us to adopt precise subjective probabilities concerning the very-long-run effects of our actions, imprecise probabilities (and if so, how imprecise?), or some other sort of doxastic state entirely?
Faced with the task of comparing actions in terms of expected value, it often seems that the agent is ‘clueless’: that is, that the available empirical and theoretical evidence simply supplies too thin a basis for guiding decisions in any principled way (Lenman 2000; Greaves 2016; Mogensen 2020) (INFORMAL: Tomasik 2013; Askell 2018). How is this situation best modelled, and what is the rational way of making decisions when in this predicament? Does cluelessness systematically favour some types of action over others?
On addressing cluelessness—for the most part I agree with the authors’ views, which includes the view that there needs to be further research in this area.
I do find it odd however that they attempt to counter the worry of ‘simple cluelessness’ but not that of ‘complex cluelessness’ i.e. to counter the possibility that there could be semi-foreseeable unintended consequences of longtermist interventions that make us ultimately uncertain on the sign of the expected-value assessment of these interventions. Maybe they see this as obviously not an issue...but I would have appreciated some thoughts on this.
I think complex cluelessness is essentially covered by the other subsections in the Cluelessness section. It’s an issue of assigning numbers arbitrarily to the point that what you should do depends on your arbitrary beliefs. I don’t think they succeed in addressing the issue, though, since they don’t sufficiently discuss and address ways each of their proposed interventions could backfire despite our best intentions (they do discuss some in section 4, though). The bar is pretty high to satisfy any “reasonable” person.
Thanks, I really haven’t given sufficient thought to the cluelessness section which seems the most novel and tricky. Fanaticism is probably just as important, if not more so, but is also easier to get one’s head around.
I agree with you in your other comment though that the following seems to imply that the authors are not “complexly clueless” about AI safety:
I mean I guess it is probably the case that if you’re saying it’s unreasonable for a probability function associated with very small positive expected value to be contained in your representor, you’ll also say a probability function associated with negative expected value also isn’t contained in it. This does seem to me to be a slightly extreme view.
Ya, maybe your representor should be a convex set, so that for any two functions in it, you can take any probabilistic mixture of them, and that would also be in your representor. This way, if you have one with expected value x and another with expected value y, you should have functions with each possible expected value between. So, if you have positive and negative EVs in your representor, you would also have 0 EV.
Do you mean negative EV is slightly extreme or ruling out negative EV is slightly extreme?
I think neglecting to look into and address ways something could be negative (e.g. a probability difference, EV) often leads us to unjustifiably assuming a positive lower bound, and I think this is an easy mistake to make or miss. Combining a positive lower bound with astronomical stakes would make the argument appear very compelling.
Yeah I meant ruling out negative EV in a representor may be slightly extreme, but I’m not really sure—I need to read more.
Thanks for this post Jack, I found it really useful as I haven’t got round yet to reading the updated paper. This break down in the cluelessness section was a new arrangement to me. Does anyone know if this break down has been used elsewhere? If not this seems like useful progress in better defining the cluelessness objections to longtermism.
Thanks Robert. I’ve never seen this breakdown of cluelessness and it could be a useful way for further research to define the issue.
The Global Priorities Institute raised the modelling of cluelessness in their research agenda and I’m looking forward to further work on this. If interested, see below for the two research questions related to cluelessness in the GPI research agenda. I have a feeling that there is still quite a bit of research that could be conducted in this area.
------------------
Forecasting the long-term effects of our actions often requires us to make difficult comparisons between complex and messy bodies of competing evidence, a situation Greaves (2016) calls “complex cluelessness”. We must also reckon with our own incomplete awareness, that is, the likelihood that the long-run future will be shaped by events we’ve never considered and perhaps can’t fully imagine. What is the appropriate response to this sort of epistemic situation? For instance, does rationality require us to adopt precise subjective probabilities concerning the very-long-run effects of our actions, imprecise probabilities (and if so, how imprecise?), or some other sort of doxastic state entirely?
Faced with the task of comparing actions in terms of expected value, it often seems that the agent is ‘clueless’: that is, that the available empirical and theoretical evidence simply supplies too thin a basis for guiding decisions in any principled way (Lenman 2000; Greaves 2016; Mogensen 2020) (INFORMAL: Tomasik 2013; Askell 2018). How is this situation best modelled, and what is the rational way of making decisions when in this predicament? Does cluelessness systematically favour some types of action over others?