I think cluelessness-ish worries. From the perspective of longtermism, for any particular action, there are thousands of considerations/ scenarios that point in the direction of the action being good, and thousands of considerations/ scenarios that point in the direction of the action being bad. The standard response to that is that you should weigh all these and do what is in expectation best, according to your best-guess credences. But maybe we just don’t have sufficiently fine-grained credences for this to work, and there’s some principled grounds for saying “I’m confident that this short-run good thing I do is good, and (given my not-completely-precise credences) I shouldn’t think that the expected value of the more speculative stuff is either positive or negative.”
> From the perspective of longtermism, for any particular action, there are thousands of considerations/ scenarios that point in the direction of the action being good, and thousands of considerations/ scenarios that point in the direction of the action being bad.
I worry that this type of problem is often exaggerated, e.g. with the suggestion that ‘proposed x-risk A has some arguments going for it, but one could make arguments for thousands of other things’ when the thousands of other candidates are never produced and could not be produced and appear to be in the same ballpark. When one makes a serious effort to catalog serious candidates at reasonable granularity the scope of considerations is vastly more manageable than initially suggested, but cluelessness is invoked in lieu of actually doing the search, or a representative subset of the search.
I think you might be misunderstanding what I was referring to. An example of what I mean: Suppose Jane is deciding whether to work for Deepmind on the AI safety team. She’s unsure whether this speeds up or slows down AI development; her credence is imprecise, represented by the interval [0.4, 0.6]. She’s confident, let’s say, that speeding up AI development is bad. Because there’s some precisification of her credences on which taking the job is good, and some on which taking the job is bad, then if she uses a Liberal decision rule (= it is permissible for you to perform any action that is permissible according to at least one of the credence functions in your set), it’s permissible for her to take the job or not take the job.
The issue is that, if you have imprecise credences and a Liberal decision rule, and are a longtermist, then almost all serious contenders for actions are permissible.
So the neartermist would need to have some way of saying (i) we can carve out the definitely-good part of the action, which is better than not-doing the action on all precisifications of the credence; (ii) we can ignore the other parts of the action (e.g. the flow-through effects) that are good on some precisifications and bad on some precisifications. It seems hard to make that theoretically justified, but I think it matches how people actually think, so at least has some common-sense motivation.
But you could do it if you could argue for a pseudodominance principle that says: “If there’s some interval of time t_i over which action x does more expected good than action y on all precisifications of one’s credence function, and there’s no interval of time t_j at which action y does more expected good than action x on all precisifications of one’s credence function, then you should choose x over y”.
(In contrast, it seems you thought I was referring to AI vs some other putative great longtermist intervention. I agree that plausible longtermist rivals to AI and bio are thin on the ground.)
She’s unsure whether this speeds up or slows down AI development; her credence is imprecise, represented by the interval [0.4, 0.6]. She’s confident, let’s say, that speeding up AI development is bad.
That’s an awfully (in)convenient interval to have! That is the unique position for an interval of that length with no distinguishing views about any parts of the interval, such that integrating over it gives you a probability of 0.5 and expected impact of 0.
The standard response to that is that you should weigh all these and do what is in expectation best, according to your best-guess credences. But maybe we just don’t have sufficiently fine-grained credences for this to work,
If the argument from cluelessness depends on giving that kind of special status to imprecise credences, then I just reject them for the general reason that coarsening credences leads to worse decisions and predictions (particularly if one has done basic calibration training and has some numeracy and skill at prediction). There is signal to be lost in coarsening on individual questions. And for compound questions with various premises or contributing factors making use of the signal on each of those means your views will be moved by signal.
Chapter 3 of Jeffrey Friedman’s book War and Chance: Assessing Uncertainty in International Politics presents data and arguments showing large losses from coarsening credences instead of just giving a number between 0 and 1. I largely share his negative sentiments about imprecise credences, especially.
[VOI considerations around less investigated credences that are more likely to be moved by investigation are fruitful grounds to delay action to acquire or await information that one expects may be actually attained, but are not the same thing as imprecise credences.]
(In contrast, it seems you thought I was referring to AI vs some other putative great longtermist intervention. I agree that plausible longtermist rivals to AI and bio are thin on the ground.)
That was an example of the phenomenon of not searching a supposedly vast space and finding that in fact the # of top-level considerations are manageable (at least compared to thousands), based off experience with other people saying that there must be thousands of similarly plausible risks. I would likewise say that the DeepMind employee in your example doesn’t face thousands upon thousands of ballpark-similar distinct considerations to assess.
I give some examples here; the “stratospheric aerosol injection to blunt impacts of climate change” example is an x-risk reduction one.
It’s pretty straightforward to tell a story about how any well-intentioned action could have unintended, negative consequences in the long run. Lots of sci-fi uses this premise.
This doesn’t mean the stories are always plausible (though note that “plausibility” here is usually assessed by intuition), and it’s not the same as generating a comprehensive catalog of stories about how an action could go (the state space here is too large to generate such a catalog).
What do you think the best argument is against strong longtermism?
I think cluelessness-ish worries. From the perspective of longtermism, for any particular action, there are thousands of considerations/ scenarios that point in the direction of the action being good, and thousands of considerations/ scenarios that point in the direction of the action being bad. The standard response to that is that you should weigh all these and do what is in expectation best, according to your best-guess credences. But maybe we just don’t have sufficiently fine-grained credences for this to work, and there’s some principled grounds for saying “I’m confident that this short-run good thing I do is good, and (given my not-completely-precise credences) I shouldn’t think that the expected value of the more speculative stuff is either positive or negative.”
> From the perspective of longtermism, for any particular action, there are thousands of considerations/ scenarios that point in the direction of the action being good, and thousands of considerations/ scenarios that point in the direction of the action being bad.
I worry that this type of problem is often exaggerated, e.g. with the suggestion that ‘proposed x-risk A has some arguments going for it, but one could make arguments for thousands of other things’ when the thousands of other candidates are never produced and could not be produced and appear to be in the same ballpark. When one makes a serious effort to catalog serious candidates at reasonable granularity the scope of considerations is vastly more manageable than initially suggested, but cluelessness is invoked in lieu of actually doing the search, or a representative subset of the search.
I think you might be misunderstanding what I was referring to. An example of what I mean: Suppose Jane is deciding whether to work for Deepmind on the AI safety team. She’s unsure whether this speeds up or slows down AI development; her credence is imprecise, represented by the interval [0.4, 0.6]. She’s confident, let’s say, that speeding up AI development is bad. Because there’s some precisification of her credences on which taking the job is good, and some on which taking the job is bad, then if she uses a Liberal decision rule (= it is permissible for you to perform any action that is permissible according to at least one of the credence functions in your set), it’s permissible for her to take the job or not take the job.
The issue is that, if you have imprecise credences and a Liberal decision rule, and are a longtermist, then almost all serious contenders for actions are permissible.
So the neartermist would need to have some way of saying (i) we can carve out the definitely-good part of the action, which is better than not-doing the action on all precisifications of the credence; (ii) we can ignore the other parts of the action (e.g. the flow-through effects) that are good on some precisifications and bad on some precisifications. It seems hard to make that theoretically justified, but I think it matches how people actually think, so at least has some common-sense motivation.
But you could do it if you could argue for a pseudodominance principle that says: “If there’s some interval of time t_i over which action x does more expected good than action y on all precisifications of one’s credence function, and there’s no interval of time t_j at which action y does more expected good than action x on all precisifications of one’s credence function, then you should choose x over y”.
(In contrast, it seems you thought I was referring to AI vs some other putative great longtermist intervention. I agree that plausible longtermist rivals to AI and bio are thin on the ground.)
That’s an awfully (in)convenient interval to have! That is the unique position for an interval of that length with no distinguishing views about any parts of the interval, such that integrating over it gives you a probability of 0.5 and expected impact of 0.
If the argument from cluelessness depends on giving that kind of special status to imprecise credences, then I just reject them for the general reason that coarsening credences leads to worse decisions and predictions (particularly if one has done basic calibration training and has some numeracy and skill at prediction). There is signal to be lost in coarsening on individual questions. And for compound questions with various premises or contributing factors making use of the signal on each of those means your views will be moved by signal.
Chapter 3 of Jeffrey Friedman’s book War and Chance: Assessing Uncertainty in International Politics presents data and arguments showing large losses from coarsening credences instead of just giving a number between 0 and 1. I largely share his negative sentiments about imprecise credences, especially.
[VOI considerations around less investigated credences that are more likely to be moved by investigation are fruitful grounds to delay action to acquire or await information that one expects may be actually attained, but are not the same thing as imprecise credences.]
That was an example of the phenomenon of not searching a supposedly vast space and finding that in fact the # of top-level considerations are manageable (at least compared to thousands), based off experience with other people saying that there must be thousands of similarly plausible risks. I would likewise say that the DeepMind employee in your example doesn’t face thousands upon thousands of ballpark-similar distinct considerations to assess.
I give some examples here; the “stratospheric aerosol injection to blunt impacts of climate change” example is an x-risk reduction one.
It’s pretty straightforward to tell a story about how any well-intentioned action could have unintended, negative consequences in the long run. Lots of sci-fi uses this premise.
This doesn’t mean the stories are always plausible (though note that “plausibility” here is usually assessed by intuition), and it’s not the same as generating a comprehensive catalog of stories about how an action could go (the state space here is too large to generate such a catalog).
Shameless plug for my essay on cluelessness: 1, 2, 3, 4