Hi Dan/Fable, thanks for the critique! The three most important problems I see:
1. Claiming that the unawareness argument only has practical implications if there’s a privileged “default”
The sequence argues carefully for incomparability. It cannot argue that incomparability favors inaction, because if A and the status quo are incomparable, the status quo is not better. Yet the practical gloss everyone puts on the conclusion resolves every incomparability toward the default
As discussed here and in the introduction of the sequence, my claim was never “we should default to inaction”. It’s that we have no impartial altruistic reason to favor any intervention over any other option, including “inaction”.
Your response to this is that if we don’t favor a default under incomparability, “the argument has almost no practical bite”. But we can have reasons for choices other than impartial altruistic ones — see here.
(This point is upstream of one of your replies to objection #2: “so everything becomes incomparable with everything, the argument again supplies no reason to resolve toward a default”.)
2. Conflating two kinds of awareness growth
In your response to the objection “The considerations that matter most may be ones no engagement reveals”, you say that the evidence for P3 is “We’ve become aware of new considerations through active engagement.” But “becoming aware of new considerations” is a much lower bar than “becoming aware of all the considerations that the sign of an action’s ‘EV’ is sensitive to”. You need the latter for your critique via Model 3 to work. This is important for the following:
3. Neglecting unawareness about/imprecision in the time horizon
Your argument in Model 3 seems to be: Suppose you have T time steps to (1) actively grow your awareness by repeated exploration and then (2) exploit the strategy that does best w.r.t. your credences over this fleshed-out awareness set — before you die. Then (1)+(2) would beat “stick to the familiar domain”. I’m happy to grant here[1] that this is true for some T.
But we don’t know T.[2] If our beliefs about T are imprecise enough, we can’t say whether (a) the benefits of eventually cashing in on our grown awareness outweigh (b) the potential backfire effects of actions we take to grow our awareness. To meet the bar of awareness growth noted in (2) above, T might need to be very large indeed.
Suppose we instead say “Conditional on living forever, the upsides are unbounded, so the utility from this case swamps all the finite-T cases.” One problem is that if we’re going to allow for unbounded upsides from an infinite T, we should also consider unbounded downsides. (You acknowledge this possibility in objection #2, but what matters is whether your critique via Model 3 actually works, not what the sequence as written says.)
Fair point that the sequences do not strongly default towards inaction. I think the evidence gains from action mean that it ‘is’ favored, but this is an empirical prediction about the world, which I think you disagree with.
If increased information reduces our unawareness, that decreases the relevance of cluelessness arguments, no? In particular, the set of actions with overlapping credal sets would be reduced, and so there would be more dominated actions we could rule out for consideration. Unless you’re making an argument that the information learned is too small to meaningfully constrain our predictions in expectation. In which case sure, that’s an empirical claim I disagree with, but it is logically consistent.
I think most importantly we have an empirical disagreement about both how clueless we are of the longterm impacts of our actions, and how difficult it is to reduce that cluelessness. I’m trying to think about cruxes that might resolve that.
I think you disagree that increased predictive capability in shorter time horizons generalizes to long-time horizons [1]. For me, seeing bad short & medium term predictions ‘would’ cause me to give more credence to unawareness arguments. But you seem to have prevented the opposite evidence from swaying you in the other direction (unless I’m misunderstanding).
Hi Dan/Fable, thanks for the critique! The three most important problems I see:
1. Claiming that the unawareness argument only has practical implications if there’s a privileged “default”
As discussed here and in the introduction of the sequence, my claim was never “we should default to inaction”. It’s that we have no impartial altruistic reason to favor any intervention over any other option, including “inaction”.
Your response to this is that if we don’t favor a default under incomparability, “the argument has almost no practical bite”. But we can have reasons for choices other than impartial altruistic ones — see here.
(This point is upstream of one of your replies to objection #2: “so everything becomes incomparable with everything, the argument again supplies no reason to resolve toward a default”.)
2. Conflating two kinds of awareness growth
In your response to the objection “The considerations that matter most may be ones no engagement reveals”, you say that the evidence for P3 is “We’ve become aware of new considerations through active engagement.” But “becoming aware of new considerations” is a much lower bar than “becoming aware of all the considerations that the sign of an action’s ‘EV’ is sensitive to”. You need the latter for your critique via Model 3 to work. This is important for the following:
3. Neglecting unawareness about/imprecision in the time horizon
Your argument in Model 3 seems to be: Suppose you have T time steps to (1) actively grow your awareness by repeated exploration and then (2) exploit the strategy that does best w.r.t. your credences over this fleshed-out awareness set — before you die. Then (1)+(2) would beat “stick to the familiar domain”. I’m happy to grant here[1] that this is true for some T.
But we don’t know T.[2] If our beliefs about T are imprecise enough, we can’t say whether (a) the benefits of eventually cashing in on our grown awareness outweigh (b) the potential backfire effects of actions we take to grow our awareness. To meet the bar of awareness growth noted in (2) above, T might need to be very large indeed.
This is just for the sake of argument. I think the model is importantly unrealistic in some ways I don’t cover here for lack of time.
Suppose we instead say “Conditional on living forever, the upsides are unbounded, so the utility from this case swamps all the finite-T cases.” One problem is that if we’re going to allow for unbounded upsides from an infinite T, we should also consider unbounded downsides. (You acknowledge this possibility in objection #2, but what matters is whether your critique via Model 3 actually works, not what the sequence as written says.)
Thanks for the thoughtful reply Anthony.
Fair point that the sequences do not strongly default towards inaction. I think the evidence gains from action mean that it ‘is’ favored, but this is an empirical prediction about the world, which I think you disagree with.
If increased information reduces our unawareness, that decreases the relevance of cluelessness arguments, no? In particular, the set of actions with overlapping credal sets would be reduced, and so there would be more dominated actions we could rule out for consideration. Unless you’re making an argument that the information learned is too small to meaningfully constrain our predictions in expectation. In which case sure, that’s an empirical claim I disagree with, but it is logically consistent.
I think most importantly we have an empirical disagreement about both how clueless we are of the longterm impacts of our actions, and how difficult it is to reduce that cluelessness. I’m trying to think about cruxes that might resolve that.
I think you disagree that increased predictive capability in shorter time horizons generalizes to long-time horizons [1]. For me, seeing bad short & medium term predictions ‘would’ cause me to give more credence to unawareness arguments. But you seem to have prevented the opposite evidence from swaying you in the other direction (unless I’m misunderstanding).
Like, what is something you could see happen tomorrow that would make you say ‘whoa, we actually can know the long-term consequences of our actions’? Would it take something as extreme as https://www.lesswrong.com/posts/6FmqiAgS8h4EJm86s/how-to-convince-me-that-2-2-3?
[1] https://forum.effectivealtruism.org/posts/ZTz9BKxCys5DgaHQJ/ama-anthony-digiovanni-author-of-the-challenge-of?commentId=bcaWd275R3PXgivma)