My original argument was aimed at P1. I argued that there is a class of decisions—decisions about how a model learns and changes—that cannot straightforwardly be reduced to comparisons of downstream consequences within that model.
The discussion has made me realise that this points to a more constructive response to DiGiovanni’s challenge.
Suppose the cluelessness argument is correct. What should an impartial altruist do?
I think the answer is: we should shift part of our attention from predicting the future to preserving our capacity to correct our predictions.
This is not a rejection of expected value. EV remains useful whenever our model is sufficiently informative to support a comparison. The problem is what to do when we cannot know whether it is.
A finite intelligence faces a peculiar problem:
It can discover errors in its model, but it cannot guarantee that its mechanisms for detecting errors will detect all of its errors.
This creates a distinction between two forms of rationality.
Predictive rationality: Given my model, which action has the best expected consequences?
Corrective rationality: Given that my model may be wrong, what keeps me capable of discovering and correcting that wrongness?
The second is not simply epistemic humility. It concerns the architecture of the decision-making process itself.
And this matters because all models are necessarily limited. The relevant question is therefore not whether we can construct a model that is complete, but whether we can maintain an intelligence that remains responsive to what its model leaves out.
Consider two strategies. One attempts to maximise the apparent value of actions under a particular model. The other preserves the conditions through which that model can be challenged: listening to independent perspectives, genuinely exchanging information, allowing disagreement to remain visible, acknowledging errors, and changing one’s representation when it no longer accommodates what one encounters.
The second strategy does not necessarily have higher expected value. That is precisely the point. If we cannot reliably compare long-run consequences, we need to consider properties of the intelligence doing the comparing.
This suggests a progression:
Prediction → model uncertainty → correction → meta-correction.
But correction introduces a further problem.
An intelligence can fail not only because its model is wrong, but because the way it responds to being wrong is itself inadequate. It may hear another perspective without allowing that perspective to affect its model. It may encounter disagreement without investigating it. It may acknowledge an error without examining what made the error possible. It may correct a conclusion while preserving the assumptions that generated the error.
So the problem is no longer simply:
“Is my model wrong?”
It becomes:
“Is the way I respond to being wrong itself capable of correction?”
This also gives a more precise answer to the question raised in the comments: what makes a model “open to correction” if all models are flawed?
Not the expectation that correction will make it correct.
Rather:
A model is corrigible when it preserves channels through which what it encounters can change how it represents reality.
And because those channels can themselves fail, we need to examine and change the way we listen, interpret, disagree, update, and correct.
This opens a new domain of rationality: the practice of meta-thinking. Under cluelessness, the impartial altruist should ask not only whether its model is wrong, but what it must do to remain capable of discovering that it is wrong: Whose perspective must it genuinely hear? What information must it allow to challenge its current representation? When confronted with disagreement, must it defend its model, or make the disagreement itself an object of inquiry? When an error is exposed, must it merely correct the conclusion, or examine why the error was possible in the first place? What must it be willing to change, and what must it keep open, for correction to remain possible?
These are not merely questions about acquiring better information about the future. They concern the present conditions under which an impartial altruist can justifiably move from an “is” to an “ought.” The paradox is that the actions required to maintain those conditions are themselves actions for which we need an “ought.”
The impartial altruist therefore has to reason reflexively:
To determine what it ought to do, it must also determine what it ought to preserve about the conditions that make determining what it ought to do possible.
This is what I mean by reflexive intelligence: intelligence begins to reason about the conditions of its own ability to reason.
And I think this gives us a constructive response to cluelessness that is neither a rejection of consequentialism nor an attempt to restore confidence in prediction.
Under genuine cluelessness, rationality does not end. It becomes reflexive.
My original argument was aimed at P1. I argued that there is a class of decisions—decisions about how a model learns and changes—that cannot straightforwardly be reduced to comparisons of downstream consequences within that model.
The discussion has made me realise that this points to a more constructive response to DiGiovanni’s challenge.
Suppose the cluelessness argument is correct. What should an impartial altruist do?
I think the answer is: we should shift part of our attention from predicting the future to preserving our capacity to correct our predictions.
This is not a rejection of expected value. EV remains useful whenever our model is sufficiently informative to support a comparison. The problem is what to do when we cannot know whether it is.
A finite intelligence faces a peculiar problem:
This creates a distinction between two forms of rationality.
Predictive rationality:
Given my model, which action has the best expected consequences?
Corrective rationality:
Given that my model may be wrong, what keeps me capable of discovering and correcting that wrongness?
The second is not simply epistemic humility. It concerns the architecture of the decision-making process itself.
And this matters because all models are necessarily limited. The relevant question is therefore not whether we can construct a model that is complete, but whether we can maintain an intelligence that remains responsive to what its model leaves out.
Consider two strategies. One attempts to maximise the apparent value of actions under a particular model. The other preserves the conditions through which that model can be challenged: listening to independent perspectives, genuinely exchanging information, allowing disagreement to remain visible, acknowledging errors, and changing one’s representation when it no longer accommodates what one encounters.
The second strategy does not necessarily have higher expected value. That is precisely the point. If we cannot reliably compare long-run consequences, we need to consider properties of the intelligence doing the comparing.
This suggests a progression:
But correction introduces a further problem.
An intelligence can fail not only because its model is wrong, but because the way it responds to being wrong is itself inadequate. It may hear another perspective without allowing that perspective to affect its model. It may encounter disagreement without investigating it. It may acknowledge an error without examining what made the error possible. It may correct a conclusion while preserving the assumptions that generated the error.
So the problem is no longer simply:
It becomes:
This also gives a more precise answer to the question raised in the comments: what makes a model “open to correction” if all models are flawed?
Not the expectation that correction will make it correct.
Rather:
And because those channels can themselves fail, we need to examine and change the way we listen, interpret, disagree, update, and correct.
This opens a new domain of rationality: the practice of meta-thinking. Under cluelessness, the impartial altruist should ask not only whether its model is wrong, but what it must do to remain capable of discovering that it is wrong: Whose perspective must it genuinely hear? What information must it allow to challenge its current representation? When confronted with disagreement, must it defend its model, or make the disagreement itself an object of inquiry? When an error is exposed, must it merely correct the conclusion, or examine why the error was possible in the first place? What must it be willing to change, and what must it keep open, for correction to remain possible?
These are not merely questions about acquiring better information about the future. They concern the present conditions under which an impartial altruist can justifiably move from an “is” to an “ought.” The paradox is that the actions required to maintain those conditions are themselves actions for which we need an “ought.”
The impartial altruist therefore has to reason reflexively:
This is what I mean by reflexive intelligence: intelligence begins to reason about the conditions of its own ability to reason.
And I think this gives us a constructive response to cluelessness that is neither a rejection of consequentialism nor an attempt to restore confidence in prediction.
Under genuine cluelessness, rationality does not end. It becomes reflexive.