I may not have a good understanding of this, but premise 1 is an EV claim, which rests on measuring outcomes, since perhaps EV calculations lead to better decisions.
Second, even if the A vs. B preference is justified, unawareness means there are countless other Xâs you havenât found yet.
If we challenge a premise I assume this must lead to the conclusion not obtaining. Your argument if successful here no longer forces the conclusion, it does not mean that the conclusion is false (a successful premise attack would make the conclusion false?).
Finally, it seems your claim also depends on a prediction model somehow breaking a messy, indeterminate universe (p3). Since this is impossible this is a heuristic. P1 references the idealized self; unawareness wins if we cannot make EV calculations /â know everything (relevant).
Thank you â I think these are exactly the right objections, and they help me clarify what I am claiming.
I think there are three separate issues here.
1. I am not challenging EV as a decision procedure
I agree that P1 is fundamentally an EV claim, and I am not arguing that EV calculations are useless. My objection is narrower.
P1 says that a justified preference for A over B requires an assessment that Aâs downstream consequences are better than Bâs. My counterexample concerns cases where A and B differ in what the agent takes to be relevant information in the first place.
Suppose I discover X, where X is a genuine feature of the world that my model previously omitted.
I now have two options:
A: incorporate X into the model;
B: knowingly continue using the model that excludes X.
My claim is not that A has higher expected value.
My claim is that the justification for A is not itself an EV comparison. A is a decision about the model through which subsequent EV comparisons are made.
Once X is incorporated, I can of course calculate expected values within the expanded model. But that does not tell me why I should have incorporated X in the first place.
That is the step I think P1 overlooks.
2. âThere are countless other Xâsâ is true â and I think it strengthens the point
Absolutely. There are presumably indefinitely many things that I do not know about.
But I donât think this creates a problem for the argument. It reveals the distinction I am trying to make.
I am not claiming that an agent can construct a complete model of reality. It cannot.
The relevant question is instead:
What should an agent do when it encounters information that it recognises as relevant but which its current model does not represent?
My answer is that the agent needs a structural norm of openness to correction.
That norm does not say âfind every X.â That would obviously be impossible.
It says: when relevant information enters the agentâs epistemic field, the agent should not systematically exclude it merely because incorporating it would disrupt the existing model.
This is important because P3 already grants that our models are incomplete. My argument is about what follows from that incompleteness for the maintenance of the model itself.
3. I think I may have confused âthe conclusion is falseâ with âthe conclusion does not followâ
Youâre right to flag this.
A successful attack on P1 does not establish that the conclusion is false. It establishes that the conclusion does not follow from the premises as stated.
So my claim should be:
If my counterexample succeeds, P1 is not universal, and therefore the argument is not deductively valid as stated.
The conclusion could nevertheless be true for other reasons.
Thatâs actually an important distinction, and I should have made it explicit.
4. I donât think my argument requires a prediction model to âbreakâ the universe
I agree with you that no prediction model can fully capture a messy, changing universe. But I think this is precisely why I am interested in the model-revision problem.
I am not assuming that the model can become complete.
Quite the opposite.
I am assuming:
the agent acts through a model;
the model is necessarily incomplete;
reality can therefore provide information that the model does not contain;
some of that information can be recognised as relevant;
the agent must decide what to do with that information.
At step 5, I think there is a choice that is prior to ordinary EV comparison:
Should X be incorporated into my model?
Only after answering that question can I ask:
Given my model, which action has the highest EV?
So I would distinguish:
model revision vs. action selection within a modelâ
P1 seems to describe the latter. My counterexample concerns the former.
5. This is where I think the âidealized selfâ becomes interesting
You say that P1 references the idealized self and that unawareness wins if we cannot make EV calculations or know everything relevant.
I think this may actually expose the deeper issue.
If the idealized agent is defined as an agent that already has the correct model of all relevant consequences, then of course P1 becomes very difficult to challenge. But then the model-revision problem has been assumed away.
The interesting question for a finite agent is precisely how it can become better informed.
And this is where my proposed inversion comes from.
We normally write:
ISâOUGHTâACTION.
But the IS available to an agent is itself model-dependent.
So there is a prior question:
What must I do to maintain a model capable of producing a reliable IS?â
The first ought is not âI ought to choose A rather than B.â
It is something like:
I ought to maintain the conditions under which my model remains responsive to relevant information.
That includes openness to correction, willingness to update, andâwhen information is distributed among agentsâmaintaining channels through which other agents can correct my model.
This is why I think the argument has implications beyond this particular question.
I am not trying to replace EV with some alternative decision rule.
I am suggesting that EV itself operates downstream of a prior epistemic/âontological layer: the conditions under which the model on which the EV calculation operates can remain a model of reality.
To this I would say adding more information to the model doesnât = better decision. P1 means an idealized agent would chose A over B given they know everything (e.g., full causal universe). Normatively we would choose this over flawed/âlimited decision making.
What makes a model âopen to correctionâ if all models are flawed?
I think a strong challenge to a critique would show that the conclusion is not forced based on a flaw with the premise as presented by DiGiovanni.
On 5) this seems pragmatic, not a critique to cluelessness.
Correct me if you think I am misled on any of these.
I think I need to correct you, because I believe you have misunderstood the role of the counterexample.
I am not claiming that adding more information to a model necessarily produces better decisions. All models are necessarily flawed. The important point is that they are not necessarily flawed in the same domains.
A model can therefore improve by interacting with other models and treating genuinely different perspectives as information about its own blind spots. That interaction is what I mean by learning.
And I think this is the step that is missing from P1.
P1 describes the choice between A and B given a model of the world. My counterexample introduces a prior class of decisions: decisions about how the model itself learns and changes.
That learning step does not straightforwardly have an EV.
Consider the classic example of the six blind men and the elephant. Each person encounters a different part of the elephant and consequently constructs a different model: one thinks it is a snake, another a wall, another a tree, and so on.
None of the individual models is simply âthe correct modelâ.
But the six agents can exchange information. They can recognise that their observations conflict. They can ask questions, compare perspectives, change their interpretations, and revise their models.
Through this interaction, the group can construct a representation that is closer to the elephant than any individual model.
Those decisions are not choices between A and B within a fixed model. They are decisions about how the models interact so that their blind spots can become visible to one another.
That is the step I am introducing into the premises.
And I think this is why I call the resulting principles ontological oughts. They are not rules about which outcome is better. They are rules concerning the conditions under which a finite model can remain aligned with a reality it can never completely represent.
For example, suppose I am modelling a situation and decide not to include the perspective of person Z.
That exclusion does not necessarily have an identifiable EV. I may not even be able to calculate what information I have excluded, because I have excluded it from the model through which I am doing the calculation.
But it can nevertheless matter structurally. My model may become increasingly misaligned with Zâs model, while Zâs model may simultaneously become increasingly misaligned with mine. We lose the possibility of correcting each otherâs blind spots.
The important point is therefore not that including Z necessarily produces better outcomes.
It is that excluding a potentially informative model removes a possible mechanism through which the limitations of my own model can be exposed.
This is where I think the issue goes deeper than the fact that the universe is messy or changing.
The fundamental problem is that a model is, by definition, not reality. It is a compression or representation of reality.
A model can test aspects of its own internal consistency. But it cannot, from within itself, establish that its own criteria for evaluating that consistency are sufficient to detect every way in which the model might be wrong.
In other words, the model has a blind spot concerning the adequacy of the mechanism by which it checks itself.
To resolve that blind spot, it requires another model.
But the other model has its own blind spots.
So we get:
M1ââM2â
where each model can provide information about what the other cannot see from within itself.
This makes the models epistemically interdependent.
And this is the sense in which I think the learning step is prior to the EV step.
The sequence is not merely:
MODELâA vs. BâACTION.
There is a prior sequence:
MODELâINTERACTIONâCORRECTIONâUPDATED MODELâA vs. B.
The principles governing that interaction are therefore not themselves simply another instance of the A-vs-B problem.
They determine whether the model can continue to learn from reality at all.
I am not claiming that collaboration guarantees truth, or that every perspective should be accepted, or that more information necessarily produces better decisions.
I am claiming something more limited:
A finite model cannot fully identify its own blind spots from within itself. Therefore, if it is to improve its correspondence with reality, it requires mechanisms through which information from outside the model can challenge and modify it.
And that, I think, is the counterexample to the universality of P1.
It introduces a class of actions that P1 does not describe: actions whose object is not choosing between outcomes within the model, but maintaining and improving the model through which outcomes can subsequently be evaluated.
I may not have a good understanding of this, but premise 1 is an EV claim, which rests on measuring outcomes, since perhaps EV calculations lead to better decisions.
Second, even if the A vs. B preference is justified, unawareness means there are countless other Xâs you havenât found yet.
If we challenge a premise I assume this must lead to the conclusion not obtaining. Your argument if successful here no longer forces the conclusion, it does not mean that the conclusion is false (a successful premise attack would make the conclusion false?).
Finally, it seems your claim also depends on a prediction model somehow breaking a messy, indeterminate universe (p3). Since this is impossible this is a heuristic. P1 references the idealized self; unawareness wins if we cannot make EV calculations /â know everything (relevant).
Would love clarification.
Thank you â I think these are exactly the right objections, and they help me clarify what I am claiming.
I think there are three separate issues here.
1. I am not challenging EV as a decision procedure
I agree that P1 is fundamentally an EV claim, and I am not arguing that EV calculations are useless. My objection is narrower.
P1 says that a justified preference for A over B requires an assessment that Aâs downstream consequences are better than Bâs. My counterexample concerns cases where A and B differ in what the agent takes to be relevant information in the first place.
Suppose I discover X, where X is a genuine feature of the world that my model previously omitted.
I now have two options:
A: incorporate X into the model;
B: knowingly continue using the model that excludes X.
My claim is not that A has higher expected value.
My claim is that the justification for A is not itself an EV comparison. A is a decision about the model through which subsequent EV comparisons are made.
Once X is incorporated, I can of course calculate expected values within the expanded model. But that does not tell me why I should have incorporated X in the first place.
That is the step I think P1 overlooks.
2. âThere are countless other Xâsâ is true â and I think it strengthens the point
Absolutely. There are presumably indefinitely many things that I do not know about.
But I donât think this creates a problem for the argument. It reveals the distinction I am trying to make.
I am not claiming that an agent can construct a complete model of reality. It cannot.
The relevant question is instead:
My answer is that the agent needs a structural norm of openness to correction.
That norm does not say âfind every X.â That would obviously be impossible.
It says: when relevant information enters the agentâs epistemic field, the agent should not systematically exclude it merely because incorporating it would disrupt the existing model.
This is important because P3 already grants that our models are incomplete. My argument is about what follows from that incompleteness for the maintenance of the model itself.
3. I think I may have confused âthe conclusion is falseâ with âthe conclusion does not followâ
Youâre right to flag this.
A successful attack on P1 does not establish that the conclusion is false. It establishes that the conclusion does not follow from the premises as stated.
So my claim should be:
The conclusion could nevertheless be true for other reasons.
Thatâs actually an important distinction, and I should have made it explicit.
4. I donât think my argument requires a prediction model to âbreakâ the universe
I agree with you that no prediction model can fully capture a messy, changing universe. But I think this is precisely why I am interested in the model-revision problem.
I am not assuming that the model can become complete.
Quite the opposite.
I am assuming:
the agent acts through a model;
the model is necessarily incomplete;
reality can therefore provide information that the model does not contain;
some of that information can be recognised as relevant;
the agent must decide what to do with that information.
At step 5, I think there is a choice that is prior to ordinary EV comparison:
Should X be incorporated into my model?
Only after answering that question can I ask:
Given my model, which action has the highest EV?
So I would distinguish:
model revision vs. action selection within a modelâ
P1 seems to describe the latter. My counterexample concerns the former.
5. This is where I think the âidealized selfâ becomes interesting
You say that P1 references the idealized self and that unawareness wins if we cannot make EV calculations or know everything relevant.
I think this may actually expose the deeper issue.
If the idealized agent is defined as an agent that already has the correct model of all relevant consequences, then of course P1 becomes very difficult to challenge. But then the model-revision problem has been assumed away.
The interesting question for a finite agent is precisely how it can become better informed.
And this is where my proposed inversion comes from.
We normally write:
ISâOUGHTâACTION.
But the IS available to an agent is itself model-dependent.
So there is a prior question:
What must I do to maintain a model capable of producing a reliable IS?â
That gives something like:
ONTOLOGICAL OUGHTâRELIABLE ISâPRACTICAL OUGHTâACTIONâ
The first ought is not âI ought to choose A rather than B.â
It is something like:
That includes openness to correction, willingness to update, andâwhen information is distributed among agentsâmaintaining channels through which other agents can correct my model.
This is why I think the argument has implications beyond this particular question.
I am not trying to replace EV with some alternative decision rule.
I am suggesting that EV itself operates downstream of a prior epistemic/âontological layer: the conditions under which the model on which the EV calculation operates can remain a model of reality.
And that is the part I am interested in.
To this I would say adding more information to the model doesnât = better decision. P1 means an idealized agent would chose A over B given they know everything (e.g., full causal universe). Normatively we would choose this over flawed/âlimited decision making.
What makes a model âopen to correctionâ if all models are flawed?
I think a strong challenge to a critique would show that the conclusion is not forced based on a flaw with the premise as presented by DiGiovanni.
On 5) this seems pragmatic, not a critique to cluelessness.
Correct me if you think I am misled on any of these.
I think I need to correct you, because I believe you have misunderstood the role of the counterexample.
I am not claiming that adding more information to a model necessarily produces better decisions. All models are necessarily flawed. The important point is that they are not necessarily flawed in the same domains.
A model can therefore improve by interacting with other models and treating genuinely different perspectives as information about its own blind spots. That interaction is what I mean by learning.
And I think this is the step that is missing from P1.
P1 describes the choice between A and B given a model of the world. My counterexample introduces a prior class of decisions: decisions about how the model itself learns and changes.
That learning step does not straightforwardly have an EV.
Consider the classic example of the six blind men and the elephant. Each person encounters a different part of the elephant and consequently constructs a different model: one thinks it is a snake, another a wall, another a tree, and so on.
None of the individual models is simply âthe correct modelâ.
But the six agents can exchange information. They can recognise that their observations conflict. They can ask questions, compare perspectives, change their interpretations, and revise their models.
Through this interaction, the group can construct a representation that is closer to the elephant than any individual model.
Those decisions are not choices between A and B within a fixed model. They are decisions about how the models interact so that their blind spots can become visible to one another.
That is the step I am introducing into the premises.
And I think this is why I call the resulting principles ontological oughts. They are not rules about which outcome is better. They are rules concerning the conditions under which a finite model can remain aligned with a reality it can never completely represent.
For example, suppose I am modelling a situation and decide not to include the perspective of person Z.
That exclusion does not necessarily have an identifiable EV. I may not even be able to calculate what information I have excluded, because I have excluded it from the model through which I am doing the calculation.
But it can nevertheless matter structurally. My model may become increasingly misaligned with Zâs model, while Zâs model may simultaneously become increasingly misaligned with mine. We lose the possibility of correcting each otherâs blind spots.
The important point is therefore not that including Z necessarily produces better outcomes.
It is that excluding a potentially informative model removes a possible mechanism through which the limitations of my own model can be exposed.
This is where I think the issue goes deeper than the fact that the universe is messy or changing.
The fundamental problem is that a model is, by definition, not reality. It is a compression or representation of reality.
A model can test aspects of its own internal consistency. But it cannot, from within itself, establish that its own criteria for evaluating that consistency are sufficient to detect every way in which the model might be wrong.
In other words, the model has a blind spot concerning the adequacy of the mechanism by which it checks itself.
To resolve that blind spot, it requires another model.
But the other model has its own blind spots.
So we get:
M1ââM2â
where each model can provide information about what the other cannot see from within itself.
This makes the models epistemically interdependent.
And this is the sense in which I think the learning step is prior to the EV step.
The sequence is not merely:
MODELâA vs. BâACTION.
There is a prior sequence:
MODELâINTERACTIONâCORRECTIONâUPDATED MODELâA vs. B.
The principles governing that interaction are therefore not themselves simply another instance of the A-vs-B problem.
They determine whether the model can continue to learn from reality at all.
I am not claiming that collaboration guarantees truth, or that every perspective should be accepted, or that more information necessarily produces better decisions.
I am claiming something more limited:
And that, I think, is the counterexample to the universality of P1.
It introduces a class of actions that P1 does not describe: actions whose object is not choosing between outcomes within the model, but maintaining and improving the model through which outcomes can subsequently be evaluated.
That is the distinction I was trying to make.