I think I need to correct you, because I believe you have misunderstood the role of the counterexample.
I am not claiming that adding more information to a model necessarily produces better decisions. All models are necessarily flawed. The important point is that they are not necessarily flawed in the same domains.
A model can therefore improve by interacting with other models and treating genuinely different perspectives as information about its own blind spots. That interaction is what I mean by learning.
And I think this is the step that is missing from P1.
P1 describes the choice between A and B given a model of the world. My counterexample introduces a prior class of decisions: decisions about how the model itself learns and changes.
That learning step does not straightforwardly have an EV.
Consider the classic example of the six blind men and the elephant. Each person encounters a different part of the elephant and consequently constructs a different model: one thinks it is a snake, another a wall, another a tree, and so on.
None of the individual models is simply “the correct model”.
But the six agents can exchange information. They can recognise that their observations conflict. They can ask questions, compare perspectives, change their interpretations, and revise their models.
Through this interaction, the group can construct a representation that is closer to the elephant than any individual model.
Those decisions are not choices between A and B within a fixed model. They are decisions about how the models interact so that their blind spots can become visible to one another.
That is the step I am introducing into the premises.
And I think this is why I call the resulting principles ontological oughts. They are not rules about which outcome is better. They are rules concerning the conditions under which a finite model can remain aligned with a reality it can never completely represent.
For example, suppose I am modelling a situation and decide not to include the perspective of person Z.
That exclusion does not necessarily have an identifiable EV. I may not even be able to calculate what information I have excluded, because I have excluded it from the model through which I am doing the calculation.
But it can nevertheless matter structurally. My model may become increasingly misaligned with Z’s model, while Z’s model may simultaneously become increasingly misaligned with mine. We lose the possibility of correcting each other’s blind spots.
The important point is therefore not that including Z necessarily produces better outcomes.
It is that excluding a potentially informative model removes a possible mechanism through which the limitations of my own model can be exposed.
This is where I think the issue goes deeper than the fact that the universe is messy or changing.
The fundamental problem is that a model is, by definition, not reality. It is a compression or representation of reality.
A model can test aspects of its own internal consistency. But it cannot, from within itself, establish that its own criteria for evaluating that consistency are sufficient to detect every way in which the model might be wrong.
In other words, the model has a blind spot concerning the adequacy of the mechanism by which it checks itself.
To resolve that blind spot, it requires another model.
But the other model has its own blind spots.
So we get:
M1↔M2
where each model can provide information about what the other cannot see from within itself.
This makes the models epistemically interdependent.
And this is the sense in which I think the learning step is prior to the EV step.
The sequence is not merely:
MODEL→A vs. B→ACTION.
There is a prior sequence:
MODEL→INTERACTION→CORRECTION→UPDATED MODEL→A vs. B.
The principles governing that interaction are therefore not themselves simply another instance of the A-vs-B problem.
They determine whether the model can continue to learn from reality at all.
I am not claiming that collaboration guarantees truth, or that every perspective should be accepted, or that more information necessarily produces better decisions.
I am claiming something more limited:
A finite model cannot fully identify its own blind spots from within itself. Therefore, if it is to improve its correspondence with reality, it requires mechanisms through which information from outside the model can challenge and modify it.
And that, I think, is the counterexample to the universality of P1.
It introduces a class of actions that P1 does not describe: actions whose object is not choosing between outcomes within the model, but maintaining and improving the model through which outcomes can subsequently be evaluated.
I think I need to correct you, because I believe you have misunderstood the role of the counterexample.
I am not claiming that adding more information to a model necessarily produces better decisions. All models are necessarily flawed. The important point is that they are not necessarily flawed in the same domains.
A model can therefore improve by interacting with other models and treating genuinely different perspectives as information about its own blind spots. That interaction is what I mean by learning.
And I think this is the step that is missing from P1.
P1 describes the choice between A and B given a model of the world. My counterexample introduces a prior class of decisions: decisions about how the model itself learns and changes.
That learning step does not straightforwardly have an EV.
Consider the classic example of the six blind men and the elephant. Each person encounters a different part of the elephant and consequently constructs a different model: one thinks it is a snake, another a wall, another a tree, and so on.
None of the individual models is simply “the correct model”.
But the six agents can exchange information. They can recognise that their observations conflict. They can ask questions, compare perspectives, change their interpretations, and revise their models.
Through this interaction, the group can construct a representation that is closer to the elephant than any individual model.
Those decisions are not choices between A and B within a fixed model. They are decisions about how the models interact so that their blind spots can become visible to one another.
That is the step I am introducing into the premises.
And I think this is why I call the resulting principles ontological oughts. They are not rules about which outcome is better. They are rules concerning the conditions under which a finite model can remain aligned with a reality it can never completely represent.
For example, suppose I am modelling a situation and decide not to include the perspective of person Z.
That exclusion does not necessarily have an identifiable EV. I may not even be able to calculate what information I have excluded, because I have excluded it from the model through which I am doing the calculation.
But it can nevertheless matter structurally. My model may become increasingly misaligned with Z’s model, while Z’s model may simultaneously become increasingly misaligned with mine. We lose the possibility of correcting each other’s blind spots.
The important point is therefore not that including Z necessarily produces better outcomes.
It is that excluding a potentially informative model removes a possible mechanism through which the limitations of my own model can be exposed.
This is where I think the issue goes deeper than the fact that the universe is messy or changing.
The fundamental problem is that a model is, by definition, not reality. It is a compression or representation of reality.
A model can test aspects of its own internal consistency. But it cannot, from within itself, establish that its own criteria for evaluating that consistency are sufficient to detect every way in which the model might be wrong.
In other words, the model has a blind spot concerning the adequacy of the mechanism by which it checks itself.
To resolve that blind spot, it requires another model.
But the other model has its own blind spots.
So we get:
M1↔M2
where each model can provide information about what the other cannot see from within itself.
This makes the models epistemically interdependent.
And this is the sense in which I think the learning step is prior to the EV step.
The sequence is not merely:
MODEL→A vs. B→ACTION.
There is a prior sequence:
MODEL→INTERACTION→CORRECTION→UPDATED MODEL→A vs. B.
The principles governing that interaction are therefore not themselves simply another instance of the A-vs-B problem.
They determine whether the model can continue to learn from reality at all.
I am not claiming that collaboration guarantees truth, or that every perspective should be accepted, or that more information necessarily produces better decisions.
I am claiming something more limited:
And that, I think, is the counterexample to the universality of P1.
It introduces a class of actions that P1 does not describe: actions whose object is not choosing between outcomes within the model, but maintaining and improving the model through which outcomes can subsequently be evaluated.
That is the distinction I was trying to make.