The orthogonality thesis is trivially a motte and bailey—you’re using it as one right here! The original claim by Bostrom was a statement against logical necessity: ‘an artificial mind need not care intrinsically about any of those things’ (emphasis mine); yet in your comment you’re equivocating with a statement that’s effectively about probability: ‘sampling from an extremely tiny section of an enormously broad space’.
You might be right in your claim, but your claim is not what the arguments in the orthogonality thesis papers purport to show.
I would also like to make a stronger counterclaim: I think a priori arguments about ‘probability space’ (dis)prove way too much. If you disregard empirical data, you can use them to disprove anything, like ‘the height of Earth fauna is contingent on specific details of various evolutionary pressures and environmental circumstances, and is sampled from a tiny section on the number line, so we should expect that alien fauna we encounter will be arbitrarily tall (or perhaps have negative height)’. If Earth-evolved intelligence tends even weakly to have e.g. sympathy towards non-kin, that is evidence that Earth-evolved intelligence is a biased sample, but also evidence that there exists some pull towards non-kin-sympathy in intelligence space.
My sense is that (as your footnote hints at), the more intelligent animals are, the more examples we seem to see of individual non-reciprocal altruism to non-kin (there are many clear examples of non-reciprocal altruism across species in cetaceans for e.g., and less numerous but still convincing examples of it in corvids).
The orthogonality thesis is trivially a motte and bailey—you’re using it as one right here! The original claim by Bostrom was a statement against logical necessity: ‘an artificial mind need not care intrinsically about any of those things’ (emphasis mine); yet in your comment you’re equivocating with a statement that’s effectively about probability: ‘sampling from an extremely tiny section of an enormously broad space’.
You might be right in your claim, but your claim is not what the arguments in the orthogonality thesis papers purport to show.
I would also like to make a stronger counterclaim: I think a priori arguments about ‘probability space’ (dis)prove way too much. If you disregard empirical data, you can use them to disprove anything, like ‘the height of Earth fauna is contingent on specific details of various evolutionary pressures and environmental circumstances, and is sampled from a tiny section on the number line, so we should expect that alien fauna we encounter will be arbitrarily tall (or perhaps have negative height)’. If Earth-evolved intelligence tends even weakly to have e.g. sympathy towards non-kin, that is evidence that Earth-evolved intelligence is a biased sample, but also evidence that there exists some pull towards non-kin-sympathy in intelligence space.
My sense is that (as your footnote hints at), the more intelligent animals are, the more examples we seem to see of individual non-reciprocal altruism to non-kin (there are many clear examples of non-reciprocal altruism across species in cetaceans for e.g., and less numerous but still convincing examples of it in corvids).