I think the idea is intriguing, and I agree that this is possible in principle, but I’m not convinced of your take on its practical implications. Apart from heuristic reasons to be sceptical of a new idea on this level of abstractness and speculativeness, my main objection is that a high degree of similarity with respect to reasoning (which is required for the decisions to be entangled) probably goes along with at least some degree of similarity with respect to values. (And if the values of the agents that correlate with me are similar to mine, then the result of taking them into account is also closer to my own values than the compromise value system of all agents.)
You write:
Superrationality only motivates cooperation if one has good reason to believe that another party’s decision algorithm is indeed extremely similar to one’s own. Human reasoning processes differ in many ways, and sympathy towards superrationality represents only one small dimension of one’s reasoning process. It may very well be extremely rare that two people’s reasoning is sufficiently similar that, having common knowledge of this similarity, they should rationally cooperate in a prisoner’s dilemma.
Conditional on this extremely high degree of similarity to me, isn’t it also more likely that their values are also similar to mine? For instance, if my reasoning is shaped by the experiences I’ve made, my genetic makeup, or the set of all ideas I’ve read about over the course of my life, then an agent with identical or highly similar reasoning would also share a lot of these characteristics. But of course, my experiences, genes, etc. also determine my values, so similarity with respect to these factors implies similarity with respect to values.
This is not the same as claiming that a given characteristic X that’s relevant to decision-making is generally linked to values, in the sense that people with X have systematically different values. It’s a subtle difference: I’m not saying that certain aspects of reasoning generally go along with certain values across the entire population; I’m saying that a high degree of similarity regarding reasoning goes along with similarity regarding values.
Thanks for writing this up!
I think the idea is intriguing, and I agree that this is possible in principle, but I’m not convinced of your take on its practical implications. Apart from heuristic reasons to be sceptical of a new idea on this level of abstractness and speculativeness, my main objection is that a high degree of similarity with respect to reasoning (which is required for the decisions to be entangled) probably goes along with at least some degree of similarity with respect to values. (And if the values of the agents that correlate with me are similar to mine, then the result of taking them into account is also closer to my own values than the compromise value system of all agents.)
You write:
Conditional on this extremely high degree of similarity to me, isn’t it also more likely that their values are also similar to mine? For instance, if my reasoning is shaped by the experiences I’ve made, my genetic makeup, or the set of all ideas I’ve read about over the course of my life, then an agent with identical or highly similar reasoning would also share a lot of these characteristics. But of course, my experiences, genes, etc. also determine my values, so similarity with respect to these factors implies similarity with respect to values.
This is not the same as claiming that a given characteristic X that’s relevant to decision-making is generally linked to values, in the sense that people with X have systematically different values. It’s a subtle difference: I’m not saying that certain aspects of reasoning generally go along with certain values across the entire population; I’m saying that a high degree of similarity regarding reasoning goes along with similarity regarding values.