I’ve mostly arrived at similar conclusions through Evidential Cooperation in Large Worlds (ECL; see, e.g., Lukas’s commentary). ECL adds a few extra steps that make the implications a lot clearer to me (though not to the extent of them actually being clear). I’d be curious how they transfer to acausal normalcy!
Set of cooperators. When it comes acausal normalcy, what is the set of all cooperation partners? I imagine it’s everyone who honestly tries to figure out acausal normalcy and tries to adhere to it, so that people who try to exploit it are automatically expelled from the set of cooperation partners? That’s how it works with ECL. There could be correlations between beings who are wont to cooperate vs. defect (be it out of ignorance), so this could rule out some moral norms.
Circular convincingness. In the ECL paper, Caspar cites some more and less formal studies of the efficacy of acausal cooperation on earth. They paint a bit of a mixed picture. The upshot was something like that acausal cooperation can’t be trusted with few participants. There were some arguments that it works in elections (a few million participants), but I don’t remember the details. So the step that the universe is big (possibly infinite) was critical to convince every actor that acausal cooperation is worth it and thus convince them that others out there will also consider it worth it.
Is this covered by Payor’s Lemma? I’m still trying to wrap my head around it, especially which of the assumptions are ones it proves and which are assumptions I need to make to apply it… It also looks a bit funky to me. Is the character missing from the typeface or is there meant to be a box there?
Power. Another problem that I feel quite uncertain about is power. In the extremes, someone powerless will be happy to acausally cooperate rather than be crushed, and someone all-powerful who knows it (i.e. there’s no one else equally or more powerful) has probably no interest in acausal cooperation. The second one of these beings is probably impossible, but there seems to be a graduation where the more powerful a being is, the less interested it is in cooperation. This is alleviated by the uncertainty even the powerful beings will have about their relative rank in the power hierarchy, and norms around pride and honor and such where beings may punish others for defections even at some cost to themselves. I don’t see why these alleviating factors should precisely cancel out the greater power. I would rather expect there to be complex tradeoffs.
That all makes me inclined to think that the moral norms of more powerful beings should be weighed more (a bit less than proportionally more) than the moral norms of less powerful beings. These powerful beings are of course superintelligences and grabby aliens but also our ancestors. (Are there already writings on this? I’m only aware of a fairly short Brian Tomasik essay that came to a slightly different conclusion.)
Convergent drives. Self-preservation, value preservation, and resource acquisition may be convergent drives of powerful beings because they would otherwise perhaps not be powerful. Self-preservation implies existence in the first place, so that it may be optimal to help these powerful beings to come into existence (but which ones?). Value preservation is a difficult one since whatever powerful being ends up existing might’ve gotten there only because its previous versions still value-drifted. Resource acquisition may push toward instilling some indexical uncertainty in AIs so that they can figure out whether we’re in a simulation, and whether we can control or trade with higher levels or other branches. I feel highly unsure about these implications, but I think it would be important to get them right.
Idealization. But the consideration of our ancestors brings me to my next point of confusion, which is how to idealize norms. You mention the the distinction between instrumental and terminal goals. I think it’s an important one. E.g., maybe monogamy limited the spread of STDs among other benefits; some of these are less important now with tests, vaccinations, treatments, condoms, etc. So if our ancestors valued monogamy instrumentally, we don’t need to continue upholding it to cooperate with them even though they have a lot of power over our very existence. But if they valued it terminally, we might have to! Perhaps I really have to bite bullet on this one (if I understand you correctly) and turn it into a tradeoff between the interests of powerful people in the past with weird norms and the interests of less powerful people today…
Positive and negative reinforement. When it comes to the particular values, especially reinforcement learning stands out to me. When a being is much smaller than its part of the universe, it’ll probably need to learn, so my guess is that reinforcement learning is very common among all sorts of beings. Buck once argued that any kind of stimulation, positive or negative, expends energy, so that we’re calibrated such that most beings who survive and reproduce will spend most the time close the energy-conserving neutral state. That sounds pretty universal (though perhaps there are civilizations where only resources other than energy are scarce), so that the typical utilitarian values of increasing happiness and reducing suffering may be very common too.
(When I was new to EA, I read a lot of Wikipedia and SEP articles on various moral philosophies. My impression was that reducing suffering is “not dispreferred” by all of them (or to the extent that some of them might’ve been explicitly pro suffering, it was in specific, exceptional instances such as retribution) whereas all other norms were either dispreferred by some moral philosophies or really specific. My tentative guess at the time was that the maximally morally cooperative actions are those that reduced suffering while not running into any complicated tradeoffs against other values.)
Unique optimal ethics. An interesting observation is that ECL implies that there is one optimal compromise morality – that anyone who deviates from it leaves gains from moral trade on the table. So my own values should inform my actions only to a very minor degree. (I’m one sample among countless samples when it comes to inferring the distribution of values across the universe.) Instead my actions should be informed by the distribution of all values around the universe to the extent that I can infer it; by the distribution of relevant resources around the universe; and by the resources I have causally at my disposal, which could put me in a comparatively advantageous position to benefit some set of values. I wonder, is there one optimal acausal normalcy too?
I’ve mostly arrived at similar conclusions through Evidential Cooperation in Large Worlds (ECL; see, e.g., Lukas’s commentary). ECL adds a few extra steps that make the implications a lot clearer to me (though not to the extent of them actually being clear). I’d be curious how they transfer to acausal normalcy!
Set of cooperators. When it comes acausal normalcy, what is the set of all cooperation partners? I imagine it’s everyone who honestly tries to figure out acausal normalcy and tries to adhere to it, so that people who try to exploit it are automatically expelled from the set of cooperation partners? That’s how it works with ECL. There could be correlations between beings who are wont to cooperate vs. defect (be it out of ignorance), so this could rule out some moral norms.
Circular convincingness. In the ECL paper, Caspar cites some more and less formal studies of the efficacy of acausal cooperation on earth. They paint a bit of a mixed picture. The upshot was something like that acausal cooperation can’t be trusted with few participants. There were some arguments that it works in elections (a few million participants), but I don’t remember the details. So the step that the universe is big (possibly infinite) was critical to convince every actor that acausal cooperation is worth it and thus convince them that others out there will also consider it worth it.
Is this covered by Payor’s Lemma? I’m still trying to wrap my head around it, especially which of the assumptions are ones it proves and which are assumptions I need to make to apply it… It also looks a bit funky to me. Is the character missing from the typeface or is there meant to be a box there?
Power. Another problem that I feel quite uncertain about is power. In the extremes, someone powerless will be happy to acausally cooperate rather than be crushed, and someone all-powerful who knows it (i.e. there’s no one else equally or more powerful) has probably no interest in acausal cooperation. The second one of these beings is probably impossible, but there seems to be a graduation where the more powerful a being is, the less interested it is in cooperation. This is alleviated by the uncertainty even the powerful beings will have about their relative rank in the power hierarchy, and norms around pride and honor and such where beings may punish others for defections even at some cost to themselves. I don’t see why these alleviating factors should precisely cancel out the greater power. I would rather expect there to be complex tradeoffs.
That all makes me inclined to think that the moral norms of more powerful beings should be weighed more (a bit less than proportionally more) than the moral norms of less powerful beings. These powerful beings are of course superintelligences and grabby aliens but also our ancestors. (Are there already writings on this? I’m only aware of a fairly short Brian Tomasik essay that came to a slightly different conclusion.)
Convergent drives. Self-preservation, value preservation, and resource acquisition may be convergent drives of powerful beings because they would otherwise perhaps not be powerful. Self-preservation implies existence in the first place, so that it may be optimal to help these powerful beings to come into existence (but which ones?). Value preservation is a difficult one since whatever powerful being ends up existing might’ve gotten there only because its previous versions still value-drifted. Resource acquisition may push toward instilling some indexical uncertainty in AIs so that they can figure out whether we’re in a simulation, and whether we can control or trade with higher levels or other branches. I feel highly unsure about these implications, but I think it would be important to get them right.
Idealization. But the consideration of our ancestors brings me to my next point of confusion, which is how to idealize norms. You mention the the distinction between instrumental and terminal goals. I think it’s an important one. E.g., maybe monogamy limited the spread of STDs among other benefits; some of these are less important now with tests, vaccinations, treatments, condoms, etc. So if our ancestors valued monogamy instrumentally, we don’t need to continue upholding it to cooperate with them even though they have a lot of power over our very existence. But if they valued it terminally, we might have to! Perhaps I really have to bite bullet on this one (if I understand you correctly) and turn it into a tradeoff between the interests of powerful people in the past with weird norms and the interests of less powerful people today…
Positive and negative reinforement. When it comes to the particular values, especially reinforcement learning stands out to me. When a being is much smaller than its part of the universe, it’ll probably need to learn, so my guess is that reinforcement learning is very common among all sorts of beings. Buck once argued that any kind of stimulation, positive or negative, expends energy, so that we’re calibrated such that most beings who survive and reproduce will spend most the time close the energy-conserving neutral state. That sounds pretty universal (though perhaps there are civilizations where only resources other than energy are scarce), so that the typical utilitarian values of increasing happiness and reducing suffering may be very common too.
(When I was new to EA, I read a lot of Wikipedia and SEP articles on various moral philosophies. My impression was that reducing suffering is “not dispreferred” by all of them (or to the extent that some of them might’ve been explicitly pro suffering, it was in specific, exceptional instances such as retribution) whereas all other norms were either dispreferred by some moral philosophies or really specific. My tentative guess at the time was that the maximally morally cooperative actions are those that reduced suffering while not running into any complicated tradeoffs against other values.)
Unique optimal ethics. An interesting observation is that ECL implies that there is one optimal compromise morality – that anyone who deviates from it leaves gains from moral trade on the table. So my own values should inform my actions only to a very minor degree. (I’m one sample among countless samples when it comes to inferring the distribution of values across the universe.) Instead my actions should be informed by the distribution of all values around the universe to the extent that I can infer it; by the distribution of relevant resources around the universe; and by the resources I have causally at my disposal, which could put me in a comparatively advantageous position to benefit some set of values. I wonder, is there one optimal acausal normalcy too?