I think your first section is correct, but you conclude far too much from it: failing to act in perfect accord with the moral truth does not mean you’re not influenced by it at all. Humans fail your conditions 4-7 and yet are occasionally influenced by moral facts in ways that matter.
On the one hand, you assume that civilizations are agents which can simply decide to adopt this or that strategy; on the other hand, you expect intense selection within civilizations, such that their members behave so as to maximize their own reproductive success. But these can’t both be true: you can throw all of your surplus into expanding as fast as possible, or you can spend it on internal competition, or you can do something in between, but you can’t spend it all on both.
I don’t think
> Within a civilization, we should expect the agents who have the values that are the most adapted/competitive to survival, replication, and expansion to eventually be selected for
is the right conclusion to draw from Hanson’s paper. The selection effects he’s talking about act on different regions of the frontier of a given civilization. Those living in the interior (who will be the vast majority of the population in the extreme long-run) may be disproportionately descended from fast expanders, but will not face the same pressure themselves.
There are many different units of selection, and they can’t all be subject to arbitrarily intense selection pressure simultaneously: cancer cells don’t build spaceships.
Insightful! Thanks for taking the time to write these.
failing to act in perfect accord with the moral truth does not mean you’re not influenced by it at all. Humans fail your conditions 4-7 and yet are occasionally influenced by moral facts in ways that matter.
Agreed and I didn’t mean to argue against that so thanks for clarifying! Note however that the more you expect the moral truth to be fragile/complex, the further from it you should expect agents’ actions to be.
you expect intense selection within civilizations, such that their members behave so as to maximize their own reproductive success.
Hum… I don’t think the “such that...” part logically follows. I don’t think this is how selection effects work. All I’m saying is that those who are the most bullish on space colonization will colonize more space.
I’m not sure what to say regarding your last two points. I think I need to think/read more, here. Thanks :)
All I’m saying is that those who are the most bullish on space colonization will colonize more space.
Sure, but that doesn’t tell you much about what happens afterwards. If the initial colonists’ values are locked in ~forever, we should probably expect value drift to be weak in general, which means frontier selection effects have a lot less variation to work with.
At the extreme lower limit with no drift at all, most agents within a mature civilization are about as expansionist as the most expansionist of the initial colonists—but no more so. And this might not be all that much in the grand scheme of things.
At the other end, where most of the space of possible values gets explored, maybe you do get a shockwave of superintelligent sociopaths racing outwards at relativistic speeds—but you also get a vast interior that favors (relatively speaking) long-term survival and material efficiency.
Several objections in no particular order
I think your first section is correct, but you conclude far too much from it: failing to act in perfect accord with the moral truth does not mean you’re not influenced by it at all. Humans fail your conditions 4-7 and yet are occasionally influenced by moral facts in ways that matter.
On the one hand, you assume that civilizations are agents which can simply decide to adopt this or that strategy; on the other hand, you expect intense selection within civilizations, such that their members behave so as to maximize their own reproductive success. But these can’t both be true: you can throw all of your surplus into expanding as fast as possible, or you can spend it on internal competition, or you can do something in between, but you can’t spend it all on both.
I don’t think
> Within a civilization, we should expect the agents who have the values that are the most adapted/competitive to survival, replication, and expansion to eventually be selected for
is the right conclusion to draw from Hanson’s paper. The selection effects he’s talking about act on different regions of the frontier of a given civilization. Those living in the interior (who will be the vast majority of the population in the extreme long-run) may be disproportionately descended from fast expanders, but will not face the same pressure themselves.
There are many different units of selection, and they can’t all be subject to arbitrarily intense selection pressure simultaneously: cancer cells don’t build spaceships.
Insightful! Thanks for taking the time to write these.
Agreed and I didn’t mean to argue against that so thanks for clarifying! Note however that the more you expect the moral truth to be fragile/complex, the further from it you should expect agents’ actions to be.
Hum… I don’t think the “such that...” part logically follows. I don’t think this is how selection effects work. All I’m saying is that those who are the most bullish on space colonization will colonize more space.
I’m not sure what to say regarding your last two points. I think I need to think/read more, here. Thanks :)
Sure, but that doesn’t tell you much about what happens afterwards. If the initial colonists’ values are locked in ~forever, we should probably expect value drift to be weak in general, which means frontier selection effects have a lot less variation to work with.
At the extreme lower limit with no drift at all, most agents within a mature civilization are about as expansionist as the most expansionist of the initial colonists—but no more so. And this might not be all that much in the grand scheme of things.
At the other end, where most of the space of possible values gets explored, maybe you do get a shockwave of superintelligent sociopaths racing outwards at relativistic speeds—but you also get a vast interior that favors (relatively speaking) long-term survival and material efficiency.