Thanks! I agree that this issue is very important—this is why intersubstrate welfare comparisons are one of the four main AI welfare research priorities that I discuss in the post. FYI, Bob Fischer (who you might know from the moral weight project at Rethink Priorities) and I have a paper in progress on this topic. We plan to share a draft in late July or early August, but the short version is that intersubstrate welfare comparisons are extremely important and difficult, and the main question is whether these comparisons are tractable. Bob and I think that the tractability of these comparisons is an open question, but we also think that we have several reasons for cautious optimism, and we discuss these reasons and call for more research on the topic.
With that said, one minor caveat: Even if you think that (a) all systems are potential welfare subjects and (b) we should give moral weight to all welfare subjects, you might or might not think that (c) we should give moral weight to all systems. The reason is that you might or might not think that we should give moral weight to extremely low risks. if you do, then yes, it follows that we should give at least some moral weight to all systems, including systems with an extremely low chance of being welfare subjects at all. If not, then it follows that we should give at least some moral weight to all systems with a non-negligible chance of being welfare subjects, but not to systems with only a negligible chance of being welfare subjects.
Thanks! I agree that this issue is very important—this is why intersubstrate welfare comparisons are one of the four main AI welfare research priorities that I discuss in the post. FYI, Bob Fischer (who you might know from the moral weight project at Rethink Priorities) and I have a paper in progress on this topic. We plan to share a draft in late July or early August, but the short version is that intersubstrate welfare comparisons are extremely important and difficult, and the main question is whether these comparisons are tractable. Bob and I think that the tractability of these comparisons is an open question, but we also think that we have several reasons for cautious optimism, and we discuss these reasons and call for more research on the topic.
With that said, one minor caveat: Even if you think that (a) all systems are potential welfare subjects and (b) we should give moral weight to all welfare subjects, you might or might not think that (c) we should give moral weight to all systems. The reason is that you might or might not think that we should give moral weight to extremely low risks. if you do, then yes, it follows that we should give at least some moral weight to all systems, including systems with an extremely low chance of being welfare subjects at all. If not, then it follows that we should give at least some moral weight to all systems with a non-negligible chance of being welfare subjects, but not to systems with only a negligible chance of being welfare subjects.