Thanks for the food for thought. I thought I’d share three related notes on framing which might be relevant to the rest of your series:
1) Tiny probabilities appear to not be fundamental to long-termism. The mugging argument you attribute to long-termists is indeed espoused as the key argument in favour by some, especially on online forums, but many people researching these problems assign non-mugging (e.g. ~1%) probability of their organizations having an overall large effect. For example, Yudkowsky in the interesting piece you linked (thanks for that):
You cannot justifiably trade off tiny probabilities of x-risk improvement against efforts that do not effectuate a happy intergalactic civilization, but there is nonetheless no need to go on tracking tiny probabilities when you’d expect there to be medium-sized probabilities of x-risk reduction.
It would be excellent to see someone write this up, since the divergence in argumentation by different parties interested in long-termism is large.
2) Projects tend not to have binary outcomes, and may have large potential positive and negative effects which have unclear sign on net. This makes robustness considerations (Knightian uncertainty, confidence in the sign, etc.) somewhere between quite important and the central consideration. This is the key reason why I pay little attention to the mugging arguments, which typically assume no negative outcomes without justifying this assumption. Instead, I think that the strongest case for aiming directly at improving the long-run may revolve around the robustness gained from aiming directly at where most of the expected value is (if one considers future generations to be morally relevant). Might be valuable to explore the relative merits of these approaches.
3) Consider explicitly separating the overall promisingness of a project from the marginal effect of additional resources. This is relevant e.g. for case ‘You give $7,000 to the Future of Humanity Institute, and your donation makes the difference between human extinction and long-term flourishing.’ i.e. consider separating out ‘your donation makes the difference’ from ‘the Future of Humanity Institute has a large positive impact’. When looking at marginal contributions, these things can get conflated. For example, there is a low probability that there has been or will be a distribution of bednets which would not have happened had I not donated $3,000 to AMF in 2014, but this uncertainty does not worry me. Uncertainty about whether increasing economic growth is good is a much larger deal. It looks like Eliezer summarised this well:
In this case the average marginal added dollar can only account for a very tiny slice of probability, but this is not Pascal’s Wager. Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginally increased probabilities of success per added small unit of effort. It would only be Pascal’s Wager if the whole route-to-an-OK-outcome were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.
Based on your ‘Summary’ section, I suspect that you are already intending to tackle some of these points in ‘Sheltering in the herd’ or elsewhere. Good luck!
Thanks for the comments—yeah, future posts are going to discuss these topics, though not necessarily using this terminology. In general I’m engaging at a rather foundational level: why is Knightian uncertainty importantly different from risk, and why is the distinction Yudkowsky mentions at the end a good and legitimate distinction to make?
Thanks for the food for thought. I thought I’d share three related notes on framing which might be relevant to the rest of your series:
1) Tiny probabilities appear to not be fundamental to long-termism. The mugging argument you attribute to long-termists is indeed espoused as the key argument in favour by some, especially on online forums, but many people researching these problems assign non-mugging (e.g. ~1%) probability of their organizations having an overall large effect. For example, Yudkowsky in the interesting piece you linked (thanks for that):
It would be excellent to see someone write this up, since the divergence in argumentation by different parties interested in long-termism is large.
2) Projects tend not to have binary outcomes, and may have large potential positive and negative effects which have unclear sign on net. This makes robustness considerations (Knightian uncertainty, confidence in the sign, etc.) somewhere between quite important and the central consideration. This is the key reason why I pay little attention to the mugging arguments, which typically assume no negative outcomes without justifying this assumption. Instead, I think that the strongest case for aiming directly at improving the long-run may revolve around the robustness gained from aiming directly at where most of the expected value is (if one considers future generations to be morally relevant). Might be valuable to explore the relative merits of these approaches.
3) Consider explicitly separating the overall promisingness of a project from the marginal effect of additional resources. This is relevant e.g. for case ‘You give $7,000 to the Future of Humanity Institute, and your donation makes the difference between human extinction and long-term flourishing.’ i.e. consider separating out ‘your donation makes the difference’ from ‘the Future of Humanity Institute has a large positive impact’. When looking at marginal contributions, these things can get conflated. For example, there is a low probability that there has been or will be a distribution of bednets which would not have happened had I not donated $3,000 to AMF in 2014, but this uncertainty does not worry me. Uncertainty about whether increasing economic growth is good is a much larger deal. It looks like Eliezer summarised this well:
Based on your ‘Summary’ section, I suspect that you are already intending to tackle some of these points in ‘Sheltering in the herd’ or elsewhere. Good luck!
Thanks for the comments—yeah, future posts are going to discuss these topics, though not necessarily using this terminology. In general I’m engaging at a rather foundational level: why is Knightian uncertainty importantly different from risk, and why is the distinction Yudkowsky mentions at the end a good and legitimate distinction to make?