Executive summary: This exploratory paper challenges Paul Christiano’s 2013 argument that the future is likely to be good by systematically critiquing its premises and introducing a population dynamics model, ultimately concluding that we are in a state of deep uncertainty about whether the future will be good—especially given the unpredictable influence of fanatics, shorttermists, and technological complexities.
Key points:
Christiano’s argument is based on three key assumptions: (1) those who care about the long-term (longtermists) will disproportionately influence the future; (2) these longtermists have values close to Christiano’s; and (3) if these two are true, the future will be good. The author challenges each of these premises.
A replicator dynamics model shows how aligned longtermists might dominate over time, but this outcome depends on optimistic assumptions—such as longtermists having consistently higher propagation rates than others—which are later shown to be fragile.
Introducing “fanatical” longtermists undermines the model’s optimistic conclusion, as such groups may propagate more effectively and have values radically different from Christiano’s, threatening a positive future.
Even if fanatics remain a minority, they could wield disproportionate influence, especially since they may act more decisively or ruthlessly than aligned longtermists, who tend to be more cautious and uncertain.
Shorttermists, too, could shape the future more than expected, as their willingness to act radically (e.g. for personal gain) may outweigh longtermist caution, undermining Christiano’s appeal to selection effects.
Even a future led solely by aligned longtermists could go awry, due to challenges in aligning AGI with ideal values, the unpredictability of emerging risks, and the inherent difficulty of achieving robustly good outcomes under deep uncertainty.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
Executive summary: This exploratory paper challenges Paul Christiano’s 2013 argument that the future is likely to be good by systematically critiquing its premises and introducing a population dynamics model, ultimately concluding that we are in a state of deep uncertainty about whether the future will be good—especially given the unpredictable influence of fanatics, shorttermists, and technological complexities.
Key points:
Christiano’s argument is based on three key assumptions: (1) those who care about the long-term (longtermists) will disproportionately influence the future; (2) these longtermists have values close to Christiano’s; and (3) if these two are true, the future will be good. The author challenges each of these premises.
A replicator dynamics model shows how aligned longtermists might dominate over time, but this outcome depends on optimistic assumptions—such as longtermists having consistently higher propagation rates than others—which are later shown to be fragile.
Introducing “fanatical” longtermists undermines the model’s optimistic conclusion, as such groups may propagate more effectively and have values radically different from Christiano’s, threatening a positive future.
Even if fanatics remain a minority, they could wield disproportionate influence, especially since they may act more decisively or ruthlessly than aligned longtermists, who tend to be more cautious and uncertain.
Shorttermists, too, could shape the future more than expected, as their willingness to act radically (e.g. for personal gain) may outweigh longtermist caution, undermining Christiano’s appeal to selection effects.
Even a future led solely by aligned longtermists could go awry, due to challenges in aligning AGI with ideal values, the unpredictability of emerging risks, and the inherent difficulty of achieving robustly good outcomes under deep uncertainty.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.