Executive summary: This exploratory essay proposes that marginal work on “Optimal Reflection”—humanity’s deliberate effort to figure out what it should do with the deep future—may be more valuable than AI safety work aimed at preventing extinction, due to its greater neglectedness, potentially high tractability, and essential role in avoiding suboptimal value lock-in; however, the author does not currently endorse any conclusions and seeks feedback on this early-stage model.
Key points:
Definition and significance of Optimal Reflection (OR): OR refers to the most effective, feasible, and actionable process for evaluating all crucial considerations about humanity’s future—examples include the Long Reflection, Coherent Extrapolated Volition, and Good Reflective Governance.
Preliminary model suggests OR may be ~5x higher EV than AI safety work: Despite the overwhelming focus on extinction prevention, early modeling indicates OR might yield more expected value on the current margin due to its neglectedness and high potential upside.
OR is especially crucial due to the risk of “value lock-in”: As humanity approaches the capability to irreversibly shape the future (e.g., via advanced AI), a small window remains to positively “lock in” the conditions for future flourishing; OR helps ensure this lock-in is intentional, inclusive, and wisely chosen.
Marginal tractability is the decisive crux: While both cause areas are vital, OR may currently offer more low-hanging fruit due to a smaller existing field (rough estimate: <100 people working on OR vs. ~1,200 in AI safety) and a large, motivated potential workforce.
Risks and dependencies acknowledged: The essay stresses that OR only matters conditional on surviving AI-related extinction risks, and political buy-in—especially from AI labs and policymakers—may be harder to secure for OR than for AI safety.
Author is seeking feedback and emphasizes tentativeness: The post is part of Existential Choices Debate Week, and the author is explicit about the speculative and preliminary nature of the claims, encouraging critical engagement from readers.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
Executive summary: This exploratory essay proposes that marginal work on “Optimal Reflection”—humanity’s deliberate effort to figure out what it should do with the deep future—may be more valuable than AI safety work aimed at preventing extinction, due to its greater neglectedness, potentially high tractability, and essential role in avoiding suboptimal value lock-in; however, the author does not currently endorse any conclusions and seeks feedback on this early-stage model.
Key points:
Definition and significance of Optimal Reflection (OR): OR refers to the most effective, feasible, and actionable process for evaluating all crucial considerations about humanity’s future—examples include the Long Reflection, Coherent Extrapolated Volition, and Good Reflective Governance.
Preliminary model suggests OR may be ~5x higher EV than AI safety work: Despite the overwhelming focus on extinction prevention, early modeling indicates OR might yield more expected value on the current margin due to its neglectedness and high potential upside.
OR is especially crucial due to the risk of “value lock-in”: As humanity approaches the capability to irreversibly shape the future (e.g., via advanced AI), a small window remains to positively “lock in” the conditions for future flourishing; OR helps ensure this lock-in is intentional, inclusive, and wisely chosen.
Marginal tractability is the decisive crux: While both cause areas are vital, OR may currently offer more low-hanging fruit due to a smaller existing field (rough estimate: <100 people working on OR vs. ~1,200 in AI safety) and a large, motivated potential workforce.
Risks and dependencies acknowledged: The essay stresses that OR only matters conditional on surviving AI-related extinction risks, and political buy-in—especially from AI labs and policymakers—may be harder to secure for OR than for AI safety.
Author is seeking feedback and emphasizes tentativeness: The post is part of Existential Choices Debate Week, and the author is explicit about the speculative and preliminary nature of the claims, encouraging critical engagement from readers.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.