Thank you for engaging, and especially for the intelligence curse point—that’s exactly the structural issue I’m trying to get at.
You suggest I’m arguing “we should care about some of those things intrinsically.” Let me use AGI as an example to show why I don’t think this is about intrinsic value at all:
What would an AGI need to persist for a million years?
Not “what targets should it optimize for” but “what maintains the AGI itself across that timespan?”
I think the answer is: diversity (multiple approaches for unforeseen challenges), error correction (detecting when models fail), adaptive capacity (sensing and learning, not just executing), and substrate maintenance (keeping the infrastructure running).
An AGI optimizing toward distant targets while destroying these properties would be destroying its own substrate for persistence. The daily maintenance—power, sensors, error detection—isn’t preparation for the target. It IS what persistence consists of.
I think the same logic applies to longtermist societies. The question would shift from “how to allocate resources between present and future” to “are we maintaining or destroying the adaptive loop properties that enable any future to exist?” That changes what institutions would need to do—the essay explores some specific examples of what this might look like.
Does the AGI example help clarify the reframe I’m proposing?
Thanks for the generous response. You write that we “may have been compatible” and I’m “reacting to something you’re not saying.”
Here’s my concern: I’ve come to recognize that reality operates as a dynamic network—nodes (people, institutions) whose capacity is constituted by the relationships among them. This isn’t just a modeling choice; it’s how cities function, how pandemics spread, how states maintain capacity. You don’t work from this explicit recognition.
This creates an asymmetry. Once you see reality as a network, your Section 5 framework becomes incompatible with mine—not just incomplete, but incoherent. You explicitly frame the state as separate from people, optimizing for longtermist goals while managing preferences as constraints. But from the network perspective, this separation doesn’t exist—the state’s capacity just IS those relationships. You can’t optimize one while managing the other.
Let me try to say this more directly: I’ve come to understand my own intelligence as existing not AT my neurons, but BETWEEN them—as a pattern of activation across connections. I am the edge, not the node. And I see society the same way: capacity isn’t located IN institutions, it emerges FROM relationships. From this perspective, your Section 5 (state separate from people) isn’t a simplification—it’s treating edges as if they were nodes, which fundamentally misunderstands what state capacity is.
That’s the asymmetry: your explicit framing (state separate from people) is incompatible with how I now understand reality. But if you haven’t recognized the network structure, you’d just see my essay as “adding important considerations” rather than revealing a foundational incompatibility.
Does this help clarify where I’m coming from?