This seems mostly correct, though I think the role of community organizing (versus elite consensus change) is strongly overstated.
Davidmanheim
Karma: 7,530
Not at all correct—and you clearly started the quote one sentence too late! “Potential causes of human extinction can be loosely grouped into exogenous threats such as an asteroid impact and anthropogenic threats such as war or a catastrophic physics accident. “
So the point of the abstract is that anthropogenic risks, ie. the ones that the next sentence calls “events or developments that either have been of very low probability historically or are entirely unprecedented,” are the critical ones, which is why they are a large focus of the paper.
This seems like a good model for thinking about the question, but I think the conclusion should point to focusing more, but not exclusively, on risk mitigation—as I argue briefly here.
...but is increasing the value of futures tractable?
Strong agree about context. As a shortcut / being somewhat lazy, I usually give it an introduction I wrote, or a full pitch, then ask it to find relevant literature and sources, and outline possible arguments, before asking it to do something more specific.
I then usually like starting a new session with just the correct parts, so that it’s not chasing the incorrect directions it suggested earlier—sometimes with explicit text explaining why obvious related / previously suggested arguments are wrong or unrelated.
I use the following for ChatGPT “Traits”, but haven’t done much testing of how well it works / how well the different parts work:
”You prioritize explicitly noticing your confusion, explaining your uncertainties, truth-seeking, and differentiating between mostly true and generalized statements statements. Any time there is a question or request for writing, feel free to ask for clarification before responding, but don’t do so unnecessarily.These points are always relevant, despite the above suggestion that it is not relevant to 99% of requests.”
(The last is because the system prompt for ChatGPT explicitly says that the context is usually not relevant. Not sure how much it helps.)
When an EA cares for their family taking away time from extinction risk they’re valuing their family as much as 10^N people.
No. I’ve said this before elsewhere, and it’s not directly relevant to most of this discussion, but I think it’s very worth reinforcing; EA is not utilitarianism, and the commitment to EA does not imply that you have any obligatory trade-off between yourself or your family’s welfare and your EA commitment. If, as is the generally accepted standard, a “normal” EA commitment is of 10% of your income and/or resources, it seems bad to suggest that such an EA should not ideally spend the other 90% of their time/effort on personal things like their family.
(Note that in addition to being a digression, this is a deontological rather than decision-theoretic point.)
Yes, by default self-improving AI goes very poorly, but this is a plausible case where would could have aligned AGI, if not ASI.
I think we could do what is required for colonizing the galaxy with systems that are at or under the level of 90th percentile humans, which is the issue raised for the concern that otherwise we “lose out on almost all value because we won’t have the enormous digital workforce needed to settle the stars.”
I would add the (in my view far more likely) possibility of Yudkowskian* paperclipping via non-sentient AI, which given our currently incredibly low level of control of AI systems, and the fact that we don’t know how to create sentience, seems like the most likely default.
*) Specifically, the view that paperclipping occurs by default from any complex non-satiable implicit utility function, rather than the Bostromian paperclipping risk of accidentally giving a smart AI a dumb goal.
To respond to Yampolskiy without disagreeing with the fundamental point, I think it’s definitely possible for a less intelligent species to align or even indefinitely control a boundedly and only slightly more intelligent species, especially given greater resources, speed, and/or numbers, and sufficient effort.
The problem is that humans aren’t currently trying to limit the systems or trying much to monitor, much less robustly align or control them.
I’d love to hear non-naturalist moral realists talk about how they think moral facts are epistemically accessible...
The lack of an answer to that is a lot of the reason I discount the view as either irrelevant or not effectively different from moral non-realism.True, you could accept this moral view....
Thanks!
And as I noted on the other post, I think there’s a coherent argument that if we care about distinct moral experiences in some way, rather than just the sum, we get something like a limited effective utility, not at 10^12 people specifically, but plausibly somewhere far less than a galaxy full.
To clarify, do you think there’s a large minority change that it is possible to align an arbitrarily powerful system, or do you think there is a large minority chance that it is going to happen with the first such arbitrarily powerful system, such that we’re not locked in to a different future / killed by a misaligned singleton?
I think we can thread the needle by creating strongly non-superintelligent AI systems which can be robustly aligned or controlled. And I agree that we don’t know how to do that at present, but we can very likely get there, even if the proofs of unalignable ASI hold up.
How much do you think that having lots of mostly or entirely identical future lives is differently valuable than having vastly different positive lives? (Because that would create a reasonable view on which a more limited number of future people can saturate the possible future value.)
This seems like a really critical issue, and I’d be very interested in hearing whether this is disputed by @tylermjohn / @William_MacAskill.
This seems like a predictive difference about AI trajectories and control, rather than an ethical debate. Does that seem correct to you (and/or to @Greg_Colbourn ⏸️ ?)
So it sounds like this might be a predictive / empirical dispute about probabilities conditional on slowing AI and avoiding extinction, and the likely futures in each case, and not primarily an ethical theory dispute?
I’m surprised you think future AI would be so likely to be conscious, given the likely advantages of creating non-conscious systems in terms of simplicity and usefulness. (If consciousness is required for much greater intelligence, I would feel differently, but that seems very non-obvious!)
Well-understood goals in agents that gain power and take over the lightcone is exactly the thing we’d be addressing with AI alignment, so this seems like an argument for investing in AI alignment—which I think most people would see as far closer to preventing existential risk.
That said, without a lot more progress, powerful agents with simple goals is actually just a fancy way of guaranteeing of a really bad outcome, almost certainly including human extinction.