Wei Dai
For example, in What We Owe The Future, Will said he thought that the expected value of the future, given survival, was less than 1% of what it might be.1 After being exposed to some of the arguments in this essay, he revised his views closer to 10%; after analysing them in more depth, that percentage dropped a little bit, to 5%-10%.
[...]
However, it’s unlikely to me that companies will in fact produce morally uncertain AIs that are motivated by doing good de dicto. They probably won’t have thought about this issue, and won’t be motivated by trying to improve scenarios in which humanity is disempowered.
Given this combination of views, I’m surprised that Will doesn’t support what @Holly Elmore ⏸️ 🔸 calls “Pause NOW” and instead want to see a pause later (after we have human-level AI). I’m curious if your own views are similar or how they differ from Will’s. (My own “expected value of the future, given survival” I would say is similarly pessimistic, but I’m reluctant to put into numbers due to being very unsure how to quantify it.)
Aside from what Holly said in the linked comment, which I agree with, another argument more relevant to the current discussion is that many opportunities for making the future better seem to exist during the AI transition, including the early parts of it, so by not pausing ASAP (and currently having few resources for such interventions), we’re permanently giving up these opportunities. Conversely, by pausing NOW, we buy more time to think and strategize about how to better intervene on these opportunities, or otherwise lay the groundwork for them.
For example, during the pause, we could:
Try to solve metaphilosophy, or otherwise think about how to improve AI philosophical competence or moral epistemology.
Try to get AI companies to “think about this issue” (of morally uncertain AIs that are motivated by doing good de dicto).
Research ways to make such AIs safer from our (human) perspective so that there’s less of a tradeoff between safety and Better Futures.
Spread the idea of Better Futures generally so that when AI development resumes, there will be more people aware of and working on these issues.
Such interventions could mean the difference between the first human-level AIs being competent and critical moral/philosophical advisors, or independent moral (and safe) agents, vs uncritically doing what humans seem to want and/or giving bad/incompetent/sycophantic “advice” (when humans think to ask for it), which seemingly can make a big difference to how well the future goes.
What do you think about this argument, and overall about pause now vs later?
Thanks for clarifying. Sill, this suggests that the Chinese participants were on average much less conscientious about answering truthfully/carefully than the US/UK ones, which implies that even the filtered samples may still be relatively more noisy.
Perplexity w/ GPT-5.2 Thinking when I asked “Are there standard methods for dealing with this in surveying/statistics?”, among other ideas (sorry I don’t know how good the answer actually is):
Model carelessness (don’t only drop)
If you’re worried that “filtered China” still contains more residual noise, a more formal option is to use response-time mixture models / latent-class approaches that treat careful vs. careless responding as latent states and use RT information to infer them (reducing reliance on arbitrary RT cutoffs).
This can yield posterior probabilities of careless responding that you can use to down-weight or run analyses both with and without likely-careless respondents.
If I’m interpreting this correctly, 25% of people in China think that at least 58% of all people in the world deserve eternal unbearable pain (with similar results in 3 other countries). This is so crazy that I think there must be another explanation, e.g., results got mixed up, or a lot of people weren’t paying attention and just answered randomly.
Cultivating people’s ability and motivation to reflect on their values.
Structuring collective deliberations so that better arguments and ideas win out over time.
Problems with this:
People disagree strongly about which arguments and ideas are better than others.
The vast majority of people seem to lack both the hardware (raw cognitive capacity) and software (a reasonable philosophical tradition) to reflect on their values in a positive way.
AI seems likely to make things worse or not sufficiently better, for these reasons.
Solving these problems seem to require (at least) some kind of breakthrough in metaphilosophy, such that we have a solution for what is actually the right way to reflect/deliberate about philosophical topics like values, and the solution is so convincing that everyone ends up agreeing with it. But I would love to know if you (or anyone else) have other ideas for solving or getting around these problems for building a viable Viatopia.
Another problem is that morality/values is in large part a status game, but talking about status is generally bad for one’s status (who wants to admit that they’re espousing some morality/values to win a status game) so this important aspect of morality/values is generally ignored for structural reasons that may be unfixable regardless of other technical and philosophical advances.
See also this post, which occurred to me after writing my previous reply to you.
@Wei Dai, I understand that your plan A is an AI pause (+ human intelligence enhancement). And I agree with you that this is the best course of action. Nonetheless, I’m interested in what you see as plan B: If we don’t get an AI pause, is there any version of ‘hand off these problems to AIs’/ ‘let ‘er rip’ that you feel optimistic about? or which you at least think will result in lower p(catastrophe) than other versions? If you have $1B to spend on AI labour during crunch time, what do you get the AIs to work on?
The answer would depend a lot on what the alignment/capabilities profile of the AI is. But one recent update I’ve made is that humans are really terrible at strategy (in addition to philosophy) so if there was no way to pause AI, it would help a lot to get good strategic advice from AI during crunch time, which implies that maybe AI strategic competence > AI philosophical competence in importance (subject to all the usual disclaimers like dual use and how to trust or verify its answers). My latest LW post has a bit more about this.
(By “strategy” here I especially mean “grand strategy” or strategy at the highest levels, which seems more likely to be neglected versus “operational strategy” or strategy involved in accomplishing concrete tasks, which AI companies are likely to prioritize by default.)
So for example if we had an AI that’s highly competent at answering strategic questions, we could ask it “What questions should I be asking you, or what else should I be doing with my $1B?” (but this may have to be modified based on things like how much can we trust its answers of various kinds, how good is it at understanding my values/constraints/philosophies, etc.).
If we do manage to get good and trustworthy AI advice his way, another problem would be how to get key decision makers (including the public) to see and trust such answers, as they wouldn’t necessarily think to ask such questions themselves nor by default trust the AI answers. But that’s another thing that a strategically competent AI could help with.
BTW your comment made me realize that it’s plausible that AI could accelerate strategic thinking and philosophical progress much more relative to science and technology, because the latter could become bottlenecked on feedback from reality (e.g., waiting for experimental results) whereas the former seemingly wouldn’t be. I’m not sure what implications this has, but want to write it down somewhere.
Moreover, above we were comparing AIs to the best human philosophers / to a well-organised long reflection, but the actual humans calling the shots are far below that bar. For instance, I’d say that today’s Claude has better philosophical reasoning and better starting values than the US president, or Elon Musk, or the general public. All in all, best to hand off philosophical thinking to AIs.
One thought I have here is that AIs could give very different answers to different people. Do we have any idea what kind of answers Grok is (or will be) giving to Elon Musk when it comes to philosophy?
I wish you titled the post something like “The option value argument for preventing extinction doesn’t work”. Your current title (“The option value argument doesn’t work when it’s most needed”) has the unfortunate side effects of:
People being more likely to misinterpret or misremember your post as claiming that trying to increase option value doesn’t work in general.
Reducing extinction risk becomes the most salient example of an idea for increasing option value.
People using “the option value argument” to mean the the option value argument for preventing extinction, even when this can’t be inferred from context. (See example.)
It’s harder to use the phrase “the option value argument” contextually to refer to the option value argument currently or previously discussed, when it’s not about extinction risk, due to it becoming a term of art for “the option value argument for preventing extinction”.
I think it may not be too late to change the title and stop or reverse these effects.
The argument tree (arguments, counterarguments, counter-counterarguments, and so on) is exponentially sized and we don’t know how deep or wide we need to expand it, before some problem can be solved. We do know that different humans looking at the same partial tree (i.e., philosophers who have read the same literature on some problem) can have very different judgments as to what the correct conclusion is. There’s also a huge amount of intuition/judgment involved in choosing which part of the tree to focus on or expand further. With AIs helping to expand the tree for us, there are potential advantages like you mentioned, but also potential disadvantages, like AIs not having good intuition/judgment about what lines of arguments to pursue, or the argument tree (or AI-generated philosophical literature) becoming too large for any humans to read and think about in a relevant time frame. Many will be very tempted to just let AIs answer the questions / make the final conclusions for us, especially if AIs also accelerate technological progress, creating many urgent philosophical problems related to how to use them safely and beneficially. Or if humans try to make the conclusions, can easily get them wrong despite AI help with expanding the argument tree.
So I think undergoing the AI transition without solving metaphilosophy, or making AIs autonomously competent at philosophy (good at getting correct conclusions by themselves) is enormously risky, even if we have corrigible AIs helping us.
Do you want to talk about why you’re relatively optimistic? I’ve tried to explain my own concerns/pessimism at https://www.lesswrong.com/posts/EByDsY9S3EDhhfFzC/some-thoughts-on-metaphilosophy and https://forum.effectivealtruism.org/posts/axSfJXriBWEixsHGR/ai-doing-philosophy-ai-generating-hands.
Thanks! I hope this means you’ll spend some more time on this type of work, and/or tell other philosophers about this argument. It seems apparent that we need more philosophers to work on philosophical problems related to AI x-safety (many of which do not seem to be legible to most non-philosophers). Not necessarily by attacking them directly (this is very hard and probably not the best use of time, as we previously discussed) but instead by making them more legible to AI researchers, decisionmakers, and the general public.
In my view, there isn’t much desire for work like this from people in the field and they probably wouldn’t use it to inform deployment unless a lot of effort is also added from the author to meet the right people, convince theme to spend the time to take it seriously etc.
Any thoughts on Legible vs. Illegible AI Safety Problems, which is in part a response to this?
I wrote a post that I think was partly inspired by this discussion. The implication of it here is that I don’t necessarily want philosophers to directly try to solve the many hard philosophical problems relevant to AI alignment/safety (especially given how few of them are in this space or concerned about x-safety), but initially just to try to make them “more legible” to others, including AI researchers, key decision makers, and the public. Hopefully you agree that this is a more sensible position.
I agree that many of the problems on my list are very hard and probably not the highest marginal value work to be doing from an individual perspective. Keep in mind that the list was written 6 years ago, when it was less clear when the AI takeoff would start in earnest, or how many philosophers will become motivated to work on AI safety when AGI became visibly closer. I still had some hope that when the time came, a significant fraction of all philosophers would become self-motivated or would be “called to arms” by a civilization-wide AI safety effort, and would be given sufficient resources including time, so the list was trying to be more comprehensive (listing every philosophical problem that I thought relevant to AI safety) than prioritizing. Unfortunately, the reality is nearly the completely opposite of this.
Currently, one of my main puzzles is why philosophers with public AI x-risk estimates still have numbers in the 10% range, despite reality being near the most pessimistic of my range of expectations, and it looking like that the AI takeoff/transition will occur while most of these philosophical problems will remain in a wide open or totally confused state, and AI researchers seem almost completely oblivious or uncaring about this. Why are they not making the same kind of argument that I’ve been making, that philosophical difficulty is a reason that AI alignment/x-safety is harder than many think, and an additional reason to pause/stop AI?
Right, I know about Will MacAskill, Joe Carlsmith, and your work in this area, but none of you are working on alignment per se full time or even close to full time AFAIK, and the total effort is clearly far from adequate to the task at hand.
I think some have given up philosophy to work on other things such as AI alignment.
Any other names you can cite?
In my view, there isn’t much desire for work like this from people in the field and they probably wouldn’t use it to inform deployment unless a lot of effort is also added from the author to meet the right people, convince them to spend the time to take it seriously etc.
Thanks, this makes sense to me, and my follow-up is how concerning do you think this situation is?
One perspective I have is that at this point, several years into a potential AI takeoff, with AI companies now worth trillions in aggregate, alignment teams at AI companies still have virtually no professional philosophical oversight (or outside consultants that they rely on), and are kind of winging it based on their own philosophical beliefs/knowledge. It seems rather like trying to build a particle collider or fusion reaction with no physicists on the staff, only engineers.
(Or worse, unlike engineers’ physics knowledge, I doubt that receiving a systematic education in fields like ethics and metaethics is a hard requirement for working as an alignment researcher. And even worse, unlike the situation in physics, we don’t even have settled ethics/metaethics/metaphilosophy/etc. that alignment researchers can just learn and apply.)
Maybe the AI companies are reluctant to get professional philosophers involved, because in the fields that do have “professional philosophical oversight”, e.g., bioethics, things haven’t worked out that well. (E.g. human challenge trials being banned during COVID.) But to me, this would be a signal to yell loudly that our civilization is far from ready to attempt or undergo an AI transition, rather than a license to wing it based on one’s own philosophical beliefs/knowledge.
As an outsider, the situation seems cray alarming to me, and I’m confused that nobody else is talking about it, including philosophers like you who are in the same overall space and looking at roughly the same things. I wonder if you have a perspective that makes the situation not quite as alarming as it appears to me.
Do you have any insights into why there are so few philosophers working in AI alignment, or closely with alignment researchers? (Amanda Askell is the only one I know.) Do you think this is actually a reasonable state of affairs (i.e., it’s right or fine that almost no professional philosophers work directly as or with alignment researchers), or is this wrong/suboptimal, caused by some kind of cultural or structural problem? It’s been 6 years since I wrote Problems in AI Alignment that philosophers could potentially contribute to and I’ve gotten a few comments from philosophers saying they found the list helpful or that they’ll think about working on some of the problems, but I’m not aware of any concrete follow-ups.
If it is some kind of cultural or structural problem, it might be even higher leverage to work on solving that, instead of object level philosophical problems. I’d try to do this myself, but as an outsider to academic philosophy and also very far from any organizations who might potentially hire philosophers to work on AI alignment, it’s hard for me to even observe what the problem might be.
Interesting re belief in hell being a key factor, I wasn’t thinking about that.
It seems like the whole AI x-risk community has latched onto “align AI with human values/intent” as the solution, with few people thinking even a few steps ahead to “what if we succeeded”? I have a post related to this if you’re interested.
possibly the future economy will be so much more complicated that it will still make sense to have some distributed information processing in the market rather than have all optimisation centrally planned
I think there will be distributed information processing, but each distributed node/agent will be a copy of the central AGI (or otherwise aligned to it or shares its values), because this is what’s economically most efficient, minimizes waste from misaligned incentives and so on. So there won’t be the kind of value pluralism that we see today.
I assume we won’t be able to know with high confidence in advance what economic model will be most efficient post-ASI.
There’s probably a lot of other surprises that we can’t foresee today. I’m mostly claiming that post-AGI economics and governance probably wont look very similar to today’s.
Why do you think this work has less value than solving philosophical problems in AI safety?
From the perspective of comparative advantage and counterfactual impact, this work does not seem to require philosophical training. It seems to be straightforward empirical research, that many people could do, besides the very few professionally trained AI-risk-concerned philosophers that humanity has.
To put it another way, I’m not sure that Toby was wrong to work on this, but if he was, it’s because if he hadn’t, then someone else with more comparative advantage for working on this problem (due to lacking training or talent for philosophy) would have done so shortly afterwards.
While I appreciate this work being done, it seems a very bad sign for our world/timeline that the very few people with both philosophy training and an interest in AI x-safety are using their time/talent to do forecasting (or other) work instead of solving philosophical problems in AI x-safety, with Daniel Kokotajlo being another prominent example.
This implies one of two things: Either they are miscalculating the best way to spend their time, which indicates bad reasoning or intuitions even among humanity’s top philosophers (i.e., those who have at least realized the importance of AI x-risk and are trying to do something about it). Or they actually are the best people (in a comparative advantage sense) available to work on these other problems, in which case the world must be on fire, and they’re having to delay working on extremely urgent problems that they were trained for, to put out even bigger fires.
(Cross-posted to LW and EAF.)
The ethical schools of thought I’m most aligned with—longtermism, sentientism, effective altruism, and utilitarianism—are far more prominent in the West (though still very niche).
I want to point out that the ethical schools of thought that you’re (probably) most anti-aligned with (e.g., that certain behaviors and even thoughts are deserving of eternal divine punishment) are also far more prominent in the West, proportionately even more so than the ones you’re aligned with.
Also the Western model of governance may not last into the post-AGI era regardless of where the transition starts. Aside from the concentration risk mentioned in the linked post, driven by post-AGI economics, I think different sub-cultures in the West breaking off into AI-powered autarkies or space colonies with vast computing power, governed by their own rules, is also a very scary possibility.
I’m pretty torn and may actually slightly prefer a CCP-dominated AI future (despite my family’s past history with the CCP). But more importantly I think both possibilities are incredibly risky if the AI transition occurs in the near future.
Why assume that there can only be one pause? Pausing now could make a later pause both more likely and more useful, by building the infrastructure and precedent for pausing, and by making subsequent AIs more aligned and differentially more productive in areas that we care about. If we end the first pause only after we’ve solved the problem of building aligned AIs that are philosophically and strategically competent, that would seemingly make subsequent pauses much easier.
I wonder if you’re thinking that we won’t be able to pause long enough to make significant progress on these problems? I can see that if we only have the “willpower” for a single short pause, then it becomes unclear when to best use it.
I have been warning for several years that AI could be differentially bad at philosophy and long-horizon strategy (due in part to AI training requiring massive amounts of training data and/or fast and cheap feedback loops, which are lacking for these fields, and in part to lack of understanding of e.g. metaphilosophy). So if we don’t pause now (and use the time to fix this issue) then by the time we do pause, we’ll likely have AIs that can accelerate other fields (such as math/coding/science/tech and manipulating humans) much more than the fields that are crucial for Better Futures.
Worse, we may end up with AIs that decelerate (in an absolute sense) hard-to-verify fields like philosophy and long-horizon strategy, because these AIs are better at coming up with plausible sounding ideas and arguments, and convincing humans of their truth, or persuading humans that their own bad ideas are actually good (which is already being reported under “AI psychosis” and “sycophancy”), than making real progress in these fields.