Good idea, I reposted the article itself here: https://forum.effectivealtruism.org/posts/GyenLpfzRKK3wBPyA/the-simple-case-for-ai-catastrophe-in-four-steps
I’ve been trying to keep the “meta” and the main posts mostly separate so hopefully the discussions for the metas and the main posts aren’t as close together.
I’ve now written it here, thanks for all the feedback! :) https://linch.substack.com/p/simplest-case-ai-catastrophe
The bets I’ve seen you post seem rather disadvantageous to the other side, and I believed so at the time. Which is fine/good business from your perspective given that you managed to find takers. But it means I’m more pessimistic on finding good deals by both of our lights.
Hmm right now this seems wrong to me, and also not worth going into in an introductory post. Do you have a sense that your view is commonplace? (eg from talking to many people not involved in AI)
Here’s my current four-point argument for AI risk/danger from misaligned AIs.
We are on the path of creating intelligences capable of being better than humans at almost all economically and militarily relevant tasks.
There are strong selection pressures and trends to make these intelligences into goal-seeking minds acting in the real world, rather than disembodied high-IQ pattern-matchers.
Unlike traditional software, we have little ability to know or control what these goal-seeking minds will do, only directional input.
Minds much better than humans at seeking their goals, with goals different enough from our own, may end us all, either as a preventative measure or side effect.
Request for feedback: I’m curious whether there are points that people think I’m critically missing, and/or ways that these arguments would not be convincing to “normal people.” Original goal.
What are people’s favorite arguments/articles/essays trying to lay out the simplest possible case for AI risk/danger?
Every single argument for AI danger/risk/safety I’ve seen seems to overcomplicate things. Either they have too many extraneous details, or they appeal to overly complex analogies, or they seem to spend much of their time responding to insider debates.
I might want to try my hand at writing the simplest possible argument that is still rigorous and clear, without being trapped by common pitfalls. To do that, I want to quickly survey the field so I can learn from the best existing work as well as avoid the mistakes they make.
I often see people advocate others sacrifice their souls. People often justify lying, political violence, coverups of “your side’s” crimes and misdeeds, or professional misconduct of government officials and journalists, because their cause is sufficiently True and Just. I’m overall skeptical of this entire class of arguments.
This is not because I intrinsically value “clean hands” or seeming good over actual good outcomes. Nor is it because I have a sort of magical thinking common in movies, where things miraculously work out well if you just ignore tradeoffs.
Rather, it’s because I think the empirical consequences of deception, violence, criminal activity, and other norm violations are often (not always) quite bad, and people aren’t smart or wise enough to tell the exceptions apart from the general case, especially when they’re ideologically and emotionally compromised, as is often the case.
Instead, I think it often helps to be interpersonally nice, conduct yourself with honor, and overall be true to your internal and/or society-wide notions of ethics and integrity.
I’m especially skeptical of galaxy-brained positions where to be a hard-nosed consequentialist or whatever, you are supposed to do a specific and concrete Hard Thing (usually involving harming innocents) to achieve some large, underspecified, and far-off positive outcome.
I think it’s like those thought experiments about torturing a terrorist (or a terrorist’s child) to find the location of the a ticking nuclear bomb under Manhattan where somehow you know the torture would do it.
I mean, sure, if presented that way I’d think it’s a good idea but has anybody here checked the literature on the reliability of evidence extracted under torture? Is that really the most effective interrogation technique?
So many people seem eager to rush to sell their souls, without first checking to see if the Devil’s willing to fulfill his end of the bargain.
Thanks! I agree the math isn’t exactly right. The point about x^2 on the rationals is especially sharp.
The problem with calling it “the paradox of the heap” is to make it sound like an actual paradox, instead of a trivially easy connection re:tipping points. I wish I had a better terminology/phrase for the connection I want to make.
Thanks, the feeling is mutual.
Happy holidays to you too.
I think your comment largely addresses a version of the post that doesn’t exist.
In brief:
I don’t think I claimed novelty; the post is explicitly about existing concepts that seem obvious once you have them. I even used specific commonly known terms for them.
Theory of mind, mentalization, cognitive empathy, and perspective taking are, of course, not actually “rare” but are what almost all people are doing almost all the time. The interesting question is what kinds of failures you think are common. The more opinionated you are about this, and the more you diverge from consensus opinions of experts such as psychologists and researchers in social work, the more likely you are to be wrong.
The post gave specific examples of people with the capacity for ToM nonetheless failing to consistently apply it to political outgroups, foreign adversaries, story characters etc. Also the specific wording I wrote was:
The core idea is very simple: treat other agents as real. It sounds banal, until you realize how rare it can be, and how frequently people mess up.”
You harp on the word “rare” but miss the surrounding context. You consistently make technically true but irrelevant points.
so if the point is to understand world hunger or global poverty, it would be a better idea to just read an introductory text on international development than to think further about how the concept of net present value might or might not shed new light on global poverty.
Are you seriously implying that it takes less effort to read an entire textbook on developmental economics than it is to write a paragraph on a related question? Besides, that wasn’t the point of the post anyway, which was more like “here’s a specific conceptual error people make, NPV dissolves it.”
I don’t think anybody disagrees that ideas matter. I would say everyone agrees with that.
This blog post initially grew out of a conversation with a popular blogger about whether ideas actually matter. It’s also commonly believed in Silicon Valley that ideas are almost irrelevant compared to execution.
I personally don’t find any value in Grice’s maxims.
Clearly.
I had the same initial reaction! I’d guess others would have the same misreading too, so it’s worth rewriting. fyi @Yulia Chekhovska
For Inkhaven, I wrote 30 posts in 30 days. Most of them are not particularly related to EA, though a few of them were. I recently wrote some reflections. @Vasco Grilo🔸 thought it might be a good idea to share on the EA Forum; I don’t want to be too self-promotional so I’m splitting the difference and posting just a shortform link here:
https://linch.substack.com/p/30-posts-in-30-days
The most EA-relevant posts are probably
https://inchpin.substack.com/p/skip-phase-3
https://inchpin.substack.com/p/aging-has-no-root-cause
https://inchpin.substack.com/p/legible-ai-safety-problems-that-dont
There are a number of implicit concepts I have in my head that seem so obvious that I don’t even bother verbalizing them. At least, until it’s brought to my attention other people don’t share these concepts.
It didn’t feel like a big revelation at the time I learned the concept, just a formalization of something that’s extremely obvious. And yet other people don’t have those intuitions, so perhaps this is pretty non-obvious in reality.
Here’s a short, non-exhaustive list:
Intermediate Value Theorem
Net Present Value
Differentiable functions are locally linear
Theory of mind
Grice’s maxims
If you have not heard any of these ideas before, I highly recommend you look them up! Most *likely*, they will seem obvious to you. You might already know those concepts by a different name, or they’re already integrated enough into your worldview without a definitive name.
However, many people appear to lack some of these concepts, and it’s possible you’re one of them.
As a test: for every idea in the above list, can you think of a nontrivial real example of a dispute where one or both parties in an intellectual disagreement likely failed to model this concept? If not, you might be missing something about each idea!
Thanks, I find the polls to be much stronger evidence than the other things you’ve said.
My overall objection/argument is that you appear to selectively portray data points that show one side, and selectively dismiss data points that show the opposite view. This makes your bottom-line conclusion pretty suspicious.
I also think the rationalist community overreached and their epistemics and speed in early COVID were worse compared to, say, internet people, government officials, and perhaps even the general public in Taiwan. But I don’t think the case for them being slower than Western officials or the general public in either the US or Europe is credible, and your evidence here does not update me much.
Why does this not apply to your original point citing a single NYT article?
I like Scott’s Mistake Theory vs Conflict Theory framing, but I don’t think this is a complete model of disagreements about policy, nor do I think the complete models of disagreement will look like more advanced versions of Mistake Theory + Conflict Theory.
To recap, here’s my short summaries of the two theories:
Mistake Theory: I disagree with you because one or both of us are wrong about what we want, or how to achieve what we want)
Conflict Theory: I disagree with you because ultimately I want different things from you. The Marxists, who Scott was originally arguing against, will natively see this as about individual or class material interests but this can be smoothly updated to include values and ideological conflict as well.
I polled several people about alternative models for political disagreement at the same level of abstraction of Conflict vs Mistake, and people usually got to “some combination of mistakes and conflicts.” To that obvious model, I want to add two other theories (this list is incomplete).
Consider Thomas Schelling’s 1960 opening to Strategy of Conflict
I claim that this “rudimentary/obvious idea,” that the conflict/cooperative elements of many human disagreements is structurally inseparable, is central to a secret third thing distinct from Conflict vs Mistake. If you grok the “obvious idea,” we can derive something like
Negotiation Theory(?): I have my desires. You have yours. I sometimes want to cooperate with you, and sometimes not. I take actions maximally good for my goals and respect you well enough to assume that you will do the same; however in practice a “hot war” is unlikely to be in either of our best interests.
In the Negotiation Theory framing, disagreement/conflict arises from dividing the goods in non-zerosum games. I think the economists/game theorists’ “standard models” of negotiation theory is natively closer to “conflict theory” than “mistake theory.” (eg, often their models assume rationality, which means the “can’t agree to disagree” theorems apply). So disagreements are due to different interests, rather than different knowledge. But unlike Marxist/naive conflict theory, we see that conflicts are far from desired or inevitable, and usually there are better trade deals from both parties’ lights than not coordinating, or war.
(Failures from Negotiation Theory’s perspectives often centrally look like coordination failures, though the theory is broader than that and includes not being able to make peace with adversaries)/
Another framing that is in some ways a synthesis and in some ways a different view altogether that can sit in each of the previous theories is a thing that many LW people talk about, but not exactly in this context:
Motivated Cognition: People disagree because their interests shape their beliefs. Political disagreements happen because one or both parties are mistaken about the facts, and those mistakes are downstream of material or ideological interests shading one’s biases. Upton Sinclair: “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Note the word “difficult,” not impossible. This is Sinclair’s view and I think it’s correct. Getting people to believe (true) things against their material interests to believe is possible but the skill level required is higher than a neutral presentation of the facts to a neutral third party.
Interestingly, the Motivated Cognition framing suggests that there might not be a complete truth of the matter for whether “Mistake Theory” vs “Conflict Theory” vs Negotiation Theory is more correct for a given political disagreement. Instead, your preferred framing has a viewpoint-dependent and normative element to it.
Suppose your objective is just to get a specific policy passed (no meta-level preferences like altruism), and you believe this policy is in your interests and those of many others, and people who oppose you are factually wrong.
Someone who’s suited to explanations like Scott (or like me?), might naturally fall into a Mistake Theory framing, and write clear-headed blogposts about why people who disagree with you are wrong. If the Motivated Cognition theory is correct, most people are at least somewhat sincere, and at some level of sufficiently high level of simplicity, people can update to agree with you even if it’s not immediately in their interests (smart people in democracies usually don’t believe 2+2=5 even in situations where it’d be advantageous for them to do so)
Someone who’s good at negotiations and cooperative politics might more naturally adopt a Negotiation Theory framing, and come up with a deal that gets everybody (or enough people) what they want while having their preferred policy passed.
Finally, someone who’s good at (or temperamentally suited to) non-cooperative politics and the more Machiavellian side of politics might identify the people who are most likely to oppose their preferred policies, and destroy their political influence enough that the preferred policy gets passed.
Anyway, here are my four models of political disagreement (Mistake, Conflict, Negotiation, Motivated Cognition). I definitely don’t think these four models (or linear combinations of them) explain all disagreements, or are the only good frames for thinking of disagreement. Excited to hear alternatives [1]!
[1] In particular I’m wondering if there is a distinct case for ideological/memetic theories that operate along a similar level of abstraction as the existing theories, as opposed to thinking of ideologies as primarily given us different goals (which would make them slot in well with all the existing theories except maybe Mistake Theory).