I agree that this illustrates a counterpoint to longtermism-style arguments that is underappreciated.
As someone who believes that there are valid reasons to be concerned about the effects advanced AI systems will have and therefore that general “AI risk” ideas contain important insights and are worthy of consideration, I will offer my perspective on why this post aptly demonstrates an important point.
I think there is something of a pattern in discussions around AI risk that conflate formal, high reliability methods with less formal conceptual arguments that have some similarity to the more formal methods. This causes AI risk advocates to have an inaccurate impression of how compelling these arguments will be to people who are more skeptical. I think AI risk advocates sometimes implicitly carry over some of the high reliability and confidence of formal methods to the less formal conceptual arguments, and as a result can end up surprised and/or frustrated when skeptics don’t find these arguments to be as persuasive or warrant as a high a level of confidence as AI risk advocates sometimes have.
This post effectively demonstrates this dynamic in two areas where I have also noted this myself in the past: track prediction track records and high impact/low probability reasoning.
And since 2017 when I wrote that, I’ve kind of informally just been testing — I wish I had done it more formally now — but informally just seeing which one of these two perspectives [inside vs outside view] are making the better predictions about the world. And I do just think that that inside view perspective, in particular from a certain number of people within this kind of community, just has consistently had the right answer.
Formally tracking predictions or returns from bets that are made by members of a specific community and showing they are often correct/realize high returns would indeed be a compelling reason to give those views serious weight.
However, it is much more difficult to know how much to credit this kind of reasoning when the testing is informal. As an example, you can have a cherry-picking or selective memory issue. If AI risk inside view advocates often remember or credit flashy cases where members of a community made a good prediction or bet, but don’t similarly recall inaccurate predictions, then this informal testing may not be as compelling, and skeptics are likely to be justifiably more suspicious of this possibility compared to people who already find arguments for AI risk convincing.
As an example of the high impact/low probability reasoning case, consider this post by Richard Chappell:
Even just a 1% chance of extremely high stakes is sufficient to establish high stakes in expectation. So we should not feel assured of low stakes even if a highly credible model—warranting 99% credence—entails low stakes. It hardly matters at all how many credible models entail low stakes. What matters is whether any credible model entails extremely high stakes. If one does—while warranting just 1% credence—then we have established high stakes in expectation, no matter what the remaining 99% of credibility-weighted models imply (unless one inverts the high stakes in a way that cancels out the other high-stakes possibility).
David Thorstad provides some counterargument in this post. Commenting on Thorstad’s article on the EA forum, Chappell says this:
Saying that my “primary argumentative move is to assign nontrivial probabilities without substantial new evidence” is poor reading comprehension on Thorstad’s part. Actually, my primary argumentative move was explaining how expected value works. The numbers are illustrative, and suffice for anyone who happens to share my priors (or something close enough). Obviously, I’m not in that post trying to persuade someone who instead thinks the correct probability to assign is negligible. Thorstad is just radically misreading what my post is arguing.
My reading of this exchange is that it demonstrates the formal/informal conflation that I claim exists in these types of discussions. To my mind, the “explaining how expected value works” part suggests an implicit believe that the underlying argument carries the strength and confidence approaching that of a mathematical proof. Although the argument itself is conceptual, it experiences some amount of spillover of reliability/confidence because the concepts involved are mathematical/formal, even though the argument itself is not.
I think this dynamic can cause AI risk advocates to overestimate how convincing skeptics will (or perhaps should) find these arguments. It seems to me like this often leads to acrimony and frustration on both sides. My preferred approach to arguing for AI risk would acknowledge some of the ambiguity/uncertainty and also focus on a different set of concepts than those that often have the focus in discussions about AI risk.
Breaking character, if you want to see just how strongly people are biased toward misjudging their own track record, it’s instructive to have friends and family who do retail stock picking (essentially gambling). I have strongly advocated passive ETF investing based on the wealth of research and analysis that supports it. Even when they’re well-informed about the case for passive ETF investing, people still think they can beat the market. And the thing about going up against the market is, your performance is absolutely quantifiable! In a way that vibes-y predictions about AI aren’t. Yet, even being consummately quantifiable, people who pick stocks don’t benchmark their performance, or do it selectively when their stocks are up (obviously giving a biased impression), or somehow justify or rationalize or explain away why they’re actually winning.
I don’t trust for a second that someone judging their own performance on informal, selectively remembered, largely subjective AI predictions is doing a fair job. Even when people like Dario Amodei or Ray Kurzweil have publicly made specific AI or technology predictions that turned out to be unambiguously dead wrong, they have subsequently twisted and contorted the truth in order to make themselves right — lied, essentially, or else fooled themselves. I do not trust people to grade their own homework and to have the result be scientific-quality evidence.
The high-impact/low-probability reasoning, on the other hand, is completely sound and demonstrates why deep-pocketed donors should give me $1 million/year.
I agree that this illustrates a counterpoint to longtermism-style arguments that is underappreciated.
As someone who believes that there are valid reasons to be concerned about the effects advanced AI systems will have and therefore that general “AI risk” ideas contain important insights and are worthy of consideration, I will offer my perspective on why this post aptly demonstrates an important point.
I think there is something of a pattern in discussions around AI risk that conflate formal, high reliability methods with less formal conceptual arguments that have some similarity to the more formal methods. This causes AI risk advocates to have an inaccurate impression of how compelling these arguments will be to people who are more skeptical. I think AI risk advocates sometimes implicitly carry over some of the high reliability and confidence of formal methods to the less formal conceptual arguments, and as a result can end up surprised and/or frustrated when skeptics don’t find these arguments to be as persuasive or warrant as a high a level of confidence as AI risk advocates sometimes have.
This post effectively demonstrates this dynamic in two areas where I have also noted this myself in the past: track prediction track records and high impact/low probability reasoning.
As an example of the prediction track record case, consider this from an interview with Will MacAskill on 80,000 hours:
Formally tracking predictions or returns from bets that are made by members of a specific community and showing they are often correct/realize high returns would indeed be a compelling reason to give those views serious weight.
However, it is much more difficult to know how much to credit this kind of reasoning when the testing is informal. As an example, you can have a cherry-picking or selective memory issue. If AI risk inside view advocates often remember or credit flashy cases where members of a community made a good prediction or bet, but don’t similarly recall inaccurate predictions, then this informal testing may not be as compelling, and skeptics are likely to be justifiably more suspicious of this possibility compared to people who already find arguments for AI risk convincing.
As an example of the high impact/low probability reasoning case, consider this post by Richard Chappell:
David Thorstad provides some counterargument in this post. Commenting on Thorstad’s article on the EA forum, Chappell says this:
My reading of this exchange is that it demonstrates the formal/informal conflation that I claim exists in these types of discussions. To my mind, the “explaining how expected value works” part suggests an implicit believe that the underlying argument carries the strength and confidence approaching that of a mathematical proof. Although the argument itself is conceptual, it experiences some amount of spillover of reliability/confidence because the concepts involved are mathematical/formal, even though the argument itself is not.
I think this dynamic can cause AI risk advocates to overestimate how convincing skeptics will (or perhaps should) find these arguments. It seems to me like this often leads to acrimony and frustration on both sides. My preferred approach to arguing for AI risk would acknowledge some of the ambiguity/uncertainty and also focus on a different set of concepts than those that often have the focus in discussions about AI risk.
Thank you so much for this.
Breaking character, if you want to see just how strongly people are biased toward misjudging their own track record, it’s instructive to have friends and family who do retail stock picking (essentially gambling). I have strongly advocated passive ETF investing based on the wealth of research and analysis that supports it. Even when they’re well-informed about the case for passive ETF investing, people still think they can beat the market. And the thing about going up against the market is, your performance is absolutely quantifiable! In a way that vibes-y predictions about AI aren’t. Yet, even being consummately quantifiable, people who pick stocks don’t benchmark their performance, or do it selectively when their stocks are up (obviously giving a biased impression), or somehow justify or rationalize or explain away why they’re actually winning.
I don’t trust for a second that someone judging their own performance on informal, selectively remembered, largely subjective AI predictions is doing a fair job. Even when people like Dario Amodei or Ray Kurzweil have publicly made specific AI or technology predictions that turned out to be unambiguously dead wrong, they have subsequently twisted and contorted the truth in order to make themselves right — lied, essentially, or else fooled themselves. I do not trust people to grade their own homework and to have the result be scientific-quality evidence.
The high-impact/low-probability reasoning, on the other hand, is completely sound and demonstrates why deep-pocketed donors should give me $1 million/year.