I’m curating this post because I think it raises important points that I’d like more people to engage with, and because the discussion on it has been really interesting (124 comments right now). I also think it’d be very good to have more genuine critical engagement with the case for prioritizing work on AI existential risk.
However, the post seems wrong or at least heavily disputable in important ways, so I will also point out relevant considerations and comments. Please note that I’m not an expert; in some cases, I’m merely summarizing my understanding of what others have said.
This comment ended up being very long. Before it goes on, I want to direct attention to some other content arguing against the case that risk from AI is high or should be prioritized:
The overall claim that the post is making is the following (I think):
Markets are a good way of finding an outside view on a topic — in this case, on transformative AI. Long-term real rates would be higher than they currently are if markets believed that transformative AI was coming in the next ~30 years. So you should expect longer timelines for transformative AI. Also, if you really believe that transformative AI is coming soon, you can make money by disagreeing with the market’s current position (by shorting bonds).
One thing that seems worth pointing out before we get into the disagreements: the arguments in the post rely on the idea that, if transformative AI is coming at a certain time, we should expect that more and more people will believe this as that time draws near. This doesn’t seem unreasonable to me, but I didn’t see it outlined as an assumption in the post.
Moving on: some disagreements (a non-exhaustive list):
Interpreting or correcting markets like this is a messy business, especially when trying to predict events that are years out, especially when there’s not a lot to profit from in the near-term or continuously. And the crazier things get, the more unusual things markets might do. As Jan points out,
“the dichotomy 30% growth / everyone dies is unrealistic for trading purposes
near term, there are various outcomes like “industry X gets disrupted” or “someone loses job due to automation” or “war”
if you anticipate fears of this type to dominate in next 10 years, you should price many people increasing their savings and borrowing less”
You can sometimes make a profit off an oblivious market, if you guess narrowly enough at reactions that are much more strongly determined. Wei Dai reports making huge profits on the Covid trade against the oblivious market there, after correctly guessing that people soon noticing Covid would at least increase expected volatility and the price of downside put options would go up.
But I don’t think anybody called “there will be a huge market drop followed by an even steeper market ascent, after the Fed’s delayed reaction to a huge drop in TIPS-predicted inflation, and people staying home and saving and trading stocks, followed by skyrocketing inflation later.”
I don’t think the style of reasoning used in the OP is the kind of thing that works reliably. It’s doing the equivalent of trying to predict something harder than ‘once the pandemic really hits, people in the short term will notice and expect more volatility and option prices will go up’; it’s trying to do the equivalent of predicting the market reaction to the Fed reaction a couple of years later.
And from this comment: Expecting the market to be (decently) efficient here is less reasonable, because
“Suppose you have someone who has better insights than everyone else about some asset. They may not be rich and for various related reasons they are unable to immediately correct the market (i.e., the market is actually temporarily inefficient). However, if they are [right], they either a) can keep profiting over and over again until they become liquid/rich enough to individually correct the market, and/or b) other people see that this person is profiting over and over again so they jump in and contribute to market correction.” The problem is that it might be the case that there is only one or two cycles for profit with AGI until the world goes crazy, but it could take many years for this strategy to actually profit, during which time the market will be “temporarily” inefficient. If real interest rates don’t rise for 15 years, and only start to rise ~5 years before AGI, the market is inefficient for 15 years because small players can’t profit to fix the situation.
Markets are really bad at dealing with extreme events
The classic argument I’ve seen for prediction markets goes like this:[2] prediction markets work because people can expect to be paid for being right. Say someone sets up a market on whether humanity is extinct by 2030. If we’re not extinct, traders who bought “not extinct” shares profit. If we are… no one profits. So you don’t really have an incentive to buy “extinct” shares, even if you know for a fact that humanity will go extinct before 2030. (There are cleverer ways to set up this kind of market, but we’ll skip them for now.) (Related post/section.) (Also explained in this comment, among others.)
This applies here, to a certain extent, and several commenters point this out. The post tries to get around this by focusing on real rates, but there’s only a limited amount of profit possible if the markets are incorrect in this direction; as suggested, you can stop saving past 2030 (or discount according to your timelines) and try to short government debt[3], but this is not enough to profit massively and not enough to fully correct the markets if they are off in this way.
Jakob points this out in a comment which also notes, “You can argue that one could take a short position on interest rates (e.g., in the form of a loan) if you believe that they will rise at some point, but that is a different bet from short timelines—what you’re betting on then, is when the world will realize that timelines are short, since that’s what it will take before many people choose to pull out of the market, and thus drive interest rates up. It is entirely possible to believe both that timelines are short, and that the world won’t realize AI is near for a while yet, in which case you wouldn’t do this.” Or, as explained in other comments, “If [you find out] that everyone will be dead in 2030 with probability 1, there’s no direct way to make a market profit on that private information over the next 2 years, except insofar as foresightful traders today expect less foresightful future traders to hear about AI and read an economics textbook and decide that interest rates theoretically ought to rise and go long TIPS index funds. Foresightful traders today don’t expect this.” (More on this.)
One proposed mechanism for profiting off something like knowing that humanity will go extinct by 2030 is taking a big loan that you don’t expect to pay out. However, lenders would generally ask for a big collateral/margin on a loan with such a long timeline (otherwise lots of people would take loans like this and default). Additionally, if the treasury rate went down in the short-term, the lender might close out the trade at a loss for the borrower, even if the borrower is right on a longer time span. (A commenter points out that there might be student loans that work for this.)
I think the examples listed in the “Empirical evidence on real rates and mortality risk” section are importantly different from transformative AI because (1) the catastrophes aren’t existential (see above, on extreme events) and (2) in those cases there’s info that some people have — insider trading, which we don’t necessarily have here but which is good for correcting markets. So I think you should expect this approach to work better for the examples than for the AI case.
I don’t know what happened in situations that seem slightly more relevant, like the Cold War, and I would be interested in hearing more if someone knows or can look into it. But even those situations seem very different.
A number of people in the comments are discussing whether enough people trading/investing/borrowing etc. are aware enough of arguments about transformative AI to be able to form an opinion on this and correct the market if the market is in fact off in this direction. Rohin writes: “If you already knew that belief in AGI soon was a very contrarian position (including amongst the most wealthy, smart, and influential people), I don’t think you should update at all on the fact that the market doesn’t expect AGI.”
″… What’s going wrong, I think, is something like this. People encounter uncommonly-believed propositions now and then, like “AI safety research is the most valuable use of philanthropic money and talent in the world” or “Sikhism is true”, and decide whether or not to investigate them further. If they decide to hear out a first round of arguments but don’t find them compelling enough, they drop out of the process. (Let’s say that how compelling an argument seems is its “true strength” plus some random, mean-zero error.) If they do find the arguments compelling enough, they consider further investigation worth their time. They then tell the evangelist (or search engine or whatever) why they still object to the claim, and the evangelist (or whatever) brings a second round of arguments in reply. The process repeats.
As should be clear, this process can, after a few iterations, produce a situation in which most of those who have engaged with the arguments for a claim beyond some depth believe in it. But this is just because of the filtering mechanism: the deeper arguments were only ever exposed to people who were already, coincidentally, persuaded by the initial arguments. If people were chosen at random and forced to hear out all the arguments, most would not be persuaded. …”
ChatGPT’s explanation of what “shorting government debt” means:
In finance, “shorting” (or “short selling”) refers to the process of selling an asset that the seller does not own with the expectation that the price of the asset will decrease. The seller will then purchase the asset at the lower price, in order to make a profit. Short selling is typically used as a way to speculate on a market decline, or to hedge against potential losses in an existing long position.
Shorting government debt is done by borrowing government bonds from a lender, such as a broker or another investor, and then selling them on the open market with the expectation that their value will decrease [so you have to pay back less than you earned by selling the bonds on the open market]. The borrower hopes to be able to repurchase the bonds at a lower price in the future, and return them to the lender, while keeping the difference as profit.
Just a quick comment to highlight the responses which we have given to the list of disagreements, and to tweak your summary a bit to better reflect what I (not to speak for my other two co-authors) see our post as saying:
A good outside view is that markets are a good way of finding an outside view on a topic — in this case, on transformative AI. Long-term real rates would be higher than they currently are if markets believed that transformative AI was coming in the next ~30 years. If you believe timelines are short, you should personally be saving less or borrowing more. If you believe timelines are short and the market will realise a meaningful amount of time before transformative AI arrives, you should take a short Treasuries position. If you believe that the market should have already realised it and priced it in right now, you should rethink your timelines.
Edit: As it turns out, there’s a nice third party summary which even more concisely captures the essence of what we are trying to get across!
I appreciate the summary, and I’m especially glad to see it done with an emphasis on relatively hierarchical bullet points, rather than mostly paragraph prose. (And thanks for the reference to my comment ;)
Soliciting counterarguments or other forms of relevant information (e.g., case studies) from a crowd of people who may just want to focus on or make very specific/modular contributions, and
Showing how relevant counterarguments and information relate to each other—including where certain arguments have not been meaningfully addressed within a branch of arguments (e.g., 3 responses down), especially to help audiences who are trying to figure out questions like “has anyone responded to X.”
I’m not even confident that this debate necessarily has that many divisive branches—it seems quite plausible that there are relatively few cruxes/key insights that drive the disagreement—but this question does seem fairly important and has generated a non-trivial amount of attention and disagreement.
Does anyone else share this impression with regards to this post (e.g., “I think that it is worth exploring alternatives to the way we handle disagreements via prose and comment threads”), or do people think that summaries like this comment are in fact sufficient (or that alternatives can’t do better, etc.)?
I’m curating this post because I think it raises important points that I’d like more people to engage with, and because the discussion on it has been really interesting (124 comments right now). I also think it’d be very good to have more genuine critical engagement with the case for prioritizing work on AI existential risk.
However, the post seems wrong or at least heavily disputable in important ways, so I will also point out relevant considerations and comments. Please note that I’m not an expert; in some cases, I’m merely summarizing my understanding of what others have said.
This comment ended up being very long. Before it goes on, I want to direct attention to some other content arguing against the case that risk from AI is high or should be prioritized:
But have they engaged with the arguments? (Phil Trammell)[1]
Counterarguments to the basic AI risk case (Katja Grace)
How sure are we about this AI stuff? (Ben Garfinkel)
This thread by Matthew Barnett (which is responding to arguments against AGI being near)
Wikipedia’s section on skepticism about risk from AGI
The overall claim that the post is making is the following (I think):
Markets are a good way of finding an outside view on a topic — in this case, on transformative AI. Long-term real rates would be higher than they currently are if markets believed that transformative AI was coming in the next ~30 years. So you should expect longer timelines for transformative AI. Also, if you really believe that transformative AI is coming soon, you can make money by disagreeing with the market’s current position (by shorting bonds).
One thing that seems worth pointing out before we get into the disagreements: the arguments in the post rely on the idea that, if transformative AI is coming at a certain time, we should expect that more and more people will believe this as that time draws near. This doesn’t seem unreasonable to me, but I didn’t see it outlined as an assumption in the post.
Moving on: some disagreements (a non-exhaustive list):
Interpreting or correcting markets like this is a messy business, especially when trying to predict events that are years out, especially when there’s not a lot to profit from in the near-term or continuously. And the crazier things get, the more unusual things markets might do. As Jan points out,
“the dichotomy 30% growth / everyone dies is unrealistic for trading purposes
near term, there are various outcomes like “industry X gets disrupted” or “someone loses job due to automation” or “war”
if you anticipate fears of this type to dominate in next 10 years, you should price many people increasing their savings and borrowing less”
And Eliezer writes:
You can sometimes make a profit off an oblivious market, if you guess narrowly enough at reactions that are much more strongly determined. Wei Dai reports making huge profits on the Covid trade against the oblivious market there, after correctly guessing that people soon noticing Covid would at least increase expected volatility and the price of downside put options would go up.
But I don’t think anybody called “there will be a huge market drop followed by an even steeper market ascent, after the Fed’s delayed reaction to a huge drop in TIPS-predicted inflation, and people staying home and saving and trading stocks, followed by skyrocketing inflation later.”
I don’t think the style of reasoning used in the OP is the kind of thing that works reliably. It’s doing the equivalent of trying to predict something harder than ‘once the pandemic really hits, people in the short term will notice and expect more volatility and option prices will go up’; it’s trying to do the equivalent of predicting the market reaction to the Fed reaction a couple of years later.
And from this comment: Expecting the market to be (decently) efficient here is less reasonable, because
“Suppose you have someone who has better insights than everyone else about some asset. They may not be rich and for various related reasons they are unable to immediately correct the market (i.e., the market is actually temporarily inefficient). However, if they are [right], they either a) can keep profiting over and over again until they become liquid/rich enough to individually correct the market, and/or b) other people see that this person is profiting over and over again so they jump in and contribute to market correction.”
The problem is that it might be the case that there is only one or two cycles for profit with AGI until the world goes crazy, but it could take many years for this strategy to actually profit, during which time the market will be “temporarily” inefficient. If real interest rates don’t rise for 15 years, and only start to rise ~5 years before AGI, the market is inefficient for 15 years because small players can’t profit to fix the situation.
Markets are really bad at dealing with extreme events
The classic argument I’ve seen for prediction markets goes like this:[2] prediction markets work because people can expect to be paid for being right. Say someone sets up a market on whether humanity is extinct by 2030. If we’re not extinct, traders who bought “not extinct” shares profit. If we are… no one profits. So you don’t really have an incentive to buy “extinct” shares, even if you know for a fact that humanity will go extinct before 2030. (There are cleverer ways to set up this kind of market, but we’ll skip them for now.) (Related post/section.) (Also explained in this comment, among others.)
This applies here, to a certain extent, and several commenters point this out. The post tries to get around this by focusing on real rates, but there’s only a limited amount of profit possible if the markets are incorrect in this direction; as suggested, you can stop saving past 2030 (or discount according to your timelines) and try to short government debt[3], but this is not enough to profit massively and not enough to fully correct the markets if they are off in this way.
Jakob points this out in a comment which also notes, “You can argue that one could take a short position on interest rates (e.g., in the form of a loan) if you believe that they will rise at some point, but that is a different bet from short timelines—what you’re betting on then, is when the world will realize that timelines are short, since that’s what it will take before many people choose to pull out of the market, and thus drive interest rates up. It is entirely possible to believe both that timelines are short, and that the world won’t realize AI is near for a while yet, in which case you wouldn’t do this.” Or, as explained in other comments, “If [you find out] that everyone will be dead in 2030 with probability 1, there’s no direct way to make a market profit on that private information over the next 2 years, except insofar as foresightful traders today expect less foresightful future traders to hear about AI and read an economics textbook and decide that interest rates theoretically ought to rise and go long TIPS index funds. Foresightful traders today don’t expect this.” (More on this.)
One proposed mechanism for profiting off something like knowing that humanity will go extinct by 2030 is taking a big loan that you don’t expect to pay out. However, lenders would generally ask for a big collateral/margin on a loan with such a long timeline (otherwise lots of people would take loans like this and default). Additionally, if the treasury rate went down in the short-term, the lender might close out the trade at a loss for the borrower, even if the borrower is right on a longer time span. (A commenter points out that there might be student loans that work for this.)
I think the examples listed in the “Empirical evidence on real rates and mortality risk” section are importantly different from transformative AI because (1) the catastrophes aren’t existential (see above, on extreme events) and (2) in those cases there’s info that some people have — insider trading, which we don’t necessarily have here but which is good for correcting markets. So I think you should expect this approach to work better for the examples than for the AI case.
I don’t know what happened in situations that seem slightly more relevant, like the Cold War, and I would be interested in hearing more if someone knows or can look into it. But even those situations seem very different.
A number of people in the comments are discussing whether enough people trading/investing/borrowing etc. are aware enough of arguments about transformative AI to be able to form an opinion on this and correct the market if the market is in fact off in this direction. Rohin writes: “If you already knew that belief in AGI soon was a very contrarian position (including amongst the most wealthy, smart, and influential people), I don’t think you should update at all on the fact that the market doesn’t expect AGI.”
″… What’s going wrong, I think, is something like this. People encounter uncommonly-believed propositions now and then, like “AI safety research is the most valuable use of philanthropic money and talent in the world” or “Sikhism is true”, and decide whether or not to investigate them further. If they decide to hear out a first round of arguments but don’t find them compelling enough, they drop out of the process. (Let’s say that how compelling an argument seems is its “true strength” plus some random, mean-zero error.) If they do find the arguments compelling enough, they consider further investigation worth their time. They then tell the evangelist (or search engine or whatever) why they still object to the claim, and the evangelist (or whatever) brings a second round of arguments in reply. The process repeats.
As should be clear, this process can, after a few iterations, produce a situation in which most of those who have engaged with the arguments for a claim beyond some depth believe in it. But this is just because of the filtering mechanism: the deeper arguments were only ever exposed to people who were already, coincidentally, persuaded by the initial arguments. If people were chosen at random and forced to hear out all the arguments, most would not be persuaded. …”
Related, I think: The motivated reasoning critique of effective altruism, Epistemic learned helplessness
I can’t find a quick link about it that concisely explains just the argument — would appreciate one!
ChatGPT’s explanation of what “shorting government debt” means:
Thanks for curating the post!
Just a quick comment to highlight the responses which we have given to the list of disagreements, and to tweak your summary a bit to better reflect what I (not to speak for my other two co-authors) see our post as saying:
Edit: As it turns out, there’s a nice third party summary which even more concisely captures the essence of what we are trying to get across!
I appreciate the summary, and I’m especially glad to see it done with an emphasis on relatively hierarchical bullet points, rather than mostly paragraph prose. (And thanks for the reference to my comment ;)
Nonetheless, I am tempted to examine this question/debate as a case study for my strong belief that, relative to alternative methods for keeping track of arguments or mapping debates, prose/bullets + comment threads are an inefficient/ineffective method of
Soliciting counterarguments or other forms of relevant information (e.g., case studies) from a crowd of people who may just want to focus on or make very specific/modular contributions, and
Showing how relevant counterarguments and information relate to each other—including where certain arguments have not been meaningfully addressed within a branch of arguments (e.g., 3 responses down), especially to help audiences who are trying to figure out questions like “has anyone responded to X.”
I’m not even confident that this debate necessarily has that many divisive branches—it seems quite plausible that there are relatively few cruxes/key insights that drive the disagreement—but this question does seem fairly important and has generated a non-trivial amount of attention and disagreement.
Does anyone else share this impression with regards to this post (e.g., “I think that it is worth exploring alternatives to the way we handle disagreements via prose and comment threads”), or do people think that summaries like this comment are in fact sufficient (or that alternatives can’t do better, etc.)?