Numerous people on rationalityTwitter called it way before Feb 20th, and some of those bought put options and made big profits. This must be some interesting new take on “rational expectations”. https://twitter.com/ESYudkowsky/status/1229529150098046976?s=20&t=IGOl9Mzj1FYtcPYd1F52AQ
EliezerYudkowsky
Karma: 3,632
Alexander and Yudkowsky on AGI goals
Not only have I never heard this before, I was there and remember watching this not happen. Source?
Those look like nominal rates, not real rates.
Not until timelines are even blatantly shorter than present and long-term loans are on offer, and not unless there’s something useful to actually do with the money.
Corrected, thanks.
Trying again:
OP seems to ambiguate between two ideas, one true idea, and one false idea.
The true idea is that if Omega tells you personally that the world will end in 2030 with probability 1, you personally should not bother saving for retirement. Call this the Personal Idea.
The false idea is that if you believe in foomdoom, you should go long real interest rates and expect a market profit. Call this the Market Idea.
Intuitively, at least if you’re swayed by this essay, the idea in Market probably seems pretty close to the idea in Personal. If everybody started consuming for today and investing less, real interest rates would go up, right? So if you don’t believe that Market is about as strong as Personal, what invalid reasoning step occurs within the gap between the true premise in Personal to the false conclusion in Market?
Is it invalid that if in 2025 everyone started believing that the world would end in 2030 with probability 1, real interest rates would rise in 2025? Honestly, I’m not even sure of that in real life. People are arguing clever-ideas like ‘Shouldn’t everyone take out big loans due later?’ but maybe the lender doesn’t want to lend anymore, if everyone knows that. There’s a supply collapse and a demand collapse and yes I see the theoretical argument but real-world monetary stuff is in fact really strange and complicated; I didn’t see anybody calling the actual interest-rate trajectory surrounding Covid in advance of it actually playing out.
In real life, what zaps you when you think there’s a worldwide pandemic coming and try to trade interest rates, isn’t that you didn’t know about the pandemic ahead of the oblivious market, it’s that you guessed wrong about what the market would really actually do in real life as the pandemic played out and finally ended.
You can sometimes make a profit off an oblivious market, if you guess narrowly enough at reactions that are much more strongly determined. Wei Dai reports making huge profits on the Covid trade against the oblivious market there, after correctly guessing that people soon noticing Covid would at least increase expected volatility and the price of downside put options would go up.
But I don’t think anybody called “there will be a huge market drop followed by an even steeper market ascent, after the Fed’s delayed reaction to a huge drop in TIPS-predicted inflation, and people staying home and saving and trading stocks, followed by skyrocketing inflation later.”
I don’t think the style of reasoning used in the OP is the kind of thing that works reliably. It’s doing the equivalent of trying to predict something harder than ‘once the pandemic really hits, people in the short term will notice and expect more volatility and option prices will go up’; it’s trying to do the equivalent of predicting the market reaction to the Fed reaction a couple of years later.
The entire OP is written as if we lived in an alternate universe where it is way way easier than history has actually shown, to figure out what happens in broad markets after any sort of complicated or nontrivial event occurs in real life that isn’t on the order of “unexpected Fed rate hike” or “company reports much higher-than-expected profits”. And it’s written in such a way as to mislead EAs reading it about the general confidence that the field of economics is able to justly put in predictions about broad market reactions to strange things.
If you haven’t already looked at OP’s recommended investment instrument of (short) LPTZ, which holds inflation-protected Treasuries and is therefore their recommended way of tracking real interest rates, I recommend the following exercise: First try to figure out what you would have believed a priori, without benefit of hindsight, real interest rates would do over the course of a pandemic. Then, decide what investment strategy you’d have followed with LTPZ if you thought you knew about a pandemic ahead of the market. Then, decide what you think happened to real interest rates with benefit of hindsight. Then, go look at the actual price trajectory of their recommended instrument of LTPZ.
I am not sure I can properly convey this thought that I am trying to convey; I have had trouble actually conveying this thought to EAs before. The thought is that people often do long careful serious-sounding writeups which EAs then take Very Seriously, because they are so long and so seriously argued, but in fact fail to bind to reality entirely, in a way that doesn’t have to do with the details of the complicated arguments. Very serious arguments about what ought to happen to the price of an ETF that tracks 15-year TIPS, via the intermediate step of arguing about what logically ought to happen to real interest rates, are the sort of thing that, historically, average economists have not really been able to pull off; it’s a kind of thought that you should expect fails to achieve basic binding to reality. What would LTPZ or its post-facto equivalent have been doing around the time of the Cuban Missile Crisis? My model says ‘no prediction’; they’ll have done whatever. Afterwards somebody will make up a story about it in hindsight, but it is not the sort of thing where history says that long complicated analyses are remotely reliably good at doing it in advance.
But there are even weaker links in the argument, so let’s accept the LTPZ step arguendo and pass on.
An even bigger problem is that, since everybody is going to die before anything really pays out, the marginal foresightful trader does not have a strong incentive to early-on move the market toward where the market would end up in equilibrium after everyone agreed on the actual facts of the matter and had time to trade about them.
Prediction markets, I sometimes explain to people, are tools for transmitting future observables, or more generally propositions that people expect to publicly agree on at some future point even if they don’t agree now, lossily backward in time, manifesting as well-calibrated probability distributions.
To run a prediction market, you first and foremost need a future publicly observable measurement, which is a special case of a place where we expect most people to agree on an extreme probability assignment later, even though they don’t agree now or don’t make extreme probability assignments now. You cannot run a prediction market on whether supplementing lots of potassium can produce weight loss; you can only run a prediction market about what an experiment will report in the way of results, or what a particular judge will say the experimental evidence seems to have indicated in five years. You cannot directly observe “whether potassium causes weight loss” as an underlying fact of biology, so you can’t have a prediction market about that; you can only observe what somebody reports as an experimental result, or what a particular person appointed as judge says out loud about the state of evidence later.
The marginal foresightful trader usually has a motive to run ahead of the market and make trades now, based on where the equilibrium ought to settle later; not because they are nobly undertaking a grand altruistic project of transmitting facts backward in time and making the market behave nicely from the standpoint of theoretical economics, but because they expect to get paid after everyone makes the common observation and the market settles into a new equilibrium reflecting that state of knowledge. And then they expect to have that money, or to get a bonus for earning that money for their proprietary trading firm, and for that money to still hold its value, and for them to be able to spend the money on nice things.
In the unusual case of foomdoom, even if doom proceeds slowly enough that a large-enough group of marginal foresightful traders see the foomdoom coming, even if there is somehow a really definitive fire alarm that says “extremely high probability you are dead within two to four years”, it is incredibly unlikely that everyone in the world will agree that yes we’ll all be dead in two to four years, and that the markets will settle into the equilibrium that an economist would say corresponds to the belief that we’ll all be dead in two to four years; which is what’s required for the foresightful proprietary trader to score a huge profit and get a big bonus that year and have time to spend it on some fancy way of passing the remaining time.
People do not usually agree on what will happen in two to four years. This kind of agreement that reliably reflects a fact, and makes a market pay out in a way that you can trust to correspond to that fact, is usually achieved after that fact is publicly definitively observed, not two to four years ahead of the observation.
In case of foomdoom the world never settles into equilibrium later, the bets never pay out, there is never that moment where everybody says “What a foresightful trader that was!” and agree on the fact that yes we sure are all dead now. So even if a proprietary trader sees doom coming, they do not have much of an incentive to dutifully transmit that information backward in time in order to make the market behave now in the way that an economist thinks ought to correspond to the equilibrium it would settle into after everybody agreed that they were dead.
That incentive would only exist if you expected everybody to agree that they were going to die in a few years, far enough ahead of everybody actually being dead, for the markets to settle into equilibrium and foresightful traders to collect bonuses on having made the trade before that. Which is a much stronger and stranger thing to claim, about a planet like Earth, than the usual much weaker claim that a few sharp traders might see a fact coming, and move the markets a few years ahead of time to where they would go after everybody agreed on that fact later.
Though even then, of course, we have cases like the financial crisis of 2006-2008, where some traders did see it coming and turn huge profits, but couldn’t move enough marginal money around to actually shift the entire broader market.
To suppose that the market is broken around foomdoom is really not a remotely surprising market behavior to suppose! Even in a world where nearly all the prices are efficient relative to you and that’s why you can’t make 10%/day trading Microsoft stock!
What happened in 2006-2008 was much more broken than that! Marginal traders saw it coming, and some of them won huge even though CDSs were not trivial to short; but they didn’t move enough money to shift anything remotely as large as ‘real interest rates’ ahead of the actual materialization of the disaster.
The market’s behavior around Covid also showed much more obliviousness than this; it showed the kind of obliviousness where people I’d previously marked as the strongest EMH challengers reported collecting vast profits over a timespan of a couple of months. (But on chains of logic much less fraught than OP’s, because in real life you can’t call LTPZ movements or real interest rate changes in advance, just things on the order of ‘buy volatility’.)
We should not believe ‘the market’, in the sense of that unusually intelligent entity whose opinions we actually pay attention to, driven by the highly incentivized marginal trader, has any opinion on AGI except that “in the next few years, not everyone will have started believing that they are going to die in a few years after that”. The market is showing no actual opinion on foomdoom, only on what most market participants will believe about foomdoom in a couple of years. The usual incentive mechanism whereby, if a pandemic starts, in a few years most market participants will agree that this pandemic happened and foresightful traders will collect bonuses and spend them—as is responsible for the market sometimes but not always being foresightful, because it is paid to be foresightful—is in this case broken. We are really always seeing, when the market foretells an observable’s value a few years later, that the foresightful marginal trader thinks that a few years later lots of people will hold a certain opinion; it’s just that usually, this common opinion is being mundanely driven by a direct observation.
The market says, “It won’t be the case that in 2030, everyone agrees that they were killed by AGIs.” The market isn’t saying anything about whether that’s because everyone agrees they are alive, or because nobody is left to agree anything.
OP reads like somebody has heard that markets sometimes anticipate things ahead of them happening, that markets sometimes transmit information lossily backward in time, and doesn’t quite seem to have understood the mechanism behind it; that what everyone will agree on later, unusually foresightful marginal traders can sometimes cause the market to reflect now even though not everyone agrees on it yet. Instead they are talking about “What if the market believed...” as if this kind of market belief reflected a numerical majority of the people in the market believing that foomdoom would kill us all before 30-year bonds paid out. But this is not where the market gets its power to say things that we ought to pay special attention to (though even then market isn’t always right about those things, or even righter than us, especially if those things are a little strange, eg Covid etc). The market gets its power from unusually foresightful marginal traders expecting to get a payoff from what everyone believes later after the thing actually happened and therefore most market participants agree about what happened. And this transmission mechanism is broken w/r/t AGI doom in a way that it wouldn’t even be broken for an asteroid strike; with an asteroid strike, you might get weird effects from money losing its value in the future, but at least people could all agree on the asteroid strike coming. With AGI, I think you’d have to be pretty naive to expect everybody to agree that AGI will kill us in two years, two years before AGI kills us. So it is inappropriate to skip over a step we can usually skip over, and compress the true proposition “The market doesn’t believe that everyone in 2030 will believe that in 2030 everyone is dead”, to the false proposition “The market doesn’t believe that in 2030 everyone will be dead.”
Though again—also just to be clear—AGI ruin is a harder call than Covid and I wouldn’t strongly expect the market to get it right, even if transmission weren’t broken; and even if I thought the market would get it right after seeing GPT-4 and that transmission wasn’t broken, I wouldn’t buy “short LTPZ” as a surefire way to profit.
- On AI and Interest Rates by (LessWrong; Jan 17, 2023, 3:00 PM; 79 points)
- Aiming for Convergence Is Like Discouraging Betting by (LessWrong; Feb 1, 2023, 12:03 AM; 62 points)
- 's comment on AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years by (Jan 13, 2023, 4:45 PM; 46 points)
I wouldn’t say that I have “a lot of” skepticism about the applicability of the EMH in this case; you only need realism to believe that the bar is above USDT and Covid, for a case where nobody ever says ‘oops’ and the market never pays out.
Suppose you are one of the 0.1% of macro bonds traders familiar with Yudkowskian foom. You reason as follows: “Suppose that in the next 2 years, we get even more alarming news out of GPT-4 and successors. Suppose it’s so incredibly alarming that 10% of macro traders notice, and then 10% of those hear about Yudkowskian foom scenarios. Putting myself into the shoes of one of those normie macro traders, I think I reason… that most actual normal people won’t change their saving behavior any time soon, even if theoretically they should decrease their saving, and that’s not likely to have macro effects. Still as a normie trader who’s heard about Yudkowsky foomdoom, I think I reason that if Yudkowsky’s right, we’re all dead, and if Yudkowsky’s not right, I’ll get embarrassed about a wrong trade and fired. So this normie trader won’t trade on Yudkowsky foomdoom premises. Therefore I don’t think I can profit over the next two years by shorting a TIPS fund… even leaving aside concerning feelings about whether going hugely short LTPZ would have model risk about LTPZ’s actual relation to real interest rates in these scenarios, or whether other traders would expect big AI impacts to hit measured inflation because of AI-driven lower prices or AI-driven unemployment.”
And then one week before the end of the world, the 1% of most clueful macro bonds traders… will take vacation days early, and draw down their rainy day funds to spend time with their family. They still won’t make macro trades about that, because the payoff matrix looks like “If you’re right, you’re dead and not paid, and if you’re wrong, you’re embarrassed and get fired.” Then haha whoops it turns out that the world didn’t end in a week after all, and people go back to work with a nervous laugh and a sick feeling in their stomachs, and everybody actually falls over dead three weeks later.
If Omega tells you today that everyone will be dead in 2030 with probability 1, there’s no direct way to make a market profit on that private information over the next 2 years, except insofar as foresightful traders today expect less foresightful future traders to hear about AI and read an economics textbook and decide that interest rates theoretically ought to rise and go short TIPS index funds. Foresightful traders today don’t expect this.
To put it another way: Yes, savvy market traders don’t believe that in 2025 everybody will realize that the world is ending. The savvy market traders are correct! Even at the point where the world is ending, everybody will not believe this, and so at no point will the savvy trader have made a profit! The death of all of humanity induces a market anomaly wherein savvy traders don’t expect to be able to profit from everyone else’s error because no event occurs where the real thing actually happens and everybody says “Oops” and the savvy trader gets paid off.
There just isn’t any mystery here. You can’t make a short-term profit off correcting these market prices even if Omega whispers the truth in your ear with certainty. That’s it, that’s the mystery explained, you’re done.
- On AI and Interest Rates by (LessWrong; Jan 17, 2023, 3:00 PM; 79 points)
- 's comment on AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years by (Jan 10, 2023, 6:34 PM; 18 points)
- 's comment on AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years by (LessWrong; Jan 12, 2023, 5:25 PM; 1 point)
If you haven’t extensively, successfully dealt with the media, someplace where the media do not start out nicely inclined towards you (i.e., your past media experience at the Center for Rare Diseases in Cute Puppies does not count), you are not qualified to give this advice. It should be given by somebody who understands how bad journalism gets and what needs to be done to avoid the usual and average negative outcome, or not at all.
I think the sort of people who look at this advice and find that it sounded plausible to them, might want to first follow the rule of only taking advice that originated in actual lawyers, because they couldn’t tell which nonlawyers had done real legal research. IDK, I don’t know what it’s like from the inside to read the original post and not scream.
Important notice to readers. Please vote up even though it is not very carefully argued here, because it may be important to some readers to read it immediately.
DO NOT FOLLOW THIS POST’S ADVICE. IT IS PROBABLY VERY BAD ADVICE FROM A LEGAL STANDPOINT. IF IT DOESN’T GET YOU IN TROUBLE IT WILL ONLY BE BECAUSE PEOPLE IGNORED YOUR LETTERS.
NEVER FOLLOW ADVICE LIKE THIS FROM PEOPLE WHO ARE NOT LAWYERS.
ONLY DO ANYTHING REMOTELY LIKE THIS IF YOU READ A POST FROM OPEN PHILANTHROPY’S LEGAL COUNSEL TELLING YOU TO DO IT.
I see no mention in either of your forum posts of the aforesaid lawyer?
I’d agree with this statement more if it acknowledged the extent to which most human minds have the kind of propositional separation between “morality” and “optics” that obtained financially between FTX and Alameda.
Yeah, I think it’s a severe problem that if you are good at decision theory you can in fact validly grab big old chunks of deontology directly out of consequentialism including lots of the cautionary parts, or to put it perhaps a bit more sharply, a coherent superintelligence with a nice utility function does not in fact need deontology; and if you tell that to a certain kind of person they will in fact decide that they’d be cooler if they were superintelligences so they must be really skillful at deriving deontology from decision theory and therefore they can discard the deontology and just do what the decision theory does. I’m not sure how to handle this; I think that the concept of “cognitohazard” gets vastly overplayed around here, but there’s still true facts that cause a certain kind of person to predictably get their brain stuck on them, and this could plausibly be one of them. It’s also too important of a fact (eg to alignment) for “keep it completely secret” to be a plausible option either.
This strikes me as a bad play of “if there was even a chance”. Is there any cognitive procedure on Earth that passes the standard of “Nobody ever might have been using this cognitive procedure at the time they made $mistake?” That more than three human beings have ever used? I think when we’re casting this kind of shade we ought to be pretty darned sure, preferably in the form of prior documentation that we think was honest, about what thought process was going on at the time.
Maybe they weren’t familiar with the overwhelming volume of previous historical incidents, hadn’t had their brains process history or the news as real events rather than mythology, or were genuinely unsure about how often these sorts of things happened in real life rather than becoming available on the news. I’m guessing #2.
The point is not “EA did as little to shape Alameda as Novik did to shape Alameda” but “here is an example of the mental motion of trying to grab too much responsibility for yourself”.
Fair point.
Trying to “steelman” the work of an experienced adversary who relies on, and is exploiting, your tendency to undercompensate and not realize how distorted these things actually are—which is the practical, hard-earned knowledge that Aella is trying to propagate—seems like a mistake.
(Actually, trying to “steelman” is a mistake in general and you should focus on passing Ideological Turing Tests instead, but that’s a much longer conversation.)