Building bespoke quantitative models to support decisionmakers in AI and bio. Right now that means: forecasting capabilities gains due to post-training enhancements on top of frontier foundation models, and estimating the annual burden of airborne disease in the US.
Joel Becker
Karma: 1,717
Thank you for feedback, Nuno!
I totally agree that part 2 is an unhelpful distraction at this stage. (Agreed with your other points at time of writing!)
I want to leave open the possibility that fellowships are not an effective thing to do regardless of their length, so maybe the minimum time is 0.
But, conditional on thinking otherwise/going ahead with it...
I think it might be helpful to think about diminishing returns for participant stays, vs. fellowship length. You can imagine a 10 year-long program where people only stay for 4 months max at a time. This isn’t as silly as it sounds—very few people stay from the very beginning to very end of these things anyway, so program beginning/end don’t seem like very important milestones to respect (vs. participant stays).
I think participant stays hit diminishing returns at 3 months. (Much more weakly held/pulled from nowhere than claims in this post.)
I worked ~24/7 as an organizer, which became challenging after ~5 months.
Thank you for such kindness Austin—I’m glad it was helpful! :)
Some intuitions about fellowship programs
An update, yeh, but how important?
I think Most Important Century still goes through if you replace extinction/TAI with “bigdealness”. In fact, bigdealness takes up considerably more space for me.
To the degree that non-extinction/TAI-bigdealness decreases the magnitude of implications for financial markets in particular, it is more consistent with the current state of financial markets.
Ah, thank you for mentioning; corrected in original comment.
I am surprised that critical commenters have focused on the irrationality or inadequacy of financial markets, rather than what feels like the more obvious point:
Unaligned AI need not imply extinction, and aligned AI need not imply 30% growth. Financial markets can be inconsistent with these implications without being inconsistent with Big Deal AI.
On unaligned AI: eyeball the reviews of the Carlsmith report. Looks like average P(xrisk | misalignment) ~= 45% among reviewers.
On aligned AI: 30% growth is crazy high! The authors are unwilling to make their claims for less-crazy growth figures:
I don’t think that you were being unclear above. The underlying reasoning still feels a little tortured to me.
The window between when “enough” traders realize that AI is near and when it arrives may be very short, meaning that even in the best case you’ll only increase your wealth for a very short time by making this bet
I mean, sure, it could be, but wouldn’t it be weird to believe this confidently? The artists are storming parliament, the accountants are on the dole, foom just around the corner—but a small number of traders have not yet clocked that an important change is coming?
It is not clear how markets would respond if most traders started thinking that AI was near. They may focus on other opportunities that they believe are stronger than to go short interest rates (e.g., they may decide to invest in tech companies), or they may decide to take some vacation
Traders are not dumb. At least, the small number of traders necessary to move the market are not dumb. They will understand the logic of this post. A mass ignoring of interest rates in favor of tech equity investing is not a stable equilibrium.
In order to get the benefits of the best case above, you need to take on massive interest rate risk, so the downside is potentially much larger than the upside (plus, in the downside case, you’re poor for a much longer time)
In order to get the benefits of the best case of anything, you need to take on risk. You could make the same directional bet with less risk. If you weaken this statement to “exposure to a good chunk of the benefits of the implications of their beliefs, by taking on reasonable risk” then the interest rate conclusion still goes through.
That second idea seems like the opposite of consumption smoothing?? Maybe it’s worthwhile because I would become rich enough that the extra volatility is worth it to me? But what’s the point of being rich for just a short time before I die, or of being rich for just a short time before TAI-induced utopia makes everyone fantastically rich anyways?
The way to resolve the apparent contradiction is to return to the logic of consumption smoothing.
Let’s say you believe that TAI is coming. You expect that you and everyone you know will be dead or fabulously rich in the medium-term. You think that this has implications for markets. You like consumption streams to be smooth.
You (hopefully) start today with savings.
The logic of consumption smoothing says that you want to save less/dissave more relative to what would have been the case if you did not believe TAI is coming. In the first instance this means spending down your assets. But no borrowing! There’s no point in (expensive) borrowing if you can still spend down your assets. It also means that you still have assets! Where should you put those assets? Well, your beliefs imply that real interest rates will be higher than the market expects, so you don’t want to hold inflation-protected treasuries.
The point about risk is neither here nor there. If this is a concern you can decrease the degree of underweighting (or the size of your short position). The point is that at the margin you disprefer inflation-protected treasuries to what would have been the case if you did not have your TAI beliefs.
At some future point, you have spent down your savings.
But, no worries, TAI is coming! You commit to take out loans. You don’t have any more savings, so there is no point in investing, so there is no point bothering to underweight inflation-protected treasuries. (Unless leverage, but logic would be similar.)
It might be more convincing to directly attack their point that the price of MST, TSM, SMSN, ASML, etc. is a function of not only future profits but future interest rates.
Their claim is that the effect on equity prices is messy because of interest rates, not that future expected profits are necessarily lower than you believe.
The authors address your first comment in this appendix.
That does not get the thought experiment through.
Mortgage rates for older people are higher. And if mortgage holders die, the mortgage must still be paid by the executor of an estate, which is a disincentive for anyone with a bequest motive.
I’m sure that we can find some corner case where young cancer victims with no friends/family or no regard for their friends/family act otherwise. But this hardly seems important for the point that you—yes, you—can make money by implementing the trades suggested in this piece. Which is the claim that Yudkowsky is using the cancer victim analogy to argue against.
Ignore the short position: you could just underweight these assets relative to global market portfolio.
Huh? Terminal cancer victims taking out 30-year mortgages is extremely different, in terms of the counter-party’s willingness to trade.
This is one of my favourite forum posts ever. A pleasure to read. Congrats to the three of you.
You can argue that one could take a short position on interest rates (e.g., in the form of a loan) if you believe that they will rise at some point, but that is a different bet from short timelines—what you’re betting on then, is when the world will realize that timelines are short, since that’s what it will take before many people choose to pull out of the market, and thus drive interest rates up. It is entirely possible to believe both that timelines are short, and that the world won’t realize AI is near for a while yet, in which case you wouldn’t do this.
This reasoning sounds pretty tortured to me.
First, should you really believe that the relatively small number of traders needed to move markets won’t come to think AI is a really big deal, given that you think AI is a really big deal?
Second, if “the world won’t realize AI is near for a while,” you can still make money by following analogous strategies to those described in the post. You don’t need the world to realize tomorrow.
Something that might be helpful, from 80,000 hours:
Around $50 million was spent on reducing catastrophic risks from AI in 2020 — while billions were spent advancing AI capabilities. While we are seeing increasing concern from AI experts, we estimate there are still only around 400 people working directly on reducing the chances of an AI-related existential catastrophe (with a 90% confidence interval ranging between 200 and 1,000). Of these, it seems like about three quarters are working on technical AI safety research, with the rest split between strategy (and other governance) research and advocacy.
Thank you for this post. The framing of your points as conditional is especially helpful.
I strongly agree with lots here. As someone who has worked on community building-ish projects that are very far from or very close to frontline/object-level work, this part rang especially true:
Insofar as it makes sense to have less of a default presumption towards the value of community building, a way of de-risking community building activities is to link them more closely to activities where the case for direct impact is stronger.
People interested in the claim might be interested in this related post and discussion.
My thought was: base rates for FTX-style disasters?
(Robi mentioned this to me in person; I thought it was insightful/asked him to comment.) Thank you for the insight Robi! This is an interesting way of thinking about my numbers claim that I had not considered.