Fun exercise: which side of this bet would you want to be on?
I’m definitely on Holden’s side of the bet.
In summary, I assign 80% to Holden’s outcome, 15% to the ambiguous “push” outcome, and 5% to Zvi’s outcome.
This is a low-information forecast, but there seem to be three outcomes to the bet, and Zvi’s outcome clearly seems to be the least likely:
(1) For Zvi to win, Covid cases (of all variants, including any future ones) need to average ~18 times higher over the first two months of 2022 than over the following 12 months (math: 18 = (0.75/2) / (0.25/12)).
18 is such a high ratio given Covid’s track record so far. The ratio of the 7-day-average of US cases from its high (1/11/2021 = ~256,000/day) to its low (6/21/2021 = ~12,000/day) is ~21, barely higher than ~18. Plus, those two weeks were months apart, giving time for cases to drop off.
I don’t see a very plausible way to get that kind of ratio for the first two months of 2022 over the 12 months afterwards. E.g. It seems unlikely that cases would drop off sufficiently quickly at the end of February to avoid adding a large number of cases in March (and to a lesser degree, April, etc). (i.e. Even if cases virtually disappeared later in 2022 (such that the second period being 12 months instead of 6 doesn’t matter much), it’s really hard for cases to drop off so quickly that the number of cases from March 1 onwards don’t end up being at least a third of the number of cases from January and February.) The prior on that steep drop-off happening by the end of February is quite low and the fact that it’s only a little over a week until January and cases are still on the rise doesn’t make it seem more likely that there will be a steep drop-off before March. There’s just no way Zvi could know that that is likely going to happen. I don’t need to read his post to know that he doesn’t know that.
Given this simple consideration that cases would have to drop off exceptionally fast at just the right time for Zvi’s outcome to happen, I assign a 5% chance to Zvi’s outcome happening.
It was really just strongly disagreeing that Zvi’s outcome seemed likely that made me want to write this comment, but I’ll go ahead and write estimates for the other two outcomes:
(2) For Holden to win, cases in the 12-month period after February 2022 have to exceed
one-third of cases in the first two months of 2022 before a new variant comes along and takes over. I haven’t heard of any new variant after Omicron. Such a new variant would have to be much more contagious than Omicron for it to have a chance to take over everywhere in the US in time to stop Omicron’s cases from exceeding that one-third threshold in March-May. I hear Omicron is super contagious so that seems unlikely. Therefore, my low-information forecast is that Holden’s outcome is 80% likely.
(3) For the bet to be a “push” (i.e. for the bet to resolve ambiguously), a new variant or variants need to take over before Omicron-and-previous-variant cases after March 1 exceed the one-third threshold mentioned above. This seems more likely than Zvi’s outcome, but still not that likely. I’ll assign 15% to the ambiguous outcome.
Given this simple consideration that cases would have to drop off exceptionally fast at just the right time for Zvi’s outcome to happen, I assign a 5% chance to Zvi’s outcome happening.
Your analysis roughly matches my independent impression, but I’m pretty sure this simple consideration didn’t escape Zvi’s attention. So, it seems that you can’t so easily jump from that analysis to the conclusion that Holden will win the bet, unless you didn’t think much of Zvi as a reasoner to begin with or had a plausible error theory to explain this particular instance.
Yes, you’re quite right, thanks. I failed to differentiate between my independent impression and my all-things-considered view when thinking about and writing the above. Thinking about it now, I realize ~5% is basically my independent impression, not my all-things-considered view. My all-things-considered view is more like ~20% Zvi wins—and if you told me yours was 40% then I’d update to ~35%, though I’d guess yours is more like ~25%. I meta-updated upwards based on knowing Zvi’s view and the fact that Holden updated upwards on Zvi to 50%. (And even if I didn’t know their views, my initial naive all-things-considered forecasts would very rarely be as far from 50% as 5% is unless there’s a clear base rate that is that extreme.).
That said, I haven’t read much of what Zvi has written in general and the one thing I do remember reading of his on Covid (his 12/24/20 Covid post) I strongly disagreed with at the time (and it turns out he was indeed overconfident). I recognize that this probably makes me biased against Zvi’s judgment, leading me to want to meta-update on his view less than I probably should (since I hear a lot of people think he has good judgment and there probably are a lot of other predictions he’s made which were good that I’m just not aware of), but at the same I really don’t personally have good evidence of his forecasting track record in the way that I do of e.g. your record, so I’m much less inclined to meta-update a lot on him than I would e.g. on you.
Additionally, I did think of a plausible error theory earlier after writing the 5% forecast (specifically: a plausible story for how Zvi could have accepted such a bet at terrible odds). (I said this out loud to someone at the time rather than type it:) My thought was that Zvi’s view in the conceptual disagreement they were betting on seems much more plausible to me than Zvi’s position in the bet operationalization. That is, there are many scenarios that would make it look like Zvi was basically right that might technically cache out as a Holden win according to the exact betting terms described here. For example, there might be a huge Omicron wave—the largest Covid wave yet—and cases might drop quickly afterwards and it might be the last wave of the pandemic, and yet despite all of that, perhaps only 50% of the cases after January 1, 2022 happen before the end of February rather than the 75% necessary for Zvi to win.
Zvi thinks there’s a 70% chance of the following: “Omicron will blow through the US by 3/1/2022, leading to herd immunity and something like the ‘end’ of the COVID-19 pandemic.”
Holden proposed a bet and apparently they went back and forth a few emails on the bet operationalization before agreeing. My hypothesis then is that Zvi was anchored on his 70% confidence from this statement and didn’t take the time to properly re-evaluate his forecast for the specific bet operationalization they ultimately agreed to. I can easily see him only spending a small amount of time thinking about the bet operationalization and agreeing to it without realizing that it’s very different than the concept he was originally assigning 70% to due to wanting to agree to a bet on principle and not wanting to go back and forth a few more times by email.
Of course this is just a story and perhaps he did give consider the bet operationalization carefully. But I think even smart people with good judgment can easily make mistakes like this. If Zvi read my comment and responded, “Your hypothesis is wrong; I thought carefully about the bet operationalization and I’m confident I made a good bet” I’d meta-update on him a lot more—maybe up to 50% like Holden did. But I don’t know enough about Zvi to know whether the mere fact that he agreed to a public bet with Holden is strong evidence that he thought about the exact terms carefully and wasn’t biased by his strong belief in his original 70% statement.
(Noting to myself publicly that I want to learn to be more concise with my thoughts. This comment was too long.)
[Holden won the bet](https://thezvi.substack.com/i/66658630/i-lose-a-bet). In retrospect, I think I was justified in having high confidence and right that Zvi’s bet was foreseeably bad. If anything, when I lowered my forecast from 95% to 68% for a couple weeks in April I was meta-updating too much on the community median and assigning too much weight to the possibility of an extremely large “cases to true infections” adjustment.
Note that I disagree with what Zvi wrote yesterday: “In hindsight, the question of ‘what counts as Omicron’ does have a strong bearing on who had the right side of this wager, and also is a key insight into the mistake that I made here.”
I disagree that that was his mistake. Even if subvariants counted as different variants that would only increase the chance that the bet resolves ambiguously. There was (IMO) never a >70% chance (or even >50%) chance that Zvi would win (conditional on someone winning), even if the language of the bet considered subvariants to be different variant.
Will Holden win his bet with Zvi?Will Holden win his bet with Zvi?Will Zvi win his bet with Holden?Will Zvi win his bet with Holden?Will Holden and Zvi's Omicron bet resolve ambiguously?Will Holden and Zvi's Omicron bet resolve ambiguously?
Also, this paragraph from Holden really resonated with me:
In an ideal world, we’d be making so many bets like this that our track records would give clear evidence of which of us was a better predictor, overall. But I don’t think that’s going to happen; it’s a lot of work even to nail down a pretty simple, vivid disagreement like this one (and most important disagreements are much harder to reach bets on, and even this one may require a third-party judgment-driven adjustment). I don’t think that whichever of us wins this one bet should gain too much credibility relative to the other.
What kind of tools, sites, or economic structures could enable this ideal world? At Mantic we’re hoping accessible, user-created prediction markets will do the trick, but would love to hear alternative proposals!
I don’t have a great answer to your question. I think the easier and more normalized it gets to make bets like this, the more of them there will be—that’s about what I’ve got.
In practice I find some of the hardest parts of making bets like this are (a) noticing when a disagreement is of the right form such that it’s likely to be tractable to turn it into a bet; (b) hashing out all the details of how the bet will be resolved and trying to make them closely match the original conceptual disagreement. (b) is usually so much work that it doesn’t end up being worth it for the direct rewards (financial and otherwise), so some sort of norm that this is a virtuous thing to do could be important (but also, if there were a way to make (b) easier, that would be amazing).
Comments on “Omicron bet” will go here.
Added to Metaculus:
Will Holden win his Bet with Zvi about Omicron, conditional on one of them winning?
Will Holden’s Bet with Zvi about Omicron resolve ambiguously? (pending approval as of this comment)
Bet with Zvi about Omicron:
I’m definitely on Holden’s side of the bet.
In summary, I assign 80% to Holden’s outcome, 15% to the ambiguous “push” outcome, and 5% to Zvi’s outcome.
This is a low-information forecast, but there seem to be three outcomes to the bet, and Zvi’s outcome clearly seems to be the least likely:
(1) For Zvi to win, Covid cases (of all variants, including any future ones) need to average ~18 times higher over the first two months of 2022 than over the following 12 months (math: 18 = (0.75/2) / (0.25/12)).
18 is such a high ratio given Covid’s track record so far. The ratio of the 7-day-average of US cases from its high (1/11/2021 = ~256,000/day) to its low (6/21/2021 = ~12,000/day) is ~21, barely higher than ~18. Plus, those two weeks were months apart, giving time for cases to drop off.
I don’t see a very plausible way to get that kind of ratio for the first two months of 2022 over the 12 months afterwards. E.g. It seems unlikely that cases would drop off sufficiently quickly at the end of February to avoid adding a large number of cases in March (and to a lesser degree, April, etc). (i.e. Even if cases virtually disappeared later in 2022 (such that the second period being 12 months instead of 6 doesn’t matter much), it’s really hard for cases to drop off so quickly that the number of cases from March 1 onwards don’t end up being at least a third of the number of cases from January and February.) The prior on that steep drop-off happening by the end of February is quite low and the fact that it’s only a little over a week until January and cases are still on the rise doesn’t make it seem more likely that there will be a steep drop-off before March. There’s just no way Zvi could know that that is likely going to happen. I don’t need to read his post to know that he doesn’t know that.
Given this simple consideration that cases would have to drop off exceptionally fast at just the right time for Zvi’s outcome to happen, I assign a 5% chance to Zvi’s outcome happening.
It was really just strongly disagreeing that Zvi’s outcome seemed likely that made me want to write this comment, but I’ll go ahead and write estimates for the other two outcomes:
(2) For Holden to win, cases in the 12-month period after February 2022 have to exceed one-third of cases in the first two months of 2022 before a new variant comes along and takes over. I haven’t heard of any new variant after Omicron. Such a new variant would have to be much more contagious than Omicron for it to have a chance to take over everywhere in the US in time to stop Omicron’s cases from exceeding that one-third threshold in March-May. I hear Omicron is super contagious so that seems unlikely. Therefore, my low-information forecast is that Holden’s outcome is 80% likely.
(3) For the bet to be a “push” (i.e. for the bet to resolve ambiguously), a new variant or variants need to take over before Omicron-and-previous-variant cases after March 1 exceed the one-third threshold mentioned above. This seems more likely than Zvi’s outcome, but still not that likely. I’ll assign 15% to the ambiguous outcome.
Your analysis roughly matches my independent impression, but I’m pretty sure this simple consideration didn’t escape Zvi’s attention. So, it seems that you can’t so easily jump from that analysis to the conclusion that Holden will win the bet, unless you didn’t think much of Zvi as a reasoner to begin with or had a plausible error theory to explain this particular instance.
Yes, you’re quite right, thanks. I failed to differentiate between my independent impression and my all-things-considered view when thinking about and writing the above. Thinking about it now, I realize ~5% is basically my independent impression, not my all-things-considered view. My all-things-considered view is more like ~20% Zvi wins—and if you told me yours was 40% then I’d update to ~35%, though I’d guess yours is more like ~25%. I meta-updated upwards based on knowing Zvi’s view and the fact that Holden updated upwards on Zvi to 50%. (And even if I didn’t know their views, my initial naive all-things-considered forecasts would very rarely be as far from 50% as 5% is unless there’s a clear base rate that is that extreme.).
That said, I haven’t read much of what Zvi has written in general and the one thing I do remember reading of his on Covid (his 12/24/20 Covid post) I strongly disagreed with at the time (and it turns out he was indeed overconfident). I recognize that this probably makes me biased against Zvi’s judgment, leading me to want to meta-update on his view less than I probably should (since I hear a lot of people think he has good judgment and there probably are a lot of other predictions he’s made which were good that I’m just not aware of), but at the same I really don’t personally have good evidence of his forecasting track record in the way that I do of e.g. your record, so I’m much less inclined to meta-update a lot on him than I would e.g. on you.
Additionally, I did think of a plausible error theory earlier after writing the 5% forecast (specifically: a plausible story for how Zvi could have accepted such a bet at terrible odds). (I said this out loud to someone at the time rather than type it:) My thought was that Zvi’s view in the conceptual disagreement they were betting on seems much more plausible to me than Zvi’s position in the bet operationalization. That is, there are many scenarios that would make it look like Zvi was basically right that might technically cache out as a Holden win according to the exact betting terms described here. For example, there might be a huge Omicron wave—the largest Covid wave yet—and cases might drop quickly afterwards and it might be the last wave of the pandemic, and yet despite all of that, perhaps only 50% of the cases after January 1, 2022 happen before the end of February rather than the 75% necessary for Zvi to win.
Holden proposed a bet and apparently they went back and forth a few emails on the bet operationalization before agreeing. My hypothesis then is that Zvi was anchored on his 70% confidence from this statement and didn’t take the time to properly re-evaluate his forecast for the specific bet operationalization they ultimately agreed to. I can easily see him only spending a small amount of time thinking about the bet operationalization and agreeing to it without realizing that it’s very different than the concept he was originally assigning 70% to due to wanting to agree to a bet on principle and not wanting to go back and forth a few more times by email.
Of course this is just a story and perhaps he did give consider the bet operationalization carefully. But I think even smart people with good judgment can easily make mistakes like this. If Zvi read my comment and responded, “Your hypothesis is wrong; I thought carefully about the bet operationalization and I’m confident I made a good bet” I’d meta-update on him a lot more—maybe up to 50% like Holden did. But I don’t know enough about Zvi to know whether the mere fact that he agreed to a public bet with Holden is strong evidence that he thought about the exact terms carefully and wasn’t biased by his strong belief in his original 70% statement.
(Noting to myself publicly that I want to learn to be more concise with my thoughts. This comment was too long.)
[Holden won the bet](https://thezvi.substack.com/i/66658630/i-lose-a-bet). In retrospect, I think I was justified in having high confidence and right that Zvi’s bet was foreseeably bad. If anything, when I lowered my forecast from 95% to 68% for a couple weeks in April I was meta-updating too much on the community median and assigning too much weight to the possibility of an extremely large “cases to true infections” adjustment.
Note that I disagree with what Zvi wrote yesterday: “In hindsight, the question of ‘what counts as Omicron’ does have a strong bearing on who had the right side of this wager, and also is a key insight into the mistake that I made here.”
I disagree that that was his mistake. Even if subvariants counted as different variants that would only increase the chance that the bet resolves ambiguously. There was (IMO) never a >70% chance (or even >50%) chance that Zvi would win (conditional on someone winning), even if the language of the bet considered subvariants to be different variant.
I set up a prediction market for this bet! https://mantic.markets/AustinChen/will-at-least-75-of-the-usa-covid19
Also, this paragraph from Holden really resonated with me:
What kind of tools, sites, or economic structures could enable this ideal world? At Mantic we’re hoping accessible, user-created prediction markets will do the trick, but would love to hear alternative proposals!
Very cool, thanks!
I don’t have a great answer to your question. I think the easier and more normalized it gets to make bets like this, the more of them there will be—that’s about what I’ve got.
In practice I find some of the hardest parts of making bets like this are (a) noticing when a disagreement is of the right form such that it’s likely to be tractable to turn it into a bet; (b) hashing out all the details of how the bet will be resolved and trying to make them closely match the original conceptual disagreement. (b) is usually so much work that it doesn’t end up being worth it for the direct rewards (financial and otherwise), so some sort of norm that this is a virtuous thing to do could be important (but also, if there were a way to make (b) easier, that would be amazing).