Here’s why the post reminds me of the planning fallacy: When people make flawed plans, these plans don’t seem unrealistic. They often consist of detailed steps, each of which is quite likely to succeed. And yet, in most cases, the world takes a different turn and the planned project ends up late and more expensive.
You describe a “happy timeline” that’s analogous to such a plan. For it to work, we would have to make many good decisions; many unknown obstacles would have to be overcome; and many novel ideas (like human challenge trials) accepted. None of these is unrealistic when looked at individually. But collectively, it is very unlikely that all these factors could realistically come together to form your happy timeline.
One example to illustrate this: Your post strongly favors vaccines and also attributes enormous costs to lockdowns. I think that this is realistic, but I can think so only in hindsight. In early 2020, it wasn’t at all clear that sufficiently strong lockdowns wouldn’t bring the pandemic to a manageable level or at least buy us the time we need for vaccine development. Remember the hammer and the dance? Yet, around that same time in your happy timeline, decisions are made to approve and pre-purchase vaccines at high costs. With hindsight, it’s easy to say that we should have paid these costs; but at the time, it wasn’t obvious at all. The answer to this question wouldn’t have been easier to find with better institutional decision-making, either. It was simply a difficult question with no clear answer at that time.
Human challenge trials are a very old idea. Not doing them is the aberration.
Lockdown uncertainty seems moot. I’m not arguing that any lockdown policy should have been different (except that we might have lifted it a few months early if vaccinations were successfully time-shifted). Did anyone think that (realistic, non-Chinese, non-remote-island) lockdowns were an alternative to vaccines—back when we thought vaccines were coming in 2022 or 2027? The UK government seriously thought they could only do lockdown for a month or two. It doesn’t add up.
But my real response is: of course the above dates aren’t exactly how it would go, of course the cost estimate isn’t exactly how much it would take; I’m not god. I’m not even Derek Lowe. But do you really think it would have been slower than what we did? Unless you do, it doesn’t seem wiser to me to refuse to estimate—even given that estimation is fallible.
Hmm… Here’s how I understand your estimate. Is that a fair summary?
If all had gone according to a perfectly happy timeline where everyone makes the right decisions, we could have had enough vaccines in August.
This would be worth approximately 205 million QALYs.
It would also cost approximately 0.7 trillion dollars.
That’s 3400 dollars per QALY.
My concern (expressed in the comments above) is mainly that the happy timeline is unrealistic, so the estimate could be off by a large factor, similarly to how the time and cost estimates of our plans are often off by a large factor.
Your estimate is probably still valuable, even if it is imprecise. We can use it to think about whether vaccine development is cost-effective; I reckon 3400$/QALY puts the cost-effectiveness an order of magnitude below effective charities and some orders of magnitude above many other public health interventions. Is that a fair conclusion?
I’d like to take away more from your post than just the estimate, but am not sure at the moment what other recommendations I can take from it...
The estimate undersells long COVID because we don’t know how many years of 3.2m QALYs to add, but yes that’s roughly it. And yes, I only claim that it was a decent deal, particularly since the funding for it couldn’t really have gone on something else.
I freely admit that it could be off by a large factor (see my final paragraph). I would love for someone to come and do a proper Bayesian interval version, which would foreground the uncertainty.
I continue to challenge calling it “unrealistic”, on priors, just because it’s very uncertain. Last January, a historical baseline would have called it unrealistic to expect completion of vaccine R&D, trials, approvals, and distribution in 11 months. But here we are. I would have you be less sure about what’s not possible, or not realistic.
Here’s why the post reminds me of the planning fallacy: When people make flawed plans, these plans don’t seem unrealistic. They often consist of detailed steps, each of which is quite likely to succeed. And yet, in most cases, the world takes a different turn and the planned project ends up late and more expensive.
You describe a “happy timeline” that’s analogous to such a plan. For it to work, we would have to make many good decisions; many unknown obstacles would have to be overcome; and many novel ideas (like human challenge trials) accepted. None of these is unrealistic when looked at individually. But collectively, it is very unlikely that all these factors could realistically come together to form your happy timeline.
One example to illustrate this: Your post strongly favors vaccines and also attributes enormous costs to lockdowns. I think that this is realistic, but I can think so only in hindsight. In early 2020, it wasn’t at all clear that sufficiently strong lockdowns wouldn’t bring the pandemic to a manageable level or at least buy us the time we need for vaccine development. Remember the hammer and the dance? Yet, around that same time in your happy timeline, decisions are made to approve and pre-purchase vaccines at high costs. With hindsight, it’s easy to say that we should have paid these costs; but at the time, it wasn’t obvious at all. The answer to this question wouldn’t have been easier to find with better institutional decision-making, either. It was simply a difficult question with no clear answer at that time.
Human challenge trials are a very old idea. Not doing them is the aberration.
Lockdown uncertainty seems moot. I’m not arguing that any lockdown policy should have been different (except that we might have lifted it a few months early if vaccinations were successfully time-shifted). Did anyone think that (realistic, non-Chinese, non-remote-island) lockdowns were an alternative to vaccines—back when we thought vaccines were coming in 2022 or 2027? The UK government seriously thought they could only do lockdown for a month or two. It doesn’t add up.
But my real response is: of course the above dates aren’t exactly how it would go, of course the cost estimate isn’t exactly how much it would take; I’m not god. I’m not even Derek Lowe. But do you really think it would have been slower than what we did? Unless you do, it doesn’t seem wiser to me to refuse to estimate—even given that estimation is fallible.
Hmm… Here’s how I understand your estimate. Is that a fair summary?
If all had gone according to a perfectly happy timeline where everyone makes the right decisions, we could have had enough vaccines in August.
This would be worth approximately 205 million QALYs.
It would also cost approximately 0.7 trillion dollars.
That’s 3400 dollars per QALY.
My concern (expressed in the comments above) is mainly that the happy timeline is unrealistic, so the estimate could be off by a large factor, similarly to how the time and cost estimates of our plans are often off by a large factor.
Your estimate is probably still valuable, even if it is imprecise. We can use it to think about whether vaccine development is cost-effective; I reckon 3400$/QALY puts the cost-effectiveness an order of magnitude below effective charities and some orders of magnitude above many other public health interventions. Is that a fair conclusion?
I’d like to take away more from your post than just the estimate, but am not sure at the moment what other recommendations I can take from it...
The estimate undersells long COVID because we don’t know how many years of 3.2m QALYs to add, but yes that’s roughly it. And yes, I only claim that it was a decent deal, particularly since the funding for it couldn’t really have gone on something else.
I freely admit that it could be off by a large factor (see my final paragraph). I would love for someone to come and do a proper Bayesian interval version, which would foreground the uncertainty.
I continue to challenge calling it “unrealistic”, on priors, just because it’s very uncertain. Last January, a historical baseline would have called it unrealistic to expect completion of vaccine R&D, trials, approvals, and distribution in 11 months. But here we are. I would have you be less sure about what’s not possible, or not realistic.