One explanation for why people commonly underestimate project completion time is this: people consider a best-case scenario where everything goes smoothly. Like the “happy timeline” in this post. Alas, reality is not like this at all.
When people are asked for a “realistic” scenario, they envision everything going exactly as planned, with no unexpected delays or unforeseen catastrophes—the same vision as their “best case.”
Reality, it turns out, usually delivers results somewhat worse than the “worst case.”
-- Eliezer Yudkowsky, in the post linked above.
The popular solution for countering the planning fallacy is taking an outside view (aka reference class forecasting). In that light, Covid vaccines looks surprisingly good. As you note at the top of your post, Covid vaccines were developed quickly compared to the reference class of vaccines.
I’m very sympathetic to the idea of writing down our collective mistakes and trying to learn from them. It just seems to me that this post contrasts our mistakes with a highly unrealistic, idealized timeline. My guess is that more realistic expectations might lead to better goals for improving the world 🤔
I don’t recognise my post in this description. I openly acknowledge that there are bottlenecks., including unknown bottlenecks. I put a 150% interval on the key uncertainty. (I am protected somewhat from Hofstadter’s law there by the reference point of the Braintree facility, with its almost known lead time.)
It’s not unrealistic to pay weekend overtime or make new weekend hires for regulators, in the biggest health crisis of the century. It’s not unrealistic for a single Chinese scientist to just decide on his own to release the genome he already sequenced, two weeks early. It’s not unrealistic to decide to stop wasting three-quarters of the precious vaccine, once you realise you are doing so and that the rules you set blindly a year ago were arbitrary choices. Approval sharing between rich countries is not unrealistic. It is not totally unrealistic to imagine that a suitably alarmed establishment might throw whole-number fractions of their wealth at solutions, rapidly; the Manhattan project had much worse odds than any COVID vaccine effort. Challenge trials didn’t happen, afaict, because a quite small number of people said so.
To the extent that any of the above were unrealistic, it’s for tractable social reasons, and tractable social reasons serve my point about institutional decision-making.
Moreover, while reference class forecasting is a powerful element in prediction, it often fails when equilibria are not efficient. Policy seems like a perfect example. Yudkowsky’s updated view is closer to my post than to one-step thought-terminating modesty.
It is important to realise how slow vaccines and all medical innovation is in general. That’s why I say it. But it is also important to investigate whether that is for hard-to-change reasons, or not.
If we had used your reference class in March 2020, you would (and we did) predict that the COVID vaccines were going to take 7 years. If this forecast was taken seriously by the wrong people—as it may well have been, given the sluggishness of the “Warp Speed” initiatives! - then a great deal of good would have been, and was, lost.
Here’s why the post reminds me of the planning fallacy: When people make flawed plans, these plans don’t seem unrealistic. They often consist of detailed steps, each of which is quite likely to succeed. And yet, in most cases, the world takes a different turn and the planned project ends up late and more expensive.
You describe a “happy timeline” that’s analogous to such a plan. For it to work, we would have to make many good decisions; many unknown obstacles would have to be overcome; and many novel ideas (like human challenge trials) accepted. None of these is unrealistic when looked at individually. But collectively, it is very unlikely that all these factors could realistically come together to form your happy timeline.
One example to illustrate this: Your post strongly favors vaccines and also attributes enormous costs to lockdowns. I think that this is realistic, but I can think so only in hindsight. In early 2020, it wasn’t at all clear that sufficiently strong lockdowns wouldn’t bring the pandemic to a manageable level or at least buy us the time we need for vaccine development. Remember the hammer and the dance? Yet, around that same time in your happy timeline, decisions are made to approve and pre-purchase vaccines at high costs. With hindsight, it’s easy to say that we should have paid these costs; but at the time, it wasn’t obvious at all. The answer to this question wouldn’t have been easier to find with better institutional decision-making, either. It was simply a difficult question with no clear answer at that time.
Human challenge trials are a very old idea. Not doing them is the aberration.
Lockdown uncertainty seems moot. I’m not arguing that any lockdown policy should have been different (except that we might have lifted it a few months early if vaccinations were successfully time-shifted). Did anyone think that (realistic, non-Chinese, non-remote-island) lockdowns were an alternative to vaccines—back when we thought vaccines were coming in 2022 or 2027? The UK government seriously thought they could only do lockdown for a month or two. It doesn’t add up.
But my real response is: of course the above dates aren’t exactly how it would go, of course the cost estimate isn’t exactly how much it would take; I’m not god. I’m not even Derek Lowe. But do you really think it would have been slower than what we did? Unless you do, it doesn’t seem wiser to me to refuse to estimate—even given that estimation is fallible.
Hmm… Here’s how I understand your estimate. Is that a fair summary?
If all had gone according to a perfectly happy timeline where everyone makes the right decisions, we could have had enough vaccines in August.
This would be worth approximately 205 million QALYs.
It would also cost approximately 0.7 trillion dollars.
That’s 3400 dollars per QALY.
My concern (expressed in the comments above) is mainly that the happy timeline is unrealistic, so the estimate could be off by a large factor, similarly to how the time and cost estimates of our plans are often off by a large factor.
Your estimate is probably still valuable, even if it is imprecise. We can use it to think about whether vaccine development is cost-effective; I reckon 3400$/QALY puts the cost-effectiveness an order of magnitude below effective charities and some orders of magnitude above many other public health interventions. Is that a fair conclusion?
I’d like to take away more from your post than just the estimate, but am not sure at the moment what other recommendations I can take from it...
The estimate undersells long COVID because we don’t know how many years of 3.2m QALYs to add, but yes that’s roughly it. And yes, I only claim that it was a decent deal, particularly since the funding for it couldn’t really have gone on something else.
I freely admit that it could be off by a large factor (see my final paragraph). I would love for someone to come and do a proper Bayesian interval version, which would foreground the uncertainty.
I continue to challenge calling it “unrealistic”, on priors, just because it’s very uncertain. Last January, a historical baseline would have called it unrealistic to expect completion of vaccine R&D, trials, approvals, and distribution in 11 months. But here we are. I would have you be less sure about what’s not possible, or not realistic.
I’m with Sjlver. There’s a lot of hand waving past problems in the happy timeline: eg, how long does it take to train new regulators who are going to work weekends? How quickly do trained regulators working constant overtime burn out? What happens when the Chinese government denies that the genome released early is the right one?
(Also, the invocation of “war time” as a model assumes that wars are run any more efficiently or even urgently. They’re not.)
You seem to be mistaking this for a white paper, or a piece of legislation, or an itemized purchase order for one different timeline please. It is not that. It is instead a thing to measure our situation against, to short-circuit the useless shrugging described in the opening section.
It would be difficult for more seriousness, more money, more personal and institutional courage to not help. I struggle to understand why you are so sure it wouldn’t, or, if you do, why you’re pointing out that unexpected things happen, on occasion.
In fact the genome was released by one person, and the government didn’t say anything [against the genome]. He could have done it a week earlier still.
I was mistaking this for a genuine analysis of what was possible, not a hand waving experiment in impossibility. But since it’s the latter, can I have a pony too?
Measuring the situation against an implausibly analyzed alternate scenario is not useful.
It’s nice to see the replies defuse things a bit, but this comment was unnecessarily rude. It’s fine to disagree on which scenarios are plausible, but please be polite in the process.
He seems to make really conservative assumptions though, what do you think was unrealistic? I don’t think its unrealistic for the u.s.a to not ban people from testing themselves early in the pandemic, for example.
Challenge trails seems obviously good and not unrealistic.
This post seems to fall victim to the planning fallacy.
One explanation for why people commonly underestimate project completion time is this: people consider a best-case scenario where everything goes smoothly. Like the “happy timeline” in this post. Alas, reality is not like this at all.
The popular solution for countering the planning fallacy is taking an outside view (aka reference class forecasting). In that light, Covid vaccines looks surprisingly good. As you note at the top of your post, Covid vaccines were developed quickly compared to the reference class of vaccines.
I’m very sympathetic to the idea of writing down our collective mistakes and trying to learn from them. It just seems to me that this post contrasts our mistakes with a highly unrealistic, idealized timeline. My guess is that more realistic expectations might lead to better goals for improving the world 🤔
I don’t recognise my post in this description. I openly acknowledge that there are bottlenecks., including unknown bottlenecks. I put a 150% interval on the key uncertainty. (I am protected somewhat from Hofstadter’s law there by the reference point of the Braintree facility, with its almost known lead time.)
It’s not unrealistic to pay weekend overtime or make new weekend hires for regulators, in the biggest health crisis of the century. It’s not unrealistic for a single Chinese scientist to just decide on his own to release the genome he already sequenced, two weeks early. It’s not unrealistic to decide to stop wasting three-quarters of the precious vaccine, once you realise you are doing so and that the rules you set blindly a year ago were arbitrary choices. Approval sharing between rich countries is not unrealistic. It is not totally unrealistic to imagine that a suitably alarmed establishment might throw whole-number fractions of their wealth at solutions, rapidly; the Manhattan project had much worse odds than any COVID vaccine effort. Challenge trials didn’t happen, afaict, because a quite small number of people said so.
To the extent that any of the above were unrealistic, it’s for tractable social reasons, and tractable social reasons serve my point about institutional decision-making.
Moreover, while reference class forecasting is a powerful element in prediction, it often fails when equilibria are not efficient. Policy seems like a perfect example. Yudkowsky’s updated view is closer to my post than to one-step thought-terminating modesty.
It is important to realise how slow vaccines and all medical innovation is in general. That’s why I say it. But it is also important to investigate whether that is for hard-to-change reasons, or not.
If we had used your reference class in March 2020, you would (and we did) predict that the COVID vaccines were going to take 7 years. If this forecast was taken seriously by the wrong people—as it may well have been, given the sluggishness of the “Warp Speed” initiatives! - then a great deal of good would have been, and was, lost.
Here’s why the post reminds me of the planning fallacy: When people make flawed plans, these plans don’t seem unrealistic. They often consist of detailed steps, each of which is quite likely to succeed. And yet, in most cases, the world takes a different turn and the planned project ends up late and more expensive.
You describe a “happy timeline” that’s analogous to such a plan. For it to work, we would have to make many good decisions; many unknown obstacles would have to be overcome; and many novel ideas (like human challenge trials) accepted. None of these is unrealistic when looked at individually. But collectively, it is very unlikely that all these factors could realistically come together to form your happy timeline.
One example to illustrate this: Your post strongly favors vaccines and also attributes enormous costs to lockdowns. I think that this is realistic, but I can think so only in hindsight. In early 2020, it wasn’t at all clear that sufficiently strong lockdowns wouldn’t bring the pandemic to a manageable level or at least buy us the time we need for vaccine development. Remember the hammer and the dance? Yet, around that same time in your happy timeline, decisions are made to approve and pre-purchase vaccines at high costs. With hindsight, it’s easy to say that we should have paid these costs; but at the time, it wasn’t obvious at all. The answer to this question wouldn’t have been easier to find with better institutional decision-making, either. It was simply a difficult question with no clear answer at that time.
Human challenge trials are a very old idea. Not doing them is the aberration.
Lockdown uncertainty seems moot. I’m not arguing that any lockdown policy should have been different (except that we might have lifted it a few months early if vaccinations were successfully time-shifted). Did anyone think that (realistic, non-Chinese, non-remote-island) lockdowns were an alternative to vaccines—back when we thought vaccines were coming in 2022 or 2027? The UK government seriously thought they could only do lockdown for a month or two. It doesn’t add up.
But my real response is: of course the above dates aren’t exactly how it would go, of course the cost estimate isn’t exactly how much it would take; I’m not god. I’m not even Derek Lowe. But do you really think it would have been slower than what we did? Unless you do, it doesn’t seem wiser to me to refuse to estimate—even given that estimation is fallible.
Hmm… Here’s how I understand your estimate. Is that a fair summary?
If all had gone according to a perfectly happy timeline where everyone makes the right decisions, we could have had enough vaccines in August.
This would be worth approximately 205 million QALYs.
It would also cost approximately 0.7 trillion dollars.
That’s 3400 dollars per QALY.
My concern (expressed in the comments above) is mainly that the happy timeline is unrealistic, so the estimate could be off by a large factor, similarly to how the time and cost estimates of our plans are often off by a large factor.
Your estimate is probably still valuable, even if it is imprecise. We can use it to think about whether vaccine development is cost-effective; I reckon 3400$/QALY puts the cost-effectiveness an order of magnitude below effective charities and some orders of magnitude above many other public health interventions. Is that a fair conclusion?
I’d like to take away more from your post than just the estimate, but am not sure at the moment what other recommendations I can take from it...
The estimate undersells long COVID because we don’t know how many years of 3.2m QALYs to add, but yes that’s roughly it. And yes, I only claim that it was a decent deal, particularly since the funding for it couldn’t really have gone on something else.
I freely admit that it could be off by a large factor (see my final paragraph). I would love for someone to come and do a proper Bayesian interval version, which would foreground the uncertainty.
I continue to challenge calling it “unrealistic”, on priors, just because it’s very uncertain. Last January, a historical baseline would have called it unrealistic to expect completion of vaccine R&D, trials, approvals, and distribution in 11 months. But here we are. I would have you be less sure about what’s not possible, or not realistic.
I’m with Sjlver. There’s a lot of hand waving past problems in the happy timeline: eg, how long does it take to train new regulators who are going to work weekends? How quickly do trained regulators working constant overtime burn out? What happens when the Chinese government denies that the genome released early is the right one?
(Also, the invocation of “war time” as a model assumes that wars are run any more efficiently or even urgently. They’re not.)
You seem to be mistaking this for a white paper, or a piece of legislation, or an itemized purchase order for one different timeline please. It is not that. It is instead a thing to measure our situation against, to short-circuit the useless shrugging described in the opening section.
It would be difficult for more seriousness, more money, more personal and institutional courage to not help. I struggle to understand why you are so sure it wouldn’t, or, if you do, why you’re pointing out that unexpected things happen, on occasion.
In fact the genome was released by one person, and the government didn’t say anything [against the genome]. He could have done it a week earlier still.
I was mistaking this for a genuine analysis of what was possible, not a hand waving experiment in impossibility. But since it’s the latter, can I have a pony too?
Measuring the situation against an implausibly analyzed alternate scenario is not useful.
It’s nice to see the replies defuse things a bit, but this comment was unnecessarily rude. It’s fine to disagree on which scenarios are plausible, but please be polite in the process.
You should go for a Connemara; so good natured, unlike some other breeds.
Oo, I like that one.
He seems to make really conservative assumptions though, what do you think was unrealistic? I don’t think its unrealistic for the u.s.a to not ban people from testing themselves early in the pandemic, for example.
Challenge trails seems obviously good and not unrealistic.