This is reasonable except that it misunderstands peer review. Peer review does not check correctness*; it often doesn’t even really check rigour. Instead, it sometimes catches the worst half of outright errors and it enforces the discipline’s conventions. Many of those conventions are good and correlate with rigour. But when it comes to new methods it’s usually an uphill battle to get them accepted, rigour be damned.
We note above that our cost analysis (designed, implemented and calculated inside of three weeks) had weaknesses, and this is on us. But the strength of opposition to the whole approach (and not to our implementation) persuaded us that it was futile to iterate.
On forecasts: We used forecasters with an extremely good public track record. I have far more confidence in their general ability to give great estimates than I do in the mathematised parts of covid science. (I can send you their stats and then you will too.) You can tell that the reviewers’ antipathy was not about rigour because they were perfectly happy for us to set our priors based on weak apriori arguments and naive averages of past studies (the normal unrigorous way that priors are chosen).
In my limited experience, even major world governments rely heavily on academic estimates of all things, including economic costs. And as we noted we see no sign that they took into account the noneconomic cost, and nor did almost all academics. The pandemic really should make you downgrade your belief in secretly competent institutions.
The IHME section is there to note their object level failure, not to criticize the Gates Foundation for not funding us instead. (I don’t know to what extent their covid debacle should affect our estimate of the whole org—but not by nothing, since management allowed them to run amok for years.)
This is reasonable except that it misunderstands peer review. Peer review does not check correctness*; it often doesn’t even really check rigour. Instead, it sometimes catches the worst half of outright errors and it enforces the discipline’s conventions. Many of those conventions are good and correlate with rigour. But when it comes to new methods it’s usually an uphill battle to get them accepted, rigour be damned.
We note above that our cost analysis (designed, implemented and calculated inside of three weeks) had weaknesses, and this is on us. But the strength of opposition to the whole approach (and not to our implementation) persuaded us that it was futile to iterate.
On forecasts: We used forecasters with an extremely good public track record. I have far more confidence in their general ability to give great estimates than I do in the mathematised parts of covid science. (I can send you their stats and then you will too.) You can tell that the reviewers’ antipathy was not about rigour because they were perfectly happy for us to set our priors based on weak apriori arguments and naive averages of past studies (the normal unrigorous way that priors are chosen).
In my limited experience, even major world governments rely heavily on academic estimates of all things, including economic costs. And as we noted we see no sign that they took into account the noneconomic cost, and nor did almost all academics. The pandemic really should make you downgrade your belief in secretly competent institutions.
The IHME section is there to note their object level failure, not to criticize the Gates Foundation for not funding us instead. (I don’t know to what extent their covid debacle should affect our estimate of the whole org—but not by nothing, since management allowed them to run amok for years.)
* Except in mathematics