That definitely can happen, and makes me realise my comment wasn’t sufficiently precise.
An expected value estimate can be reasonable even if there’s a 99% chance it would be revised downwards given more info, if there’s also a 1% chance it would be revised upwards by enough to offset the potential downward revisions. If an estimator makes such an estimate and is well-calibrated, I wouldn’t say they’re making a mistake, and thus probably wouldn’t say they’re being “overly optimistic”.
The claim I was making was that one’s naive/early/quick analyses of how good (or cost-effective, or whatever) tend to not be well-calibrated, systematically erring towards optimism in a way that means that it’s best to adjust the expected value downwards to account for this (unless one has already made such an adjustment).
But I’m not actually sure how true that claim is (I’m just basing it on my memory of GiveWell posts I read in the past). Maybe most things that look like that situation are either actually the optimiser’s curse or actually the sort of situation you describe.
That definitely can happen, and makes me realise my comment wasn’t sufficiently precise.
An expected value estimate can be reasonable even if there’s a 99% chance it would be revised downwards given more info, if there’s also a 1% chance it would be revised upwards by enough to offset the potential downward revisions. If an estimator makes such an estimate and is well-calibrated, I wouldn’t say they’re making a mistake, and thus probably wouldn’t say they’re being “overly optimistic”.
The claim I was making was that one’s naive/early/quick analyses of how good (or cost-effective, or whatever) tend to not be well-calibrated, systematically erring towards optimism in a way that means that it’s best to adjust the expected value downwards to account for this (unless one has already made such an adjustment).
But I’m not actually sure how true that claim is (I’m just basing it on my memory of GiveWell posts I read in the past). Maybe most things that look like that situation are either actually the optimiser’s curse or actually the sort of situation you describe.