That definitely can happen, and makes me realise my comment wasnāt sufficiently precise.
An expected value estimate can be reasonable even if thereās a 99% chance it would be revised downwards given more info, if thereās also a 1% chance it would be revised upwards by enough to offset the potential downward revisions. If an estimator makes such an estimate and is well-calibrated, I wouldnāt say theyāre making a mistake, and thus probably wouldnāt say theyāre being āoverly optimisticā.
The claim I was making was that oneās naive/āearly/āquick analyses of how good (or cost-effective, or whatever) tend to not be well-calibrated, systematically erring towards optimism in a way that means that itās best to adjust the expected value downwards to account for this (unless one has already made such an adjustment).
But Iām not actually sure how true that claim is (Iām just basing it on my memory of GiveWell posts I read in the past). Maybe most things that look like that situation are either actually the optimiserās curse or actually the sort of situation you describe.
That definitely can happen, and makes me realise my comment wasnāt sufficiently precise.
An expected value estimate can be reasonable even if thereās a 99% chance it would be revised downwards given more info, if thereās also a 1% chance it would be revised upwards by enough to offset the potential downward revisions. If an estimator makes such an estimate and is well-calibrated, I wouldnāt say theyāre making a mistake, and thus probably wouldnāt say theyāre being āoverly optimisticā.
The claim I was making was that oneās naive/āearly/āquick analyses of how good (or cost-effective, or whatever) tend to not be well-calibrated, systematically erring towards optimism in a way that means that itās best to adjust the expected value downwards to account for this (unless one has already made such an adjustment).
But Iām not actually sure how true that claim is (Iām just basing it on my memory of GiveWell posts I read in the past). Maybe most things that look like that situation are either actually the optimiserās curse or actually the sort of situation you describe.