Late to the party, but I was re-reading this as it relates to another post I’m working on, and I realised I have a question. You write: (note that I say “you” in this comment a lot, but I’d also be interested in anyone else’s thoughts on my questions)
The optimizer’s curse can show up even in situations where effective altruists’ prioritization decisions don’t involve formal models or explicit estimates of expected value. Someone informally assessing philanthropic opportunities in a linear manner might have a thought like:
Thing X seems like an awfully big issue. Funding Group A would probably cost only a little bit of money and have a small chance leading to a solution for Thing X. Accordingly, I feel decent about the expected cost-effectiveness of funding Group A.
Let me compare that to how I feel about some other funding opportunities…
Although the thinking is informal, there’s uncertainty, potential for bias, and an optimization-like process.
That makes sense to me, and seems a very worthwhile point. (It actually seems to me it might have been worth emphasising more, as I think a casual reader could think this post was a critique of formal/explicit/quantitative models in particular.)
But then in a footnote, you add:
Informal thinking isn’t always this linear. If the informal thinking considers an opportunity from multiple perspectives, draws on intuitions, etc., the risk of postdecision surprise may be reduced.
I’m not sure I understand what you mean by that, or if it’s true/makes sense. It seems to me that, ultimately, if we’re engaging in a process that effectively provides a ranking of how good the options seem (whether based on cost-effectiveness estimates or just how we “feel” about them), and there’s uncertainty involved, and we pick the option that seems to come out on top, the optimizer’s curse will be relevant. Even if we use multiple separate informal ways of looking at the problem, we still ultimately end up with a top ranked option, and, given that that option’s ended up on top, we should still expect that errors have inflated its apparent value (whether that’s in numerical terms or in terms of how we feel) more than average. Right?
Or did you simply mean that using multiple perspectives means that the various different errors and uncertainties might be more likely to balance out (in the same sort of way that converging lines of evidence based on different methodologies make us more confident that we’ve really found something real), and that, given that there’d effectively be less uncertainty, the significance of the optimizer’s curse would be smaller. (This seems to fit with “the risk of postdecision surprise may be reduced”.)
If that’s what you meant, that seems reasonable to me, but it seems that we could get the same sort of benefits just by doing something like gathering more data or improving our formal models. (Though of course that may often be more expensive and difficult than cluster thinking, so highlighting that we also have the option of cluster thinking does seem useful.)
Just saw this comment, I’m also super late to the party responding to you!
It actually seems to me it might have been worth emphasising more, as I think a casual reader could think this post was a critique of formal/explicit/quantitative models in particular.
Totally agree! Honestly, I had several goals with this post, and I almost complete failed on two of them:
Arguing why utilitarianism can’t be the foundation of ethics.
Without talking much about AI, explaining why I don’t think people in the EA community are being reasonable when they suggest there’s a decent chance of an AGI being developed in the near future.
Instead, I think this post came off as primarily a criticism of certain kinds of models and a criticism of GiveWell’s approach to prioritization (which is unfortunate since I think the Optimizer’s Curse isn’t as big an issue for GiveWell & global health as it is for many other EA orgs/cause areas).
-- On the second piece of your comment, I think we mostly agree. Informal/cluster-style thinking is probably helpful, but it definitely doesn’t make the Optimizer’s Curse a non-issue.
Late to the party, but I was re-reading this as it relates to another post I’m working on, and I realised I have a question. You write: (note that I say “you” in this comment a lot, but I’d also be interested in anyone else’s thoughts on my questions)
That makes sense to me, and seems a very worthwhile point. (It actually seems to me it might have been worth emphasising more, as I think a casual reader could think this post was a critique of formal/explicit/quantitative models in particular.)
But then in a footnote, you add:
I’m not sure I understand what you mean by that, or if it’s true/makes sense. It seems to me that, ultimately, if we’re engaging in a process that effectively provides a ranking of how good the options seem (whether based on cost-effectiveness estimates or just how we “feel” about them), and there’s uncertainty involved, and we pick the option that seems to come out on top, the optimizer’s curse will be relevant. Even if we use multiple separate informal ways of looking at the problem, we still ultimately end up with a top ranked option, and, given that that option’s ended up on top, we should still expect that errors have inflated its apparent value (whether that’s in numerical terms or in terms of how we feel) more than average. Right?
Or did you simply mean that using multiple perspectives means that the various different errors and uncertainties might be more likely to balance out (in the same sort of way that converging lines of evidence based on different methodologies make us more confident that we’ve really found something real), and that, given that there’d effectively be less uncertainty, the significance of the optimizer’s curse would be smaller. (This seems to fit with “the risk of postdecision surprise may be reduced”.)
If that’s what you meant, that seems reasonable to me, but it seems that we could get the same sort of benefits just by doing something like gathering more data or improving our formal models. (Though of course that may often be more expensive and difficult than cluster thinking, so highlighting that we also have the option of cluster thinking does seem useful.)
Just saw this comment, I’m also super late to the party responding to you!
Totally agree! Honestly, I had several goals with this post, and I almost complete failed on two of them:
Arguing why utilitarianism can’t be the foundation of ethics.
Without talking much about AI, explaining why I don’t think people in the EA community are being reasonable when they suggest there’s a decent chance of an AGI being developed in the near future.
Instead, I think this post came off as primarily a criticism of certain kinds of models and a criticism of GiveWell’s approach to prioritization (which is unfortunate since I think the Optimizer’s Curse isn’t as big an issue for GiveWell & global health as it is for many other EA orgs/cause areas).
--
On the second piece of your comment, I think we mostly agree. Informal/cluster-style thinking is probably helpful, but it definitely doesn’t make the Optimizer’s Curse a non-issue.