The post title somewhat confuses me since reasons 2, 4, and 5 (miscalibration, anchoring cascades, and granularity failure at the tails) are in-paradigm critiques.
I’m also confused by the link to the XPT tournament as if it illustrates the assertion that EAs take Bayesian reasoning too far, given it did in fact include domain experts and given its headline finding that the forecasts were discrepant between the groups and failed to converge after structured persuasion. Same confusion re: linked GPI paper.
I do think many of your points are correct. The strongest argument to your post title I can think of is sparse evidence + multiple models and you’re uncertain between them → precise priors are unwarranted → use an interval instead, possibly quite wide like [10⁻⁶, 0.5] (just to make something up) → updating may not collapse this wide interval → so EV-maxxing becomes undefined → so switch to other decision criteria, e.g. maybe robustness to harms, which is less Bayesian as per your post title → choose robustly good actions, maintain option value, build capacity etc. Which is basically what most meta interventions are about, no?
Thanks Anthony. Would it be fair to interpret your unawareness series as your steelman of OP’s post title, or as being relevant?
I couldn’t find on a quick look what decision-making approach you would (at least provisionally) endorse instead, bracketing maybe? For my own reference later:
Bracketing says to base our decisions on those consequences we are – in a precise sense – not clueless about, “bracketing out” the others. The idea is that the effects we’re clueless about should not override the obligations given to us by the benefits we aren’t clueless about, like the immediate benefits of malaria nets to the global poor. Thus bracketing could provide action-guidance in the face of cluelessness and in particular support neartermism.
… Take Mogensen’s (2020) example of deciding whether to donate to the Against Malaria Foundation (AMF) or the Make-a-Wish Foundation (MAWF). As Mogensen argues, you’re clueless about the overall effects of donating to AMF vs. MAWF, due to their highly ambiguous effects on population dynamics, economic growth, resource usage, etc. You can come up with lots of asymmetrical effects each intervention has on total value, and you don’t have any principled way of weighing them up, but they still may swamp the immediate effects. Thus it seems that each of the available actions – Donate to AMF, Donate to MAWF, or Do Nothing – is permissible.
And yet, if you’re like me, you suspect that even an impartial consequentialist ought to choose AMF. For we aren’t clueless about the effects of our actions on the immediate beneficiaries! Restricting attention to those who would be prevented from contracting malaria by an AMF donation and the child who would be granted a wish by an MAWF donation, we can rank our actions by their expected total value: AMF MAWF Do Nothing. So my thought is, “Those immediately affected, who I’m not clueless about, give me a reason to Donate to AMF. It is true that, once I start accounting for more moral patients, I’ll become clueless about total value. But my cluelessness about this enlarged set of patients does not overridethe reasons given to me by the immediate beneficiaries. So, still, I’m required to Donate to AMF.” And that’s the essence of bracketing.
I would be particularly interested in how you think meta- and/or longtermist-oriented grantmaking could be improved by your work. I have not been very impressed by the reasoning behind some of these (sometimes quite large) grants, at least on the rare occasions they’ve been shared publicly.
I was making in-paradigm critiques, since, if something is internally invalid, it gives us good reason to think it is also externally invalid.
I linked to the XPT tournament just as an example of asking non-experts for predictions. I thought that pandemics were a good example since most people don’t know much about pandemics at all so we should expect their guesses to be very off. I could see the argument that forecasting research has found forecasters to be better than experts so it makes sense to ask non-experts, but it’s important to note that we haven’t validated this over very long time periods.
I linked to the GPI paper since it seems to pretty absurd to me to try to guess how many future there will be. Even if we can come up with accurate estimates for carrying capacity for different regions of space, I have no clue how we could predict the likelihood of reaching carrying capacity in these regions of space.
In regards to your last point, I think my view is most basically that, by assigning probabilities to outcomes, we’re giving ourselves excessive confidence when we often have too little knowledge to warrant the confidence, that correct decision theories should take into account our cluelessness to a much greater extent, and that people probably do Bayesian reasoning a lot worse than they think.
That said, I’m still figuring out my views on decisions theories so I probably should have researched this topic a lot more before making a post.
For your last point, I agree with you that we run into the problem of EV-maxxing being undefined, but I don’t know where to go from there. It doesn’t seem to make sense to me to do anything in regards to something with such a wide probability range because it seems like you’ll just spend all your time chasing things that you know very little about but which suggest really high EV.
The post title somewhat confuses me since reasons 2, 4, and 5 (miscalibration, anchoring cascades, and granularity failure at the tails) are in-paradigm critiques.
I’m also confused by the link to the XPT tournament as if it illustrates the assertion that EAs take Bayesian reasoning too far, given it did in fact include domain experts and given its headline finding that the forecasts were discrepant between the groups and failed to converge after structured persuasion. Same confusion re: linked GPI paper.
I do think many of your points are correct. The strongest argument to your post title I can think of is sparse evidence + multiple models and you’re uncertain between them → precise priors are unwarranted → use an interval instead, possibly quite wide like [10⁻⁶, 0.5] (just to make something up) → updating may not collapse this wide interval → so EV-maxxing becomes undefined → so switch to other decision criteria, e.g. maybe robustness to harms, which is less Bayesian as per your post title → choose robustly good actions, maintain option value, build capacity etc. Which is basically what most meta interventions are about, no?
Unfortunately I don’t think there’s really much of a case for “maintain option value, build capacity etc.” being robustly good either, as argued here.
Thanks Anthony. Would it be fair to interpret your unawareness series as your steelman of OP’s post title, or as being relevant?
I couldn’t find on a quick look what decision-making approach you would (at least provisionally) endorse instead, bracketing maybe? For my own reference later:
I would be particularly interested in how you think meta- and/or longtermist-oriented grantmaking could be improved by your work. I have not been very impressed by the reasoning behind some of these (sometimes quite large) grants, at least on the rare occasions they’ve been shared publicly.
I was making in-paradigm critiques, since, if something is internally invalid, it gives us good reason to think it is also externally invalid.
I linked to the XPT tournament just as an example of asking non-experts for predictions. I thought that pandemics were a good example since most people don’t know much about pandemics at all so we should expect their guesses to be very off. I could see the argument that forecasting research has found forecasters to be better than experts so it makes sense to ask non-experts, but it’s important to note that we haven’t validated this over very long time periods.
I linked to the GPI paper since it seems to pretty absurd to me to try to guess how many future there will be. Even if we can come up with accurate estimates for carrying capacity for different regions of space, I have no clue how we could predict the likelihood of reaching carrying capacity in these regions of space.
In regards to your last point, I think my view is most basically that, by assigning probabilities to outcomes, we’re giving ourselves excessive confidence when we often have too little knowledge to warrant the confidence, that correct decision theories should take into account our cluelessness to a much greater extent, and that people probably do Bayesian reasoning a lot worse than they think.
That said, I’m still figuring out my views on decisions theories so I probably should have researched this topic a lot more before making a post.
For your last point, I agree with you that we run into the problem of EV-maxxing being undefined, but I don’t know where to go from there. It doesn’t seem to make sense to me to do anything in regards to something with such a wide probability range because it seems like you’ll just spend all your time chasing things that you know very little about but which suggest really high EV.
Thanks for the thoughtful comment.