Yes, but itâs very hard to attack any particular prior as well.
I donât think this leaves you in a good position if your estimates and rankings are very sensitive to the choice of âreasonableâ priors. Chris illustrated this in his post at the end of part 2 (with the atheist example), and in part 3.
You could try to choose some compromise between these priors, but there are multiple âreasonableâ ways to compromise. You could introduce a prior on these priors, but you could run into the same problem with multiple âreasonableâ choices for this new prior.
I donât think this leaves you in a good position if your estimates and rankings are very sensitive to the choice of âreasonableâ priors.
What do you mean by âa good positionâ?
You could try to choose some compromise between these priors, but there are multiple âreasonableâ ways to compromise. You could introduce a prior on these priors, but you could run into the same problem with multiple âreasonableâ choices for this new prior.
Ah, I guess weâll have to switch to a system of epistemology which doesnât bottom out in unproven assumptions. Hey hold on a minute, there is none.
Iâm getting a little confused about what sorts of concrete conclusions we are supposed to take away from here.
Iâm getting a little confused about what sorts of concrete conclusions we are supposed to take away from here.
Iâm not saying we shouldnât use priors or that theyâll never help. What I am saying is that they donât address the optimizerâs curse just by including them, and I suspect they wonât help at all on their own in some cases.
Maybe checking sensitivity to priors and further promoting interventions whose value depends less on them (among some set of âreasonableâ priors) would help. You could see this as a special case of Chrisâs suggestion to âEntertain multiple modelsâ.
Perhaps you could even use an explicit model to combine the estimates or posteriors from multiple models into a single one in a way that either penalizes sensitivity to priors or gives less weight to more extreme estimates, but a simpler decision rule might be more transparent or otherwise preferable. From my understanding, GiveWell already uses medians of its analystsâ estimates this way.
Ah, I guess weâll have to switch to a system of epistemology which doesnât bottom out in unproven assumptions. Hey hold on a minute, there is none.
What I am saying is that they donât address the optimizerâs curse just by including them, and I suspect they wonât help at all on their own in some cases.
You seem to be using âpeople all agreeâ as a stand-in for âthe optimizerâs curse has been addressedâ. I donât get this. Addressing the optimizerâs curse has been mathematically demonstrated. Different people can disagree about the specific inputs, so people will disagree, but that doesnât mean they havenât addressed the optimizerâs curse.
Maybe checking sensitivity to priors and further promoting interventions whose value depends less on them (among some set of âreasonableâ priors) would help. You could see this as a special case of Chrisâs suggestion to âEntertain multiple modelsâ.
Perhaps you could even use an explicit model to combine the estimates or posteriors from multiple models into a single one in a way that either penalizes sensitivity to priors or gives less weight to more extreme estimates, but a simpler decision rule might be more transparent or otherwise preferable.
I think combining into a single model is generally appropriate. And the sub-models need not be fully, explicitly laid out.
Suppose Iâm demonstrating that poverty charity > animal charity. I donât have to build one model assuming â1 human = 50 chickensâ, another model assuming â1 human = 100 chickensâ, and so on.
Instead I just set a general standard for how robust my claims are going to be, and I feel sufficiently confident saying â1 human = at least 60 chickensâ, so I use that rather than my mean expectation (e.g. 90).
You seem to be using âpeople all agreeâ as a stand-in for âthe optimizerâs curse has been addressedâ. I donât get this. Addressing the optimizerâs curse has been mathematically demonstrated. Different people can disagree about the specific inputs, so people will disagree, but that doesnât mean they havenât addressed the optimizerâs curse.
Maybe weâre thinking about the optimizerâs curse in different ways.
The proposed solution of using priors just pushes the problem to selecting good priors. Itâs also only a solution in the sense that it reduces the likelihood of mistakes happening (discovered in hindsight, and under the assumption of good priors), but not provably to its minimum, since it does not eliminate the impacts of noise. (I donât think thereâs any complete solution to the optimizerâs curse, since, as long as estimates are at least somewhat sensitive to noise, âluckyâ estimates will tend to be favoured, and you canât tell in principle between âluckyâ and âbetterâ interventions.)
If youâre presented with multiple priors, and they all seem similarly reasonable to you, but depending on which ones you choose, different actions will be favoured, how would you choose how to act? Itâs not just a matter of different people disagreeing on priors, itâs also a matter of committing to particular priors in the first place.
If one action is preferred with almost all of the priors (perhaps rare in practice), isnât that a reason (perhaps insufficient) to prefer it? To me, using this could be an improvement over just using priors, because I suspect it will further reduce the impacts of noise, and if it is an improvement, then just using priors never fully solved the problem in practice in the first place.
I agree with the rest of your comment. I think something like that would be useful.
The proposed solution of using priors just pushes the problem to selecting good priors.
The problem of the optimizerâs curse is that the EV estimates of high-EV-options are predictably over-optimistic in proportion with how unreliable the estimates are. That problem doesnât exist anymore.
The fact that you donât have guaranteed accurate information doesnât mean the optimizerâs curse still exists.
I donât think thereâs any complete solution to the optimizerâs curse
Well there is, just spend too much time worrying about model uncertainty and other peopleâs priors and too little time worrying about expected value estimation. Then youâre solving the optimizerâs curse too much, so that your charity selections will be less accurate and predictably biased in favor of low EV, high reliability options. So itâs a bad idea, but youâve solved the optimizerâs curse.
If youâre presented with multiple priors, and they all seem similarly reasonable to you, but depending on which ones you choose, different actions will be favoured, how would you choose how to act?
Maximize the expected outcome over the distribution of possibilities.
If one action is preferred with almost all of the priors (perhaps rare in practice), isnât that a reason (perhaps insufficient) to prefer it?
What do you mean by âthe priorsâ? Other peopleâs priors? Well if theyâre other peopleâs priors and I donât have reason to update my beliefs based on their priors, then itâs trivially true that this doesnât give me a reason to prefer the action. But you seem to think that other peopleâs priors will be âreasonableâ, so obviously I should update based on their priors, in which case of course this is trueâbut only in a banal, trivial sense that has nothing to do with the optimizerâs curse.
To me, using this could be an improvement over just using priors
Hm? Youâre just suggesting updating oneâs prior by looking at other peopleâs priors. Assuming that other peopleâs priors might be rational, this is banalâof course we should be reasonable, epistemically modest, etc. But this has nothing to do with the optimizerâs curse in particular, itâs equally true either way.
I ask the same question I asked of OP: give me some guidance that applies for estimating the impact of maximizing actions that doesnât apply for estimating the impact of randomly selected actions. So far it still seems like there is noneâaside from the basic idea given by Muelhauser.
just using priors never fully solved the problem in practice in the first place
Is the problem the lack of guaranteed knowledge about charity impacts, or is the problem the optimizerâs curse? You seem to (incorrectly) think that chipping away at the former necessarily means chipping away at the latter.
Itâs always worth entertaining multiple models if you can do that at no cost. However, doing that often comes at some cost (money, time, etc). In situations with lots of uncertainty (where the optimizerâs curse is liable to cause significant problems), itâs worth paying much higher costs to entertain multiple models (or do other things I suggested) than it is in cases where the optimizerâs curse is unlikely to cause serious problems.
In situations with lots of uncertainty (where the optimizerâs curse is liable to cause significant problems), itâs worth paying much higher costs to entertain multiple models (or do other things I suggested) than it is in cases where the optimizerâs curse is unlikely to cause serious problems.
I donât agree. Why is the uncertainty that comes from model uncertaintyâas opposed to any other kind of uncertaintyâuniquely important for the optimizerâs curse? The optimizerâs curse does not discriminate between estimates that are too high for modeling reasons, versus estimates that are too high for any other reason.
The mere fact that thereâs more uncertainty is not relevant, because we are talking about how much time we should spend worrying about one kind of uncertainty versus another. âDo more to reduce uncertaintyâ is just a platitude, we always want to reduce uncertainty.
I made a long top-level comment that I hope will clarify some problems with the solution proposed in the original paper.
I ask the same question I asked of OP: give me some guidance that applies for estimating the impact of maximizing actions that doesnât apply for estimating the impact of randomly selected actions.
This is a good point. Somehow, I think youâd want to adjust your posterior downward based on the set or the number of options under consideration and how unlikely the data that makes the intervention look good. This is not really useful, since I donât know how much you should adjust these. Maybe thereâs a way to model this explicitly, but it seems like youâd be trying to model your selection process itself before youâve defined it, and then you look for a selection process which satisfies some properties.
You might also want to spend more effort looking for arguments and evidence against each option the more options youâre considering.
When considering a larger number of options, you could use some randomness in your selection process or spread funding further (although the latter will be vulnerable to the satisficerâs curse if youâre using cutoffs).
What do you mean by âthe priorsâ?
If I havenât decided on a prior, and multiple different priors (even an infinite set of them) seem equally reasonable to me.
Somehow, I think youâd want to adjust your posterior downward based on the set or the number of options under consideration and how unlikely the data that makes the intervention look good.
Thatâs the basic idea given by Muelhauser. Corrected posterior EV estimates.
You might also want to spend more effort looking for arguments and evidence against each option the more options youâre considering.
As opposed to equal effort for and against? OK, Iâm satisfied. However, if Iâve done the corrected posterior EV estimation, and then my specific search for arguments-against turns up short, then I should increase my EV estimates back towards the original naive estimate.
As I recall, that post found that randomized funding doesnât make sense. Which 100% matches my presumptions, I do not see how it could improve funding outcomes.
or spread funding further
I donât see how that would improve funding outcomes.
If I havenât decided on a prior, and multiple different priors (even an infinite set of them) seem equally reasonable to me.
In Bayesian rationality, you always have a prior. You seem to be considering or defining things differently.
Here we would probably say that your actual prior exists and is simply some kind of aggregate of these possible priors, therefore itâs not the case that we should leap outside our own priors in some sort of violation of standard Bayesian rationality.
The proposed solution of using priors just pushes the problem to selecting good priors.
+1
In conversations Iâve had about this stuff, it seems like the crux is often the question of how easy it is to choose good priors, and whether a âgoodâ prior is even an intelligible concept.
Compare Chrisâ piece (âselecting good priors is really hard!â) with this piece by Luke Muehlhauser (âthe optimizerâs curse is trivial, just choose an appropriate prior!â)
it seems like the crux is often the question of how easy it is to choose good priors
Before anything like a crux can be identified, complainants need to identify what a âgood priorâ even means, or what strategies are better than others. Until then, theyâre not even wrongâitâs not even possible to say what disagreement exists. To airily talk about âgood priorsâ or âbad priorsâ, being âeasyâ or âhardâ to identify, is just empty phrasing and suggests confusion about rationality and probability.
Hey Kyle, Iâd stopped responding since I felt like we were well beyond the point where we were likely to convince one another or say things that those reading the comments would find insightful.
I understand why you think âgood priorâ needs to be defined better.
As I try to communicate (but may not quite say explicitly) in my post, I think that in situations where uncertainty is poorly understood, itâs hard to come up with priors that are good enough that choosing actions based explicit Bayesian calculations will lead to better outcomes than choosing actions based on a combination of careful skepticism, information gathering, hunches, and critical thinking.
Venture capitalists frequently fund things that theyâre extremely uncertain about. Itâs my impression that Bayesian calculations rarely play into these situations. Instead, smart VCs think hard and critically and come to conclusions based on processes that they probably donât full understand themselves.
It could be that VCs have just failed to realize the amazingness of Bayesianism. However, given that theyâre smart & thereâs a ton of money on the table, I think the much more plausible explanation is that hardcore Bayesianism wouldnât lead to better results than whatever it is that successful VCs actually do.
Again, none of this is to say that Bayesianism is fundamentally broken or that high-level Bayesian-ish things like âI have a very skeptical prior so I should not take this estimate of impact at face valueâ are crazy.
Venture capitalists frequently fund things that theyâre extremely uncertain about. Itâs my impression that Bayesian calculations rarely play into these situations. Instead, smart VCs think hard and critically and come to conclusions based on processes that they probably donât full understand themselves.
I interned for a VC, albeit a small and unknown one. Sure, they donât do Bayesian calculations, if you want to be really precise. But they make extensive use of quantitative estimates all the same. If anything, they are cruder than what EAs do. As far as I know, they donât bother correcting for the optimizerâs curse! I never heard it mentioned. VCs donât primarily rely on the quantitative models, but other areas of finance do. If what they do is OK, then what EAs do is better. This is consistent with what finance professionals told me about the financial modeling that I did.
Plus, this is not about the optimizerâs curse. Imagine that you told those VCs that they were no longer choosing which startups are best, instead they now have to select which ones are better-than-average and which ones are worse-than-average. The optimizerâs curse will no longer interfere. Yet theyâre not going to start relying more on explicit Bayesian calculations. Theyâre going to use the same way of thinking as always.
And explicit Bayesian calculation is rarely used by anyone anywhere. Humans encounter many problems which are not about optimizing, and they still donât use explicit Bayesian calculation. So clearly the optimizerâs curse is not the issue. Instead, itâs a matter of which kinds of cognition and calculation people are more or less comfortable with.
itâs hard to come up with priors that are good enough that choosing actions based explicit Bayesian calculations will lead to better outcomes than choosing actions based on a combination of careful skepticism, information gathering, hunches, and critical thinking.
Explicit Bayesian calculation is a way of choosing actions based on a combination of careful skepticism, information gathering, hunches, and critical thinking. (With math too.)
Iâm guessing you mean we should use intuition for the final selection, instead of quantitative estimates. OK, but I donât see how the original post is supposed to back it up; I donât see what the optimizerâs curse has to do with it.
I donât think this leaves you in a good position if your estimates and rankings are very sensitive to the choice of âreasonableâ priors. Chris illustrated this in his post at the end of part 2 (with the atheist example), and in part 3.
You could try to choose some compromise between these priors, but there are multiple âreasonableâ ways to compromise. You could introduce a prior on these priors, but you could run into the same problem with multiple âreasonableâ choices for this new prior.
What do you mean by âa good positionâ?
Ah, I guess weâll have to switch to a system of epistemology which doesnât bottom out in unproven assumptions. Hey hold on a minute, there is none.
Iâm getting a little confused about what sorts of concrete conclusions we are supposed to take away from here.
Iâm not saying we shouldnât use priors or that theyâll never help. What I am saying is that they donât address the optimizerâs curse just by including them, and I suspect they wonât help at all on their own in some cases.
Maybe checking sensitivity to priors and further promoting interventions whose value depends less on them (among some set of âreasonableâ priors) would help. You could see this as a special case of Chrisâs suggestion to âEntertain multiple modelsâ.
Perhaps you could even use an explicit model to combine the estimates or posteriors from multiple models into a single one in a way that either penalizes sensitivity to priors or gives less weight to more extreme estimates, but a simpler decision rule might be more transparent or otherwise preferable. From my understanding, GiveWell already uses medians of its analystsâ estimates this way.
I get your point, but the snark isnât helpful.
You seem to be using âpeople all agreeâ as a stand-in for âthe optimizerâs curse has been addressedâ. I donât get this. Addressing the optimizerâs curse has been mathematically demonstrated. Different people can disagree about the specific inputs, so people will disagree, but that doesnât mean they havenât addressed the optimizerâs curse.
I think combining into a single model is generally appropriate. And the sub-models need not be fully, explicitly laid out.
Suppose Iâm demonstrating that poverty charity > animal charity. I donât have to build one model assuming â1 human = 50 chickensâ, another model assuming â1 human = 100 chickensâ, and so on.
Instead I just set a general standard for how robust my claims are going to be, and I feel sufficiently confident saying â1 human = at least 60 chickensâ, so I use that rather than my mean expectation (e.g. 90).
Maybe weâre thinking about the optimizerâs curse in different ways.
The proposed solution of using priors just pushes the problem to selecting good priors. Itâs also only a solution in the sense that it reduces the likelihood of mistakes happening (discovered in hindsight, and under the assumption of good priors), but not provably to its minimum, since it does not eliminate the impacts of noise. (I donât think thereâs any complete solution to the optimizerâs curse, since, as long as estimates are at least somewhat sensitive to noise, âluckyâ estimates will tend to be favoured, and you canât tell in principle between âluckyâ and âbetterâ interventions.)
If youâre presented with multiple priors, and they all seem similarly reasonable to you, but depending on which ones you choose, different actions will be favoured, how would you choose how to act? Itâs not just a matter of different people disagreeing on priors, itâs also a matter of committing to particular priors in the first place.
If one action is preferred with almost all of the priors (perhaps rare in practice), isnât that a reason (perhaps insufficient) to prefer it? To me, using this could be an improvement over just using priors, because I suspect it will further reduce the impacts of noise, and if it is an improvement, then just using priors never fully solved the problem in practice in the first place.
I agree with the rest of your comment. I think something like that would be useful.
The problem of the optimizerâs curse is that the EV estimates of high-EV-options are predictably over-optimistic in proportion with how unreliable the estimates are. That problem doesnât exist anymore.
The fact that you donât have guaranteed accurate information doesnât mean the optimizerâs curse still exists.
Well there is, just spend too much time worrying about model uncertainty and other peopleâs priors and too little time worrying about expected value estimation. Then youâre solving the optimizerâs curse too much, so that your charity selections will be less accurate and predictably biased in favor of low EV, high reliability options. So itâs a bad idea, but youâve solved the optimizerâs curse.
Maximize the expected outcome over the distribution of possibilities.
What do you mean by âthe priorsâ? Other peopleâs priors? Well if theyâre other peopleâs priors and I donât have reason to update my beliefs based on their priors, then itâs trivially true that this doesnât give me a reason to prefer the action. But you seem to think that other peopleâs priors will be âreasonableâ, so obviously I should update based on their priors, in which case of course this is trueâbut only in a banal, trivial sense that has nothing to do with the optimizerâs curse.
Hm? Youâre just suggesting updating oneâs prior by looking at other peopleâs priors. Assuming that other peopleâs priors might be rational, this is banalâof course we should be reasonable, epistemically modest, etc. But this has nothing to do with the optimizerâs curse in particular, itâs equally true either way.
I ask the same question I asked of OP: give me some guidance that applies for estimating the impact of maximizing actions that doesnât apply for estimating the impact of randomly selected actions. So far it still seems like there is noneâaside from the basic idea given by Muelhauser.
Is the problem the lack of guaranteed knowledge about charity impacts, or is the problem the optimizerâs curse? You seem to (incorrectly) think that chipping away at the former necessarily means chipping away at the latter.
Itâs always worth entertaining multiple models if you can do that at no cost. However, doing that often comes at some cost (money, time, etc). In situations with lots of uncertainty (where the optimizerâs curse is liable to cause significant problems), itâs worth paying much higher costs to entertain multiple models (or do other things I suggested) than it is in cases where the optimizerâs curse is unlikely to cause serious problems.
I donât agree. Why is the uncertainty that comes from model uncertaintyâas opposed to any other kind of uncertaintyâuniquely important for the optimizerâs curse? The optimizerâs curse does not discriminate between estimates that are too high for modeling reasons, versus estimates that are too high for any other reason.
The mere fact that thereâs more uncertainty is not relevant, because we are talking about how much time we should spend worrying about one kind of uncertainty versus another. âDo more to reduce uncertaintyâ is just a platitude, we always want to reduce uncertainty.
I made a long top-level comment that I hope will clarify some problems with the solution proposed in the original paper.
This is a good point. Somehow, I think youâd want to adjust your posterior downward based on the set or the number of options under consideration and how unlikely the data that makes the intervention look good. This is not really useful, since I donât know how much you should adjust these. Maybe thereâs a way to model this explicitly, but it seems like youâd be trying to model your selection process itself before youâve defined it, and then you look for a selection process which satisfies some properties.
You might also want to spend more effort looking for arguments and evidence against each option the more options youâre considering.
When considering a larger number of options, you could use some randomness in your selection process or spread funding further (although the latter will be vulnerable to the satisficerâs curse if youâre using cutoffs).
If I havenât decided on a prior, and multiple different priors (even an infinite set of them) seem equally reasonable to me.
Thatâs the basic idea given by Muelhauser. Corrected posterior EV estimates.
As opposed to equal effort for and against? OK, Iâm satisfied. However, if Iâve done the corrected posterior EV estimation, and then my specific search for arguments-against turns up short, then I should increase my EV estimates back towards the original naive estimate.
As I recall, that post found that randomized funding doesnât make sense. Which 100% matches my presumptions, I do not see how it could improve funding outcomes.
I donât see how that would improve funding outcomes.
In Bayesian rationality, you always have a prior. You seem to be considering or defining things differently.
Here we would probably say that your actual prior exists and is simply some kind of aggregate of these possible priors, therefore itâs not the case that we should leap outside our own priors in some sort of violation of standard Bayesian rationality.
+1
In conversations Iâve had about this stuff, it seems like the crux is often the question of how easy it is to choose good priors, and whether a âgoodâ prior is even an intelligible concept.
Compare Chrisâ piece (âselecting good priors is really hard!â) with this piece by Luke Muehlhauser (âthe optimizerâs curse is trivial, just choose an appropriate prior!â)
Before anything like a crux can be identified, complainants need to identify what a âgood priorâ even means, or what strategies are better than others. Until then, theyâre not even wrongâitâs not even possible to say what disagreement exists. To airily talk about âgood priorsâ or âbad priorsâ, being âeasyâ or âhardâ to identify, is just empty phrasing and suggests confusion about rationality and probability.
Hey Kyle, Iâd stopped responding since I felt like we were well beyond the point where we were likely to convince one another or say things that those reading the comments would find insightful.
I understand why you think âgood priorâ needs to be defined better.
As I try to communicate (but may not quite say explicitly) in my post, I think that in situations where uncertainty is poorly understood, itâs hard to come up with priors that are good enough that choosing actions based explicit Bayesian calculations will lead to better outcomes than choosing actions based on a combination of careful skepticism, information gathering, hunches, and critical thinking.
As a real world example:
Venture capitalists frequently fund things that theyâre extremely uncertain about. Itâs my impression that Bayesian calculations rarely play into these situations. Instead, smart VCs think hard and critically and come to conclusions based on processes that they probably donât full understand themselves.
It could be that VCs have just failed to realize the amazingness of Bayesianism. However, given that theyâre smart & thereâs a ton of money on the table, I think the much more plausible explanation is that hardcore Bayesianism wouldnât lead to better results than whatever it is that successful VCs actually do.
Again, none of this is to say that Bayesianism is fundamentally broken or that high-level Bayesian-ish things like âI have a very skeptical prior so I should not take this estimate of impact at face valueâ are crazy.
I interned for a VC, albeit a small and unknown one. Sure, they donât do Bayesian calculations, if you want to be really precise. But they make extensive use of quantitative estimates all the same. If anything, they are cruder than what EAs do. As far as I know, they donât bother correcting for the optimizerâs curse! I never heard it mentioned. VCs donât primarily rely on the quantitative models, but other areas of finance do. If what they do is OK, then what EAs do is better. This is consistent with what finance professionals told me about the financial modeling that I did.
Plus, this is not about the optimizerâs curse. Imagine that you told those VCs that they were no longer choosing which startups are best, instead they now have to select which ones are better-than-average and which ones are worse-than-average. The optimizerâs curse will no longer interfere. Yet theyâre not going to start relying more on explicit Bayesian calculations. Theyâre going to use the same way of thinking as always.
And explicit Bayesian calculation is rarely used by anyone anywhere. Humans encounter many problems which are not about optimizing, and they still donât use explicit Bayesian calculation. So clearly the optimizerâs curse is not the issue. Instead, itâs a matter of which kinds of cognition and calculation people are more or less comfortable with.
Explicit Bayesian calculation is a way of choosing actions based on a combination of careful skepticism, information gathering, hunches, and critical thinking. (With math too.)
Iâm guessing you mean we should use intuition for the final selection, instead of quantitative estimates. OK, but I donât see how the original post is supposed to back it up; I donât see what the optimizerâs curse has to do with it.