Interesting! Thank you for writing this, this is something I was also wondering about while reading for the Warwick EA fellowship. My intuition is also that in the case of a “many-membered set of probability functions”, I’d define a prior over those and then compute an expected value as if nothing happened. I acknowledge that there is substantial (or even overwhelming) uncertainty sometimes and I can understand the impulse of wanting a separate conceptual handle for that. But it’s still “decision making under uncertainty” and should therefore be subsumable under Bayesianism.
I feel similar to ben.smith that I might be completely missing something. But I also wonder if this confusion might just be an echo of the age-old Bayesianism vs Frequentism debate, where people have different intuition about whether priors over probability distributions are a-ok.
There is an argument from intuition that carry some force by Schoenfield (2012) that we can’t use a probability function:
(1) It is permissible to be insensitive to mild evidential sweetening. (2) If we are insensitive to mild evidential sweetening, our attitudes cannot be represented by a probability function. (3) It is permissible to have attitudes that are not representable by a probability function. (1, 2)
...
You are a confused detective trying to figure out whether Smith or Jones committed the crime. You have an enormous body of evidence that to evaluate. Here is some of it: You know that 68 out of the 103 eyewitnesses claim that Smith did it but Jones’ footprints were found at the crime scene. Smith has an alibi, and Jones doesn’t. But Jones has a clear record while Smith has committed crimes in the past. The gun that killed the victim belonged to Smith. But the lie detector, which is accurate 71% percent of time, suggests that Jones did it. After you have gotten all of this evidence, you have no idea who committed the crime. You are no more confident that Jones committed the crime than that Smith committed the crime, nor are you more confident that Smith committed the crime than that Jones committed the crime.
...
Now imagine that, after considering all of this evidence, you learn a new fact: it turns out that there were actually 69 eyewitnesses (rather than 68) testifying that Smith did it. Does this make it the case that you should now be more confident in S than J? That, if you had to choose right now who to send to jail, it should be Smith? I think not.
...
In our case, you are insensitive to evidential sweetening with respect to S since you are no more confident in S than ~S (i.e. J), and no more confident in ~S (i.e. J) than S. The extra eyewitness supports S more than it supports ~S, and yet despite learning about the extra eyewitness, you are no more confident in S than you are in ~S (i.e. J).
Intuitively, this sounds right. And if you went from this problem trying to understand solve the crime on intuition, you might really have no idea. Reading the passage, it sounds mind-boggling.
On the other hand, if you applied some reasoning and study, you might be able to come up with some probability estimates. You could identify the conditioning of P(Smith did it|an eyewitness says Smith did it), including a probability distribution on that probability itself, if you like. You can identify how to combine evidence from multiple witnesses, i.e., P(Smith did it|eyewitness 1 says Smith did it) & P(Smith did it|eyewitness 2 says Smith did it), and so on up to 68 and 69. You can estimate the independence of eyewitnesses, and from that work out how to properly combine evidence from multiple eyewitnesses.
And it might turn out that you don’t update as a result of the extra eyewitness, under some circumstances. Perhaps you know the eyewitnesses aren’t independent; they’re all card-carrying members of the “We hate Smith” club. In that case it simply turns out that the extra eye-witness is irrelevant to the problem; it doesn’t qualify as evidence, so it it doesn’t mean you’re insensitive to “mild evidential sweetening”.
I think a lot of the problem here is that these authors are discussing what one could do when one sits down for the first time and tries to grapple with a problem. In those cases there’s so many undefined features of the problem that it really does seem impossible and you really are clueless.
But that’s not the same as saying that, with sufficient time, you can’t put probability distributions to everything that’s relevant and try to work out the joint probability.
----
Schoenfield, M. Chilling out on epistemic rationality. Philos Stud158, 197–219 (2012).
Now imagine that, after considering all of this evidence, you learn a new fact: it turns out that there were actually 69 eyewitnesses (rather than 68) testifying that Smith did it. Does this make it the case that you should now be more confident in S than J? That, if you had to choose right now who to send to jail, it should be Smith? I think not.
One should update towards a higher chance of Smith having commited the crime. However, if one was around 50 % confident that Smith commited the crime before the update, an update much smaller than 50 pp will still leave one around 50 % confident that Smith commited the crime. However, the best guess for the probability that Smitth commited the crime should still go up as a result of the update. If the contribution of an additional eyewitness feels completely irrelevant, one could subjectively estimate the update from “update for N additional eyewitnesses”/N. This will not feel completely irrelevant for a sufficiently large N, unless one considers all eyewitnesses testifying that Smith commited the crime no evidence at all.
But that’s not the same as saying that, with sufficient time, you can’t put probability distributions to everything that’s relevant and try to work out the joint probability.
Some questions here are whether 50-50 as precise probabilities to start is reasonable and whether the approach to assign 50-50 as precise probabilities is reasonable.
If, when looking at the scenario, you would have done something like “wow, that’s so complicated and I’m clueless, so 50-50”, then your reaction almost certainly would have been the same if the example originally included one extra eyewitness in favour of one side. But then this tells you your initial way to assign credences was insensitive to this small difference. And yet after the initial assignment, you say it should be sensitive.
Or, if you forgot your initial judgement or the number of eyewitnesses and was just given the total and looked at the situation with fresh eyes, you’d come up with 50-50 again.
Alternatively, you could build a precise probability distribution as a function of the evidence that weighs it all, but this would be very sensitive to arbitrary choices.
I could report 50 % for 68 and 69 eyewitnesses, but this does not necessarily imply I am insensitive to small changes in the number of eyewitnesses. In practice, I would be reporting my best guess rounded to the closest multiple of 0.1 or so. So I believe the reported value being exactly the same would only mean my best guesses differ by less than 10 pp, not that they are exactly the same. I would say the mean of the (rounded) reported best guesses for a given number of eyewitnesses tends to the (precise) underlying best guess as the number of reports increases. If I could hypothetically encounter the question in practically the same situation for 1 M times, I could easily see the mean of my reported values for 68 and 69 eyewitnesses being different.
If I asked you to actually decide who’s more likely to be the culprit, how would you do it?
What do you do if you don’t have reference class information for each part of the problem? How do you weigh the conflicting evidence? I’m imaginging that at many steps, you’d have to rely on direct impressions or numbers that just came to mind.
Would you feel like whatever came out was very arbitrary and depended too much on direct impressions or numbers that just came to mind? Would you actually believe and endorse what came out? Would you defend it to other people?
What I would actually do depends a lot on the situation, but I have a hard time imagining scenarios where it matters whether the probability of Jones having commited the crime is 40 % or 60 %. So I might not even try to decrease the uncertainty about this, and just focus on other considerations. What would maximise the impact of my future donations and work? What information would I have about Jones and Smith? Who would have the greater potential to contribute to a better world? How much time would I have to decide? Would I be accountable in some way for my decision? If so, how would my decision be assessed? What would be the potential consequences of people concluding I made a good or bad decision? How were decisions like mine assessed in the past?
Do you (Michael) see your views about precise and imprecise credences significantly affecting what you would actually do in the real world in a scenario where you had to blame Jones or Smith? Considerations like the ones I mentioned above would matter mode? I may be dodging your question, but I am ultimately interested in making better decisions in the real world. So I think it makes sense to discuss precise and imprecise credences in the context of realistic scenarios.
Do you (Michael) see your views about precise and imprecise credences significantly affecting what you would actually do in the real world in a scenario where you had to blame Jones or Smith?
Probably not. I see it as more illustrative of important cases. Imagine instead it’s between supporting an intervention or not, and it has similar complexity and considerations going in each direction.
More relevant examples to us could be: crops vs nature for wild animals, climate change on wild animals, fishing on wild animals, the far future effects of our actions, the acausal influence of our actions. These are all things I feel clueless enough about to mostly bracket away and ignore when they are side effects of direct interventions I’m interested in supporting. I’m not ignoring them because I think they’re small. I think they are likely much larger than the effects I’m not ignoring.
I may also want to further study some of them, but I’m often not that optimistic about making much progress (especially for far future effrcts and acausal influence) and for that progress to be used in a way that isn’t net negative overall by my lights.
How much more optimistic would you be about research on i) the welfare of soil animals and microorganisms, and ii) comparisons of (expectedhedonistic) welfare across species if you strongly endorsed expectational total hedonistic utilitarianism, moral realism, and precise probabilitites, and ignored acausal effects, and effects after 100 years?
While browsing types of uncertainties, I stumbled upon the idea of state space uncertainty and conscious unawareness, which sounds similar to your explanation of cluelessness and which might be another helpful angle for people with a more Bayesian perspective.
There are, in the real world, unforeseen contingencies: eventualities that even the educated decision maker will fail to foresee. For instance, the recent tsunami and subsequent nuclear meltdown in Japan are events that most agents would have omitted from their decision models. If a decision maker is aware of the possibility that they may not be aware of all relevant contingencies—a state that Walker and Dietz (2011) call ‘conscious unawareness’ —then they face state space uncertainty.
There are things you can do to correct for this sort of thing-for instance, go one level more meta, estimate the probability of unforeseen consequences in general, or within the class of problems that your specific problem fits into.
We couldn’t have predicted the fukushima disaster, but perhaps we can predict related things with some degree of certainty—the average cost and death toll of earthquakes worldwide, for instance. In fact, this is a fairly well explored space, since insurers have to understand the risk of earthquakes.
The ongoing pandemic is a harder example—the rarer the black swan, the more difficult it is to predict. But even then, prior to the 2020 pandemic, the WHO had estimated the amortized costs of pandemics as in the order of 1% of global GDP annually (averaged over years when there are and aren’t pandemics), which seems like a reasonable approximation.
I don’t know how much of a realistic solution that would be in practice.
I think the example Ben cites in his reply is very illustrative.
You might feel that you can’t justify your one specific choice of prior over another prior, so that particular choice is arbitrary, and then what you should do could depend on this arbitrary choice, whereas an equally reasonable prior would recommend a different decision. Someone else could have exactly the same information as you, but due to a different psychology, or just different patterns of neurons firing, come up with a different prior that ends up recommending a different decision. Choosing one prior over another without reason seems like a whim or a bias, and potentially especially prone to systematic error.
It seems bad if we’re basing how to do the most good on whims and biases.
If you’re lucky enough to have only finitely many equally reasonable priors, then I think it does make sense to just use a uniform meta-prior over them, i.e. just take their average. This doesn’t seem to work with infinitely many priors, since you could use different parametrizations to represent the same continuous family of distributions, with a different uniform distribution and therefore average for each parametrization. You’d have to justify your choice of parametrization!
As another example, imagine you have a coin that someone (who is trustworthy) has told you is biased towards heads, but they haven’t given you any hint how much, and you want to come up with a probability distribution for the fraction of heads over 1,000,000 flips. So, you want a distribution over the interval [0, 1]. Which distribution would you use? Say you give me a probability density function f. Why not (1−p)f(x)+p for some p∈(0,1)? Why not 1∫10f(xp)dxf(xp) for some p>0? If f is a weighted average of multiple distributions, why not apply one of these transformations to one of the component distributions and choose the resulting weighted average instead? Why the particular weights you’ve chosen and not slightly different ones?
Which distribution would you use? Why the particular weights you’ve chosen and not slightly different ones?
I think you just have to make your distribution uninformative enough that reasonable differences in the weights don’t change your overall conclusion. If they do, then I would concede that the solution to your specific question really is clueless. Otherwise, you can probably find a response.
come up with a probability distribution for the fraction of heads over 1,000,000 flips.
Rather than thinking of directly of appropriate distribution for the 1,000,000 flips, I’d think of a distribution to model p itself. Then you can run simulations based on the distribution of p to calculate the distribution of the fraction of 1000,000 flips. p∈(0.5,1.0], and then we need to select a distribution for p over that range.
There is no one correct probability distribution for p because any probability is just an expression of our belief, so you may use whatever probability distribution genuinely reflects your prior belief. A uniform distribution is a reasonable start. Perhaps you really are clueless about p, in which case, yes, there’s a certain amount of subjectivity about your choice. But prior beliefs are always inherently subjective, because they simply describe your belief about the state of the world as you know it now. The fact you might have to select a distribution, or set of distributions with some weighted average, is merely an expression of your uncertainty. This in itself, I think, doesn’t stop you from trying to estimate the result.
I think this expresses within Bayesian terms the philosophical idea that we can only make moral choices based on information available at the time; one can’t be held morally responsible for mistakes made on the basis of the information we didn’t have.
Perhaps you disagree with me that a uniform distribution is the best choice. You reason thus: “we have some idea about the properties of coins in general. It’s difficult to make a coin that is 100% biased towards heads. So that seems unlikely”. So we could pick a distribution that better reflects your prior belief. Perhaps a suitable choice might be Beta(2,2) with a truncation at 0.5, which will give the greatest likelihood of p just above 0.5, and a declining likelihood down to 1.0.
Maybe you and i just can’t agree after all that there is still no consistent and reasonable prior choice you can make, and not even any compromise. And let’s say we both run simulations using our own priors and find entirely different results and we can’t agree on any suitable weighting between them. In that case, yes, I can see you have cluelessness. I don’t think it follows that, if we went through the same process for estimating the longtermist moral worth of malaria bednet distribution, we must have intractable complex cluelessness about specific problems like malaria bednet distribution. I think I can admit that perhaps, right now, in our current belief state, we are genuinely clueless, but it seems that there is some work that can be done that might eliminate the cluelessness.
It seems bad if we’re basing how to do the most good on whims and biases.
I agree. However, in cases where priors are playing a crucial role, one should simply prioritise gathering more evidence until there is reasonable convergence about what to do (among a given group of people, for a particular decision)?
In some cases, we can’t gather strong enough evidence, say because:
they’re questions about very speculative or unprecedented possibilities and the evidence would either be too indirect and weak or come too late to be very action-guiding, e.g. often for AI risk, conscious subsystems, or
there will be too much noise or confounding, too small a sample size and anything like an RCT is too impractical (e.g. policy, corporate outreach) or wouldn’t generalize well, or
the disagreements are partly conceptual, definitional or philosophical, e.g. “What is consciousness?”, “What is the hedonic intensity of an experience?”
EDIT: generally, the window to intervene is too small to wait for the evidence.
In such cases, I think imprecise probabilities are the way to go to reduce arbitrariness. We can do sensitivity analysis. If whether the intervention looks good or bad overall depends highly on fairly arbitrary judgements or priors, we might disprefer it and prefer to support things that are more robustly positive. This is difference-making ambiguity aversion.
And/or we do can some kind of bracketing.
Also, you should think of research as an intervention itself that could backfire. Who could use the research, and could they use it in ways you’d judge as very negative? How likely is that? This will of course depend on the case and your own specific views.
The reasons you mentioned for gathering strong evidence not being possible (or being very difficult) apply to some extent to efforts increasing human welfare, but humans have probably still made progress on increasing human welfare over the past 200 years or so? Can one be confident similar progress cannot be extended to non-humans?
I agree research can backfire. However, at least historically, doing research on the sentience of animals, and on how to increase their welfare has mostly been beneficial for the target animals?
Interesting! Thank you for writing this, this is something I was also wondering about while reading for the Warwick EA fellowship. My intuition is also that in the case of a “many-membered set of probability functions”, I’d define a prior over those and then compute an expected value as if nothing happened. I acknowledge that there is substantial (or even overwhelming) uncertainty sometimes and I can understand the impulse of wanting a separate conceptual handle for that. But it’s still “decision making under uncertainty” and should therefore be subsumable under Bayesianism.
I feel similar to ben.smith that I might be completely missing something. But I also wonder if this confusion might just be an echo of the age-old Bayesianism vs Frequentism debate, where people have different intuition about whether priors over probability distributions are a-ok.
There is an argument from intuition that carry some force by Schoenfield (2012) that we can’t use a probability function:
Intuitively, this sounds right. And if you went from this problem trying to understand solve the crime on intuition, you might really have no idea. Reading the passage, it sounds mind-boggling.
On the other hand, if you applied some reasoning and study, you might be able to come up with some probability estimates. You could identify the conditioning of P(Smith did it|an eyewitness says Smith did it), including a probability distribution on that probability itself, if you like. You can identify how to combine evidence from multiple witnesses, i.e., P(Smith did it|eyewitness 1 says Smith did it) & P(Smith did it|eyewitness 2 says Smith did it), and so on up to 68 and 69. You can estimate the independence of eyewitnesses, and from that work out how to properly combine evidence from multiple eyewitnesses.
And it might turn out that you don’t update as a result of the extra eyewitness, under some circumstances. Perhaps you know the eyewitnesses aren’t independent; they’re all card-carrying members of the “We hate Smith” club. In that case it simply turns out that the extra eye-witness is irrelevant to the problem; it doesn’t qualify as evidence, so it it doesn’t mean you’re insensitive to “mild evidential sweetening”.
I think a lot of the problem here is that these authors are discussing what one could do when one sits down for the first time and tries to grapple with a problem. In those cases there’s so many undefined features of the problem that it really does seem impossible and you really are clueless.
But that’s not the same as saying that, with sufficient time, you can’t put probability distributions to everything that’s relevant and try to work out the joint probability.
----
Schoenfield, M. Chilling out on epistemic rationality. Philos Stud 158, 197–219 (2012).
Hi Ben.
One should update towards a higher chance of Smith having commited the crime. However, if one was around 50 % confident that Smith commited the crime before the update, an update much smaller than 50 pp will still leave one around 50 % confident that Smith commited the crime. However, the best guess for the probability that Smitth commited the crime should still go up as a result of the update. If the contribution of an additional eyewitness feels completely irrelevant, one could subjectively estimate the update from “update for N additional eyewitnesses”/N. This will not feel completely irrelevant for a sufficiently large N, unless one considers all eyewitnesses testifying that Smith commited the crime no evidence at all.
I agree.
Some questions here are whether 50-50 as precise probabilities to start is reasonable and whether the approach to assign 50-50 as precise probabilities is reasonable.
If, when looking at the scenario, you would have done something like “wow, that’s so complicated and I’m clueless, so 50-50”, then your reaction almost certainly would have been the same if the example originally included one extra eyewitness in favour of one side. But then this tells you your initial way to assign credences was insensitive to this small difference. And yet after the initial assignment, you say it should be sensitive.
Or, if you forgot your initial judgement or the number of eyewitnesses and was just given the total and looked at the situation with fresh eyes, you’d come up with 50-50 again.
Alternatively, you could build a precise probability distribution as a function of the evidence that weighs it all, but this would be very sensitive to arbitrary choices.
I could report 50 % for 68 and 69 eyewitnesses, but this does not necessarily imply I am insensitive to small changes in the number of eyewitnesses. In practice, I would be reporting my best guess rounded to the closest multiple of 0.1 or so. So I believe the reported value being exactly the same would only mean my best guesses differ by less than 10 pp, not that they are exactly the same. I would say the mean of the (rounded) reported best guesses for a given number of eyewitnesses tends to the (precise) underlying best guess as the number of reports increases. If I could hypothetically encounter the question in practically the same situation for 1 M times, I could easily see the mean of my reported values for 68 and 69 eyewitnesses being different.
If I asked you to actually decide who’s more likely to be the culprit, how would you do it?
What do you do if you don’t have reference class information for each part of the problem? How do you weigh the conflicting evidence? I’m imaginging that at many steps, you’d have to rely on direct impressions or numbers that just came to mind.
Would you feel like whatever came out was very arbitrary and depended too much on direct impressions or numbers that just came to mind? Would you actually believe and endorse what came out? Would you defend it to other people?
What I would actually do depends a lot on the situation, but I have a hard time imagining scenarios where it matters whether the probability of Jones having commited the crime is 40 % or 60 %. So I might not even try to decrease the uncertainty about this, and just focus on other considerations. What would maximise the impact of my future donations and work? What information would I have about Jones and Smith? Who would have the greater potential to contribute to a better world? How much time would I have to decide? Would I be accountable in some way for my decision? If so, how would my decision be assessed? What would be the potential consequences of people concluding I made a good or bad decision? How were decisions like mine assessed in the past?
Do you (Michael) see your views about precise and imprecise credences significantly affecting what you would actually do in the real world in a scenario where you had to blame Jones or Smith? Considerations like the ones I mentioned above would matter mode? I may be dodging your question, but I am ultimately interested in making better decisions in the real world. So I think it makes sense to discuss precise and imprecise credences in the context of realistic scenarios.
Probably not. I see it as more illustrative of important cases. Imagine instead it’s between supporting an intervention or not, and it has similar complexity and considerations going in each direction.
More relevant examples to us could be: crops vs nature for wild animals, climate change on wild animals, fishing on wild animals, the far future effects of our actions, the acausal influence of our actions. These are all things I feel clueless enough about to mostly bracket away and ignore when they are side effects of direct interventions I’m interested in supporting. I’m not ignoring them because I think they’re small. I think they are likely much larger than the effects I’m not ignoring.
I may also want to further study some of them, but I’m often not that optimistic about making much progress (especially for far future effrcts and acausal influence) and for that progress to be used in a way that isn’t net negative overall by my lights.
How much more optimistic would you be about research on i) the welfare of soil animals and microorganisms, and ii) comparisons of (expected hedonistic) welfare across species if you strongly endorsed expectational total hedonistic utilitarianism, moral realism, and precise probabilitites, and ignored acausal effects, and effects after 100 years?
While browsing types of uncertainties, I stumbled upon the idea of state space uncertainty and conscious unawareness, which sounds similar to your explanation of cluelessness and which might be another helpful angle for people with a more Bayesian perspective.
https://link.springer.com/article/10.1007/s10670-013-9518-4
A good point.
There are things you can do to correct for this sort of thing-for instance, go one level more meta, estimate the probability of unforeseen consequences in general, or within the class of problems that your specific problem fits into.
We couldn’t have predicted the fukushima disaster, but perhaps we can predict related things with some degree of certainty—the average cost and death toll of earthquakes worldwide, for instance. In fact, this is a fairly well explored space, since insurers have to understand the risk of earthquakes.
The ongoing pandemic is a harder example—the rarer the black swan, the more difficult it is to predict. But even then, prior to the 2020 pandemic, the WHO had estimated the amortized costs of pandemics as in the order of 1% of global GDP annually (averaged over years when there are and aren’t pandemics), which seems like a reasonable approximation.
I don’t know how much of a realistic solution that would be in practice.
This is a great example, thanks for sharing!
I think the example Ben cites in his reply is very illustrative.
You might feel that you can’t justify your one specific choice of prior over another prior, so that particular choice is arbitrary, and then what you should do could depend on this arbitrary choice, whereas an equally reasonable prior would recommend a different decision. Someone else could have exactly the same information as you, but due to a different psychology, or just different patterns of neurons firing, come up with a different prior that ends up recommending a different decision. Choosing one prior over another without reason seems like a whim or a bias, and potentially especially prone to systematic error.
It seems bad if we’re basing how to do the most good on whims and biases.
If you’re lucky enough to have only finitely many equally reasonable priors, then I think it does make sense to just use a uniform meta-prior over them, i.e. just take their average. This doesn’t seem to work with infinitely many priors, since you could use different parametrizations to represent the same continuous family of distributions, with a different uniform distribution and therefore average for each parametrization. You’d have to justify your choice of parametrization!
As another example, imagine you have a coin that someone (who is trustworthy) has told you is biased towards heads, but they haven’t given you any hint how much, and you want to come up with a probability distribution for the fraction of heads over 1,000,000 flips. So, you want a distribution over the interval [0, 1]. Which distribution would you use? Say you give me a probability density function f. Why not (1−p)f(x)+p for some p∈(0,1)? Why not 1∫10f(xp)dxf(xp) for some p>0? If f is a weighted average of multiple distributions, why not apply one of these transformations to one of the component distributions and choose the resulting weighted average instead? Why the particular weights you’ve chosen and not slightly different ones?
I think you just have to make your distribution uninformative enough that reasonable differences in the weights don’t change your overall conclusion. If they do, then I would concede that the solution to your specific question really is clueless. Otherwise, you can probably find a response.
Rather than thinking of directly of appropriate distribution for the 1,000,000 flips, I’d think of a distribution to model p itself. Then you can run simulations based on the distribution of p to calculate the distribution of the fraction of 1000,000 flips. p∈(0.5,1.0], and then we need to select a distribution for p over that range.
There is no one correct probability distribution for p because any probability is just an expression of our belief, so you may use whatever probability distribution genuinely reflects your prior belief. A uniform distribution is a reasonable start. Perhaps you really are clueless about p, in which case, yes, there’s a certain amount of subjectivity about your choice. But prior beliefs are always inherently subjective, because they simply describe your belief about the state of the world as you know it now. The fact you might have to select a distribution, or set of distributions with some weighted average, is merely an expression of your uncertainty. This in itself, I think, doesn’t stop you from trying to estimate the result.
I think this expresses within Bayesian terms the philosophical idea that we can only make moral choices based on information available at the time; one can’t be held morally responsible for mistakes made on the basis of the information we didn’t have.
Perhaps you disagree with me that a uniform distribution is the best choice. You reason thus: “we have some idea about the properties of coins in general. It’s difficult to make a coin that is 100% biased towards heads. So that seems unlikely”. So we could pick a distribution that better reflects your prior belief. Perhaps a suitable choice might be Beta(2,2) with a truncation at 0.5, which will give the greatest likelihood of p just above 0.5, and a declining likelihood down to 1.0.
Maybe you and i just can’t agree after all that there is still no consistent and reasonable prior choice you can make, and not even any compromise. And let’s say we both run simulations using our own priors and find entirely different results and we can’t agree on any suitable weighting between them. In that case, yes, I can see you have cluelessness. I don’t think it follows that, if we went through the same process for estimating the longtermist moral worth of malaria bednet distribution, we must have intractable complex cluelessness about specific problems like malaria bednet distribution. I think I can admit that perhaps, right now, in our current belief state, we are genuinely clueless, but it seems that there is some work that can be done that might eliminate the cluelessness.
Hi Michael.
I agree. However, in cases where priors are playing a crucial role, one should simply prioritise gathering more evidence until there is reasonable convergence about what to do (among a given group of people, for a particular decision)?
In some cases, we can’t gather strong enough evidence, say because:
they’re questions about very speculative or unprecedented possibilities and the evidence would either be too indirect and weak or come too late to be very action-guiding, e.g. often for AI risk, conscious subsystems, or
there will be too much noise or confounding, too small a sample size and anything like an RCT is too impractical (e.g. policy, corporate outreach) or wouldn’t generalize well, or
the disagreements are partly conceptual, definitional or philosophical, e.g. “What is consciousness?”, “What is the hedonic intensity of an experience?”
EDIT: generally, the window to intervene is too small to wait for the evidence.
In such cases, I think imprecise probabilities are the way to go to reduce arbitrariness. We can do sensitivity analysis. If whether the intervention looks good or bad overall depends highly on fairly arbitrary judgements or priors, we might disprefer it and prefer to support things that are more robustly positive. This is difference-making ambiguity aversion.
And/or we do can some kind of bracketing.
Also, you should think of research as an intervention itself that could backfire. Who could use the research, and could they use it in ways you’d judge as very negative? How likely is that? This will of course depend on the case and your own specific views.
The reasons you mentioned for gathering strong evidence not being possible (or being very difficult) apply to some extent to efforts increasing human welfare, but humans have probably still made progress on increasing human welfare over the past 200 years or so? Can one be confident similar progress cannot be extended to non-humans?
I agree research can backfire. However, at least historically, doing research on the sentience of animals, and on how to increase their welfare has mostly been beneficial for the target animals?