I understand you to be offering two potential stories to justify ‘speculativeness-discounting’.
First, EAs don’t (by and large) apply a speculativeness-discount ex post. Instead, there’s a more straightforward ‘Bayesian+EUM’ rationalization of the practice. For instance, the epistemic practice of EAs may be better explained with reference to more common-sense priors, potentially mediated by orthodox biases.
Or perhaps EAs do apply a speculativeness-discount ex post. This too can be justified on Bayesian grounds.
We often face doubts about our ability to reason through all the relevant considerations, particularly in speculative domains. For this reason, we update on higher-order uncertainty, and implement heuristics which themselves are justified on Bayesian grounds.
In my response, I’ll assume that your attempted rationale for Principle 4 involves justifying the norm with respect to the following two views:
Expected Utility Maximization (EUM) is the optimal decision-procedure.
The relevant probabilities to be used as inputs into our EUM calculation are our subjective credences.
The ‘Common Sense Priors’ Story
I think your argument in (1) is very unlikely to provide a rationalization of EA practice on ‘Bayesian + EUM’ grounds.[1]
Take Pascal’s Mugging. The stakes can be made high enough that the value involved can easily swamp your common-sense priors. Of course, people have stories for why they shouldn’t give the money to the mugger. But these stories are usually generated because handing over their wallet is judged to be ridiculous, rather than the judgment arising from an independent EU calculation. I think other fanatical cases will be similar. The stakes involved under (e.g.) various religious theories and our ability to acausally affect an infinite amount of value are simply going to be large enough to swamp our initial common-sense priors.
Thus, I think the only feasible ‘Bayes+EUM’ justification you could offer would have to rely on your ‘higher-order evidence’ story about the fallibility of our first-order reasoning, which we’ll turn to below.
The ‘Higher-Order Evidence’ Story
I agree that we can say: “we should be fanatical insofar as my reasoning is correct, but I am not confident in my reasoning.”
The question, then, is how to update after reflecting on your higher-order evidence. I can see two options: either you have some faith in your first-order reasoning, or no faith.
Let’s start with the case where you have some faith in your first-order reasoning. Higher-order evidence about your own reasoning might decrease the confidence in your initial conclusion. But, as you note, “we might find that the EV of pursuing the speculative path warrants fanaticism”. So, what to do in that case?
I think it’s true that many people will cite considerations of the form “let’s pragmatically deprioritize the high EV actions that are both speculative and fanatical, in anticipation of new evidence”. I don’t think that provides a sufficient justificatory story of the epistemic norms to which most of us hold ourselves.
Suppose we knew that our evidential situation was as good as it’s ever going to be. Whatever evidence we currently have about (e.g.) paradoxes in infinite ethics, or the truth of various religions constitutes ~all the evidence we’re ever going to have.
I still don’t expect people to follow through on the highest EV option, when that option is both speculative and fanatical.
Or maybe you have a bounded utility function. In that case, imagine that the world already contains a sufficiently large number of suffering entities. How blase are you, really, about the creation of arbitrarily many suffering-filled hellscapes?
There’s more to say here, but the long and short of it is: if you fail to reach a point where you entirely discount certain forms of speculative reasoning, I don’t think you’ll be able to recover anything like Principle 4. My honest view is that many EAs have a vague hope that such theories will recover something approaching normality, but very few people actually try to trace out the implications of such theories on their own terms, and follow through on these implications. I’m sympathetic to this quote from Paul Christiano:
I tried to answer questions like “How valuable is it to accelerate technological progress?” or “How bad is it if unaligned AI takes over the world?” and immediately found that EU maximization with anything like “utility linear in population size” seemed to be unworkable in practice. I could find no sort of common-sensical regularization that let me get coherent answers out of these theories, and I’m not sure what it would look like in practice to try to use them to guide our actions.
Higher-Order Evidence and Epistemic Learned Helplessness
Maybe you’d like to say: “in certain domains, we should assign our first-order calculations about which actions maximize EU zero weight. The heuristic ‘sometimes assign first-order reasoning zero weight’ can be justified on Bayesian grounds.”
I agree that we should sometimes assign our first-order calculations about which actions maximize EU zero weight. I’m doubtful that Bayesianism or EUM play much of a role in explaining why this norm is justified.
When we’re confronted with the output of an EUM calculation that feels off, we should listen to the parts of us which tell us to check again, and ask why we feel tempted to check again.
If we’re saying “no, sorry, sometimes I’m going to put zero weight on a subjective EU calculation”, then we’re already committed to a view under which subjective EU calculations only provide action-guidance in the presence of certain background conditions.
If we’re willing to grant that, then I think the interesting justificatory story is a story which informs us of what the background conditions for trusting EU calculations actually are — rather than attempts to tell post hoc stories about how our practices can ultimately be squared with more foundational theories like Bayesianism + EUM.
If you’re interested, I’ll have a post in April touching on these themes. :)
I also think the sociological claim you made is probably false. However, as you’re primarily asking about the justificatory side of things, I’ll bracket that here — though I’m happy to make this case in more detail if you’d like.
Thanks for the thorough response! I agree with a lot of what you wrote, especially the third section on Epistemic Learned Helplessness: “Bayesianism + EUM, but only when I feel like it” is not a justification in any meaningful sense.
On Priors
I agree that we can construct thought experiments (Pascal’s Mugging, acausal trade) with arbitrarily high stakes to swamp commonsense priors (even without religious scenarios or infinite value, which are so contested I think it would be difficult to extract a sociological lesson from them).
On Higher Order Evidence
I still think a lot of speculative conclusions we encounter in the wild suffer from undiscovered evidence and model uncertainty, and even barring this we might want to defer taking action until we’ve had a chance to learn more.
Your response jumps over these cases to those where we have “~all the evidence we’re ever going to have,” but I’m skeptical these cases exist. Even with religion, we might expect some future miracles or divine revelations to provide new evidence; we have some impossibility theorems in ethics, but new ideas might come to light that resolve paradoxes or avoid them completely. In fact, soteriological research and finding the worldview that best acausally benefits observers are proposals to find new evidence.
But ok, yes, I think we can probably come up with cases where we do have ~all the evidence and still refrain from acting on speculative + fanatical conclusions.
Problem 1: Nicheness
From here on, I’m abandoning the justification thing. I agree that we’ve found some instances where the Fourth Principle holds without Bayesian + EUM justification. Instead, I’m getting more into the semantics of what is a “norm.”
The problem is that the support for this behavior among EAs comes from niche pieces of philosophy like Pascal’s Mugging, noncausal decision theory, and infinite ethics, ideas that are niche not just relative to the general population, but also within EA. So I feel like the Fourth Principle amounts to “the minority of EAs who are aware of these edge cases behave this way when confronted with them,” which doesn’t really seem like a norm about EA.
Problem 2: Everyone’s Doing It
(This is also not a justification, it’s an observation about the Fourth Principle)
The first three principles capture ways that EA differs from other communities. The Fourth Principle, on the other hand, seems like the kind of thing that all people do? For example, a lot of people write off earning to give when they first learn about it because it looks speculative and fanatical. Now, maybe EAs differ from other people on which crazy train stop they deem “speculative,” and I think that would qualify as a norm, but relative to each person’s threshold for “speculative,” I think this is more of a human-norm than an EA-norm.
Would love your thoughts on this, and I’m looking forward to your April post :)
Thanks for the comment!
(Fair warning, my response will be quite long)
I understand you to be offering two potential stories to justify ‘speculativeness-discounting’.
First, EAs don’t (by and large) apply a speculativeness-discount ex post. Instead, there’s a more straightforward ‘Bayesian+EUM’ rationalization of the practice. For instance, the epistemic practice of EAs may be better explained with reference to more common-sense priors, potentially mediated by orthodox biases.
Or perhaps EAs do apply a speculativeness-discount ex post. This too can be justified on Bayesian grounds.
We often face doubts about our ability to reason through all the relevant considerations, particularly in speculative domains. For this reason, we update on higher-order uncertainty, and implement heuristics which themselves are justified on Bayesian grounds.
In my response, I’ll assume that your attempted rationale for Principle 4 involves justifying the norm with respect to the following two views:
Expected Utility Maximization (EUM) is the optimal decision-procedure.
The relevant probabilities to be used as inputs into our EUM calculation are our subjective credences.
The ‘Common Sense Priors’ Story
I think your argument in (1) is very unlikely to provide a rationalization of EA practice on ‘Bayesian + EUM’ grounds.[1]
Take Pascal’s Mugging. The stakes can be made high enough that the value involved can easily swamp your common-sense priors. Of course, people have stories for why they shouldn’t give the money to the mugger. But these stories are usually generated because handing over their wallet is judged to be ridiculous, rather than the judgment arising from an independent EU calculation. I think other fanatical cases will be similar. The stakes involved under (e.g.) various religious theories and our ability to acausally affect an infinite amount of value are simply going to be large enough to swamp our initial common-sense priors.
Thus, I think the only feasible ‘Bayes+EUM’ justification you could offer would have to rely on your ‘higher-order evidence’ story about the fallibility of our first-order reasoning, which we’ll turn to below.
The ‘Higher-Order Evidence’ Story
I agree that we can say: “we should be fanatical insofar as my reasoning is correct, but I am not confident in my reasoning.”
The question, then, is how to update after reflecting on your higher-order evidence. I can see two options: either you have some faith in your first-order reasoning, or no faith.
Let’s start with the case where you have some faith in your first-order reasoning. Higher-order evidence about your own reasoning might decrease the confidence in your initial conclusion. But, as you note, “we might find that the EV of pursuing the speculative path warrants fanaticism”. So, what to do in that case?
I think it’s true that many people will cite considerations of the form “let’s pragmatically deprioritize the high EV actions that are both speculative and fanatical, in anticipation of new evidence”. I don’t think that provides a sufficient justificatory story of the epistemic norms to which most of us hold ourselves.
Suppose we knew that our evidential situation was as good as it’s ever going to be. Whatever evidence we currently have about (e.g.) paradoxes in infinite ethics, or the truth of various religions constitutes ~all the evidence we’re ever going to have.
I still don’t expect people to follow through on the highest EV option, when that option is both speculative and fanatical.
Under MEC, EAs should plausibly be funneling all their money into soteriological research. Or perhaps you don’t like MEC, and think we should work out the most plausible worldview under which we can affect strongly-Ramsey-many sentient observers.[2]
Or maybe you have a bounded utility function. In that case, imagine that the world already contains a sufficiently large number of suffering entities. How blase are you, really, about the creation of arbitrarily many suffering-filled hellscapes?
There’s more to say here, but the long and short of it is: if you fail to reach a point where you entirely discount certain forms of speculative reasoning, I don’t think you’ll be able to recover anything like Principle 4. My honest view is that many EAs have a vague hope that such theories will recover something approaching normality, but very few people actually try to trace out the implications of such theories on their own terms, and follow through on these implications. I’m sympathetic to this quote from Paul Christiano:
Higher-Order Evidence and Epistemic Learned Helplessness
Maybe you’d like to say: “in certain domains, we should assign our first-order calculations about which actions maximize EU zero weight. The heuristic ‘sometimes assign first-order reasoning zero weight’ can be justified on Bayesian grounds.”
I agree that we should sometimes assign our first-order calculations about which actions maximize EU zero weight. I’m doubtful that Bayesianism or EUM play much of a role in explaining why this norm is justified.
When we’re confronted with the output of an EUM calculation that feels off, we should listen to the parts of us which tell us to check again, and ask why we feel tempted to check again.
If we’re saying “no, sorry, sometimes I’m going to put zero weight on a subjective EU calculation”, then we’re already committed to a view under which subjective EU calculations only provide action-guidance in the presence of certain background conditions.
If we’re willing to grant that, then I think the interesting justificatory story is a story which informs us of what the background conditions for trusting EU calculations actually are — rather than attempts to tell post hoc stories about how our practices can ultimately be squared with more foundational theories like Bayesianism + EUM.
If you’re interested, I’ll have a post in April touching on these themes. :)
I also think the sociological claim you made is probably false. However, as you’re primarily asking about the justificatory side of things, I’ll bracket that here — though I’m happy to make this case in more detail if you’d like.
Presumably acausally.
Thanks for the thorough response! I agree with a lot of what you wrote, especially the third section on Epistemic Learned Helplessness: “Bayesianism + EUM, but only when I feel like it” is not a justification in any meaningful sense.
On Priors
I agree that we can construct thought experiments (Pascal’s Mugging, acausal trade) with arbitrarily high stakes to swamp commonsense priors (even without religious scenarios or infinite value, which are so contested I think it would be difficult to extract a sociological lesson from them).
On Higher Order Evidence
I still think a lot of speculative conclusions we encounter in the wild suffer from undiscovered evidence and model uncertainty, and even barring this we might want to defer taking action until we’ve had a chance to learn more.
Your response jumps over these cases to those where we have “~all the evidence we’re ever going to have,” but I’m skeptical these cases exist. Even with religion, we might expect some future miracles or divine revelations to provide new evidence; we have some impossibility theorems in ethics, but new ideas might come to light that resolve paradoxes or avoid them completely. In fact, soteriological research and finding the worldview that best acausally benefits observers are proposals to find new evidence.
But ok, yes, I think we can probably come up with cases where we do have ~all the evidence and still refrain from acting on speculative + fanatical conclusions.
Problem 1: Nicheness
From here on, I’m abandoning the justification thing. I agree that we’ve found some instances where the Fourth Principle holds without Bayesian + EUM justification. Instead, I’m getting more into the semantics of what is a “norm.”
The problem is that the support for this behavior among EAs comes from niche pieces of philosophy like Pascal’s Mugging, noncausal decision theory, and infinite ethics, ideas that are niche not just relative to the general population, but also within EA. So I feel like the Fourth Principle amounts to “the minority of EAs who are aware of these edge cases behave this way when confronted with them,” which doesn’t really seem like a norm about EA.
Problem 2: Everyone’s Doing It
(This is also not a justification, it’s an observation about the Fourth Principle)
The first three principles capture ways that EA differs from other communities. The Fourth Principle, on the other hand, seems like the kind of thing that all people do? For example, a lot of people write off earning to give when they first learn about it because it looks speculative and fanatical. Now, maybe EAs differ from other people on which crazy train stop they deem “speculative,” and I think that would qualify as a norm, but relative to each person’s threshold for “speculative,” I think this is more of a human-norm than an EA-norm.
Would love your thoughts on this, and I’m looking forward to your April post :)