# Zeke_Sherman comments on Tiny Probabilities of Vast Utilities: Defusing the Initial Worry and Steelmanning the Problem

• >Imagine you are adding new possible outcomes to consideration, one by one. Most of the outcomes you add won’t change the EV much. But occasionally you’ll hit one that makes everything that came before look like a rounding error, and it might flip the sign of the EV.

But the probability of those rare things will be super low. It’s not obvious that they’ll change the EV as much as nearer term impacts.

This would benefit from an exercise in modeling the utilities and probabilities of a certain intervention to see what the distribution actually looks like. So far no one has bothered (or needed, perhaps) to actually enumerate the 2nd, 3rd, etc… order effects and estimate their probabilities. All this theorizing might be unnecessary if our actual expectations follow a different pattern.

>So this is a problem in theory—it means we are approximating an ideal which is both stupid and incoherent.

Are we? Expected utility is still a thing. Some actions have greater expected utility than others even if the probability distribution has huge mass across both positive and negative possibilities. If infinite utility is a problem then it’s already a problem regardless of any funnel or oscillating type distribution of outcomes.

>Arguably this behavior is the predictable result of considering more and more possibilities in your EV calculations, and it doesn’t represent progress in any meaningful sense—it just means that EAs have gone farther down the funnel-shaped rabbithole than everybody else.

Another way of describing this phenomenon is that we are simply seizing the low hanging fruit, and hard intellectual progress isn’t even needed.

• But the probability of those rare things will be super low. It’s not obvious that they’ll change the EV as much as nearer term impacts. … All this theorizing might be unnecessary if our actual expectations follow a different pattern.

Yes, if the profiles are not funnel-shaped then this whole thing is moot. I argue that they are funnel-shaped, at least for many utility functions currently in use (e.g. utility functions that are linear in QALYs) I’m afraid my argument isn’t up yet—it’s in the appendix, sorry—but it will be up in a few days!

Are we? Expected utility is still a thing. Some actions have greater expected utility than others even if the probability distribution has huge mass across both positive and negative possibilities. If infinite utility is a problem then it’s already a problem regardless of any funnel or oscillating type distribution of outcomes.

If the profiles are funnel-shaped, expected utility is not a thing. The shape of your action profiles depends on your probability function and your utility function. Yes, infinitely valuable outcomes are a problem—but I’m arguing that even if you ignore infinitely valuable outcomes, there’s still a big problem having to do with infinitely many possible finite outcomes, and moreover even if you only consider finitely many outcomes of finite value, if the profiles are funnel-shaped then what you end up doing will be highly arbitrary, determined mostly by whatever is happening at the place where you happened to draw the cutoff.

Another way of describing this phenomenon is that we are simply seizing the low hanging fruit, and hard intellectual progress isn’t even needed.

That’s what I’d like to think, and that’s what I do think. But this argument challenges that; this argument says that the low-hanging fruit metaphor is inappropriate here: there is no lowest-hanging fruit or anything close; there is an infinite series of fruit hanging lower and lower, such that for any fruit you pick, if only you had thought about it a little longer you would have found an even lower-hanging fruit that would have been so much easier to pick that it would easily justify the cost in extra thinking time needed to identify it… moreover, you never really “pick” these fruit, in that the fruit are gambles, not outcomes; they aren’t actually what you want, they are just tickets that have some chance of getting what you want. And the lower the fruit, the lower the chance...

• >The shape of your action profiles depends on your probability function

Are you saying that there is no expected utility just because people have different expectations?

Well, of course. That doesn’t mean there is no expected utility! It’s just different for different agents.

>I’m arguing that even if you ignore infinitely valuable outcomes, there’s still a big problem having to do with infinitely many possible finite outcomes,

That in itself is not a problem, imagine a uniform distribution from 0 to 1.

>if the profiles are funnel-shaped then what you end up doing will be highly arbitrary, determined mostly by whatever is happening at the place where you happened to draw the cutoff.

If you do something arbitrary like drawing a cutoff, then of course how you do it will have arbitrary results. I think the lesson here is not to draw cutoffs in the first place.

>That’s what I’d like to think, and that’s what I do think. But this argument challenges that; this argument says that the low-hanging fruit metaphor is inappropriate here: there is no lowest-hanging fruit or anything close; there is an infinite series of fruit hanging lower and lower, such that for any fruit you pick, if only you had thought about it a little longer you would have found an even lower-hanging fruit that would have been so much easier to pick that it would easily justify the cost in extra thinking time needed to identify it… moreover, you never really “pick” these fruit, in that the fruit are gambles, not outcomes; they aren’t actually what you want, they are just tickets that have some chance of getting what you want. And the lower the fruit, the lower the chance...

There must be a lowest hanging fruit out of any finite set of possible actions, as long as “better intervention than” follows basic decision theoretic properties which come automatically if they have expected utility values.

Also, remember the conservation of expected evidence. When we think about the long run effects of a given intervention, we are updating our prior to go either up or down, not predictably making it seem more attractive.