Expected value maximization hides a lot of important details.
I think a pretty underrated and forgotten part of of Rethink Priorities’ CURVE sequence is the risk aversion work. I think the defenses of EV against more risk-aware models seem to often boil down to EV’s simplicity. But I think that EV actually just hides a lot of important detail, including, most importantly, that if you only care about EV maximization, you might be forced to conclude that worlds where you’re more likely to cause harm than not are preferable.
As an example, imagine that you’re considering a choice that can cause 10 equally possible outcomes. In 6 of them, you’ll create −1 utility. In 3 of them, your impact is neutral. In 1 of them, you’ll create 7 utility. The EV of taking the action is (-6+0+7)/10 = 0.1. This is a positive number! Your expected value is positive, even though you have a 60% chance of causing harm. In expectation you’re more likely than not to cause harm, but also in expectation you should expect to increase utility a bit. This is weird.
Scenario 1
More concretely, if I consider the following choices, which are equivalent from an EV perspective:
Option A. A 0% chance of causing a harmful outcome, but in expectation will cause +10 utility
Option B. A 20% chance of causing a harmful outcome, but in expectation will cause +10 utility
It seems really bizarre to not prefer Option A. But if I prefer Option A, I’m just accepting risk aversion to at least some extent. But what if the numbers slip a little more?
Scenario 2
Option A. A 0% chance of causing a harmful outcome, but in expectation will cause +9.9999 utility
Option B. A 20% chance of causing a harmful outcome, but in expectation will cause +10 utility
Do I really want to take a 20% chance on causing harm in exchange for 0.001% gain in utility caused?
Scenario 3
Option A. A 0% chance of causing a harmful outcome, but in expectation will cause +5 utility
Option B. A 99.99999% chance of causing a harmful outcome, but in expectation will cause +10 utility
Do I really want to be exceedingly likely to cause harm, in exchange for a 100% gain in utility?
I don’t know the answers to the above scenarios, but I think it feels like just saying “the EV is X” without reference to the downside risk misses a massive part of the picture. It seems much better to say “the expected range of outcomes are a 20% chance of really bad stuff happening, a 70% chance of nothing happening and a 10% of a really really great outcome, which all averages out to an >0 average”. This is meaningfully different than saying “no downside risk, and a 10% chance of a pretty good outcome, so >0 average”.
I think that risk aversion is pretty important, but even if it isn’t incorporated into people’s thinking at all, it really doesn’t feel like EV produces a number I can take at face value, and that makes me feel like EV isn’t actually that simple.
The place where I currently see this happening the most is naive expected value maximization in reasoning about animal welfare — I feel like I’ve seen an uptick in “I think there is a 52% chance these animals live net negative lives, so we should do major irreversible things to reduce their population”. But it’s pretty easy to imagine doing those things being harmful, or your efforts backfiring, etc. in ways that cause harm.
I think this is wrong, and this intuition that many people have derives from a psychological mistake. Essentially everything in life has diminishing marginal utility, so it almost always makes sense to be risk averse. So it’s intuitive that you should be risk averse with respect to expected utility. But that doesn’t make any logical sense—by definition, you don’t have diminishing marginal utility of utility. Your utility function already accounts for risk aversion. Being risk averse with respect to utility is double-counting.
This is a valid statement but non-responsive to the actual post. The argument is that there is intuitive appeal in having a utility function with a discontinuity at zero (ie a jump in disutility from causing harm), and ~standard EV maximisation does not accommodate that intuition. That is a totally separate normative claim from arguing that we should encode diminishing marginal utility.
I don’t think this is quite what I’m referring to, but I can’t quite tell! But my quick read is we are talking about different things (I think because I used the word utility very casually). I’m not talking about my own utility function with regard to some action, but the potential outcomes of that action on others, and I don’t know if I’m embracing risk aversion views as much as relating to their appeal.
Or maybe I’m misunderstanding, and you’re just rejecting the conclusion that there is a moral difference between taking, say, an action with +1 EV and a 20% chance of causing harm and an action with +1EV and a 0% chance of causing harm / think I just shouldn’t care about that difference?
In retrospect my comment was poorly thought-out, I think you’re right that it’s not directly addressing your scenarios.
I think there are two separate issues with my comment:
My comment was about being risk-averse with respect to utility; your quick take was about wanting to avoid causing harm; those aren’t necessarily the same thing.
You can self-consistently believe in diminishing marginal utility of welfare, i.e., your utility function isn’t just “utility = sum(welfare)”. And the way your quick take used the word “utility”, you really meant something more like “welfare” (it sounds like this is what you’re saying in your reply comment).
RE #1, my sense is that “person is risk-averse with respect to utility” is isomorphic to “person disprefers a lottery with a possibility of doing harm, even if it has the same expected utility as a purely-positive lottery”. Or like, I think the person is making the same mistake in these two scenarios. But it’s not immediately obvious that these are isomorphic and I’m not 100% sure it’s true. Now I kind of want to see if I can come up with a proof but I would need to take some time to dig into the problem.
RE #2, I do in fact believe that utility = welfare, but that’s a whole other discussion and it’s not what I was trying to get at with my original comment, which means I think my comment missed the mark.
Or maybe I’m misunderstanding, and you’re just rejecting the conclusion that there is a moral difference between taking, say, an action with +1 EV and a 20% chance of causing harm and an action with +1EV and a 0% chance of causing harm / think I just shouldn’t care about that difference?
Depends on what you mean by “EV”. I do reject that conclusion if by EV you mean welfare. If by EV you mean something like “money”, then yeah I think money has diminishing marginal utility and you shouldn’t just maximized expected money.
I wish people would talk more about “sensitivity analysis”.
Your parameter estimates are just that, estimates. They probably result from intuitions or napkin math. They probably aren’t that precise. It’s easy to imagine a reasonable person generating different estimates in many cases.
If a relatively small change in parameters would lead to a relatively large change in the EV (example: in Scenario 3, just estimate the “probability of harm” a teensy bit different so it has a few more 9s, and the action looks far less attractive!) — then you should either (a) choose a different action, or (b) validate your estimates quite thoroughly since the VoI is very high, and beware of the Unilateralist’s Curse in this scenario, since other actors may be making parallel estimates for the action in question.
(I’m guessing you mean difference-making risk aversion here, based on your options being implicitly compared to doing nothing.)
When considering the potential of larger indirect effects on wild invertebrates, the far future and other butterfly effects, which interventions do you think look good (better than doing nothing) on difference-making risk aversion (or difference-making ambiguity aversion)?
I think I mean something slightly different than difference-making risk aversion, but I see what you’re saying. I don’t even know if I’m arguing against EV maximization—more just trying to point out that EV alone doesn’t feel like it fully captures the picture of the value I care about (e.g. likelihood of causing harm relative to doing nothing feels like another important thing). I think specifically, that there are plausible circumstances where I am more likely than not to cause additional harm, and in expectation that action has positive EV, feels concerning. I imagine lots of AI risk work could be like this: doing some research project has some strong chance of advancing capabilities a bit (high probability of a bit of negative value), but maybe a very small chance of massively reducing risk (low probability of tons of positive value). The EV looks good, but my median outcome will be the world being worse than it was if I hadn’t done anything.
Ok, that makes sense. I’d guess butterfly effects would be neutral in the median difference. The same could be the case for indirect effects on wild animals and the far future, although I’d say it’s highly ambiguous (imprecise probabilities) and something to be clueless about, and not precisely neutral about.
Would you say you care about the overall distribution of differences, too, and not just the median and the EV?
I think this critique misses how EV maximization works in a world with many actors taking uncorrelated risks.
Consider your Scenario 2: Individual actors choosing between Option A (0% harm chance, +9.9999 utility) vs Option B (20% harm chance, +10 utility). If we have 1000 altruistic actors each making independent choices with similar profiles, and they all choose Option B (higher EV), we’d expect:
800 successful outcomes (+8000 utility)
200 harmful outcomes (negative utility)
Net positive impact far exceeding what we’d get if everyone chose the “safe” option
This is portfolio theory applied to altruism. Just as index funds maximize returns by holding many uncorrelated assets, the altruistic community maximizes impact when individuals make risk-neutral EV calculations on independent projects.
The key caveats:
For large actors (major foundations, governments, AI companies, etc.), risk aversion makes more sense since their failures aren’t offset by others’ successes
For correlated risks (like your animal welfare example where many actors might simultaneously cause harm based on shared wrong beliefs), we need more caution
But for most EA individuals working on diverse, independent projects? Risk-neutral EV maximization is exactly what we want everyone doing. The portfolio effect means we’ll get the best aggregate outcome even though some individual bets will fail.
Are the projects of most EA individuals truly independent in the sense of their EVs being essentially uncorrelated with each other? That would be surprising to me, given that many of those projects are conditional on positive evaluation from a small number of funders, and many arose out of the same meta (so would be expected to have meaningful correlations with other active projects).
So my prediction is that most EA stuff falls into one of your two caveats. What I don’t have a good sense of is how correlated the average EA work is, and thus the degree of caution / risk aversion implied by the caveats.
In theory I agree with this, but in practise I personally think “Risk-neutral EV maximisation” can lead to bets which are far worse than they appear to be. This is because I think we often massively overrate the EV of “hits based approaches”.
Generally I think the lower probability of a bet, the higher chance there is of that EV being wrong and lower than stated. I’m keen to see evidence of high risk bets turning out well once in a while before I’m convinced that they really do have the claimed EVs...
Then your issue is with systemically flawed reasoning overestimating the likelihood of low-probability events. The solution for that would be to adjust by some factor that adjusts for this systemic epistemic bias, and then proceed with risk-neutral EV maximization (again, with the caveats that I had mentioned in my initial comment).
I think this is true as a response in certain cases, but many philanthropic interventions probably aren’t tried enough times to get the sample size and lots of communities are small. It’s pretty easy to imagine a situation like:
You and a handful of other people make some positive EV bets.
The median outcome from doing this is the world is worse, and all of the attempts at these bets end up neutral or negative.
The positive EV is never realized and the world is worse on average, despite both the individuals and the ecosystem being +EV.
It seems like this response would imply you should only do EV maximization if your movement is large (or that its impact is reliably predictable if the movement is large).
But I do think this is a fair point overall — though you could imagine a large system of interventions with the same features I describe that would have the same issues as a whole.
You’re absolutely right in my opinion Abraham. EV is a mathematical convenience and simplification that might work most of the time, but certainly not all.
I feel words like “expected value” are confusing & miselading (nothing expected about it—it’s a misnomer and accident of history/ translation from french that it’s called that), and this community would do well to avoid unnecessary jargon. I need to write up a post about this.
If we substituted the word “expected value” with mean or average, which is literally what it is (and I meet so many people in this community who get this wrong), it would be just as accurate but much more easier to understand, even to technical people. And many people understand that mean is not always the most appropriate statistic to use espescially with skewed distributions, e.g. reporting on incomes is often done with median, and that’s perfectly fine.
Expected value should also not be confused with expected utility (or as I prefer the mean utility). You use the terms interchangeably in your post as do many people in this community, it’s fine in causal conversation but it is worth being specific when necessary in discussions like this. I suspect because the word “value” in expected value implies some humanistic desireable ideal, but that’s what the word utility is for.
You also say use the term “in expectation” to mean the most likeliest outcome (i.e. the mode outcome) or more than 50%. I’m not familiar with that langauge—maybe it is accurate, but to me it is confusing when you can simply say the mode or the likeliest outcome.
So ultimately I suspect your dilemmas boil down to the recognition that there are circumstances where mean, mode, median, (and in fact even more statistics) can be maximised and we get different optimal decisions depending on the statistic we use.
At the end of the day we want to: maximise utility. But if utility has a probability distribution, it’s hard to maxmise without using some statistic (mean, mode, median etc) that maps a probability distribution to a real number. Using mean (EV) is not completely arbitrary I believe (it has some nice preperties), but it’s not fully settled. I personally think it has problems.
Expected value maximization hides a lot of important details.
I think a pretty underrated and forgotten part of of Rethink Priorities’ CURVE sequence is the risk aversion work. I think the defenses of EV against more risk-aware models seem to often boil down to EV’s simplicity. But I think that EV actually just hides a lot of important detail, including, most importantly, that if you only care about EV maximization, you might be forced to conclude that worlds where you’re more likely to cause harm than not are preferable.
As an example, imagine that you’re considering a choice that can cause 10 equally possible outcomes. In 6 of them, you’ll create −1 utility. In 3 of them, your impact is neutral. In 1 of them, you’ll create 7 utility. The EV of taking the action is (-6+0+7)/10 = 0.1. This is a positive number! Your expected value is positive, even though you have a 60% chance of causing harm. In expectation you’re more likely than not to cause harm, but also in expectation you should expect to increase utility a bit. This is weird.
Scenario 1
More concretely, if I consider the following choices, which are equivalent from an EV perspective:
Option A. A 0% chance of causing a harmful outcome, but in expectation will cause +10 utility
Option B. A 20% chance of causing a harmful outcome, but in expectation will cause +10 utility
It seems really bizarre to not prefer Option A. But if I prefer Option A, I’m just accepting risk aversion to at least some extent. But what if the numbers slip a little more?
Scenario 2
Option A. A 0% chance of causing a harmful outcome, but in expectation will cause +9.9999 utility
Option B. A 20% chance of causing a harmful outcome, but in expectation will cause +10 utility
Do I really want to take a 20% chance on causing harm in exchange for 0.001% gain in utility caused?
Scenario 3
Option A. A 0% chance of causing a harmful outcome, but in expectation will cause +5 utility
Option B. A 99.99999% chance of causing a harmful outcome, but in expectation will cause +10 utility
Do I really want to be exceedingly likely to cause harm, in exchange for a 100% gain in utility?
I don’t know the answers to the above scenarios, but I think it feels like just saying “the EV is X” without reference to the downside risk misses a massive part of the picture. It seems much better to say “the expected range of outcomes are a 20% chance of really bad stuff happening, a 70% chance of nothing happening and a 10% of a really really great outcome, which all averages out to an >0 average”. This is meaningfully different than saying “no downside risk, and a 10% chance of a pretty good outcome, so >0 average”.
I think that risk aversion is pretty important, but even if it isn’t incorporated into people’s thinking at all, it really doesn’t feel like EV produces a number I can take at face value, and that makes me feel like EV isn’t actually that simple.
The place where I currently see this happening the most is naive expected value maximization in reasoning about animal welfare — I feel like I’ve seen an uptick in “I think there is a 52% chance these animals live net negative lives, so we should do major irreversible things to reduce their population”. But it’s pretty easy to imagine doing those things being harmful, or your efforts backfiring, etc. in ways that cause harm.
I think this is wrong, and this intuition that many people have derives from a psychological mistake. Essentially everything in life has diminishing marginal utility, so it almost always makes sense to be risk averse. So it’s intuitive that you should be risk averse with respect to expected utility. But that doesn’t make any logical sense—by definition, you don’t have diminishing marginal utility of utility. Your utility function already accounts for risk aversion. Being risk averse with respect to utility is double-counting.
This is a valid statement but non-responsive to the actual post. The argument is that there is intuitive appeal in having a utility function with a discontinuity at zero (ie a jump in disutility from causing harm), and ~standard EV maximisation does not accommodate that intuition. That is a totally separate normative claim from arguing that we should encode diminishing marginal utility.
I don’t think this is quite what I’m referring to, but I can’t quite tell! But my quick read is we are talking about different things (I think because I used the word utility very casually). I’m not talking about my own utility function with regard to some action, but the potential outcomes of that action on others, and I don’t know if I’m embracing risk aversion views as much as relating to their appeal.
Or maybe I’m misunderstanding, and you’re just rejecting the conclusion that there is a moral difference between taking, say, an action with +1 EV and a 20% chance of causing harm and an action with +1EV and a 0% chance of causing harm / think I just shouldn’t care about that difference?
In retrospect my comment was poorly thought-out, I think you’re right that it’s not directly addressing your scenarios.
I think there are two separate issues with my comment:
My comment was about being risk-averse with respect to utility; your quick take was about wanting to avoid causing harm; those aren’t necessarily the same thing.
You can self-consistently believe in diminishing marginal utility of welfare, i.e., your utility function isn’t just “utility = sum(welfare)”. And the way your quick take used the word “utility”, you really meant something more like “welfare” (it sounds like this is what you’re saying in your reply comment).
RE #1, my sense is that “person is risk-averse with respect to utility” is isomorphic to “person disprefers a lottery with a possibility of doing harm, even if it has the same expected utility as a purely-positive lottery”. Or like, I think the person is making the same mistake in these two scenarios. But it’s not immediately obvious that these are isomorphic and I’m not 100% sure it’s true. Now I kind of want to see if I can come up with a proof but I would need to take some time to dig into the problem.
RE #2, I do in fact believe that utility = welfare, but that’s a whole other discussion and it’s not what I was trying to get at with my original comment, which means I think my comment missed the mark.
Depends on what you mean by “EV”. I do reject that conclusion if by EV you mean welfare. If by EV you mean something like “money”, then yeah I think money has diminishing marginal utility and you shouldn’t just maximized expected money.
I wish people would talk more about “sensitivity analysis”.
Your parameter estimates are just that, estimates. They probably result from intuitions or napkin math. They probably aren’t that precise. It’s easy to imagine a reasonable person generating different estimates in many cases.
If a relatively small change in parameters would lead to a relatively large change in the EV (example: in Scenario 3, just estimate the “probability of harm” a teensy bit different so it has a few more 9s, and the action looks far less attractive!) — then you should either (a) choose a different action, or (b) validate your estimates quite thoroughly since the VoI is very high, and beware of the Unilateralist’s Curse in this scenario, since other actors may be making parallel estimates for the action in question.
(I’m guessing you mean difference-making risk aversion here, based on your options being implicitly compared to doing nothing.)
When considering the potential of larger indirect effects on wild invertebrates, the far future and other butterfly effects, which interventions do you think look good (better than doing nothing) on difference-making risk aversion (or difference-making ambiguity aversion)?
(I suspect there are none for modest levels of difference-making risk/ambiguity aversion, and we should be thinking about difference-making in different ways.)
I think I mean something slightly different than difference-making risk aversion, but I see what you’re saying. I don’t even know if I’m arguing against EV maximization—more just trying to point out that EV alone doesn’t feel like it fully captures the picture of the value I care about (e.g. likelihood of causing harm relative to doing nothing feels like another important thing). I think specifically, that there are plausible circumstances where I am more likely than not to cause additional harm, and in expectation that action has positive EV, feels concerning. I imagine lots of AI risk work could be like this: doing some research project has some strong chance of advancing capabilities a bit (high probability of a bit of negative value), but maybe a very small chance of massively reducing risk (low probability of tons of positive value). The EV looks good, but my median outcome will be the world being worse than it was if I hadn’t done anything.
Ok, that makes sense. I’d guess butterfly effects would be neutral in the median difference. The same could be the case for indirect effects on wild animals and the far future, although I’d say it’s highly ambiguous (imprecise probabilities) and something to be clueless about, and not precisely neutral about.
Would you say you care about the overall distribution of differences, too, and not just the median and the EV?
Probably, but not sure! Yeah, the above is definitely ignoring cluelessness considerations, on which I don’t have any particularly strong opinion.
I think this critique misses how EV maximization works in a world with many actors taking uncorrelated risks.
Consider your Scenario 2: Individual actors choosing between Option A (0% harm chance, +9.9999 utility) vs Option B (20% harm chance, +10 utility). If we have 1000 altruistic actors each making independent choices with similar profiles, and they all choose Option B (higher EV), we’d expect:
800 successful outcomes (+8000 utility)
200 harmful outcomes (negative utility)
Net positive impact far exceeding what we’d get if everyone chose the “safe” option
This is portfolio theory applied to altruism. Just as index funds maximize returns by holding many uncorrelated assets, the altruistic community maximizes impact when individuals make risk-neutral EV calculations on independent projects.
The key caveats:
For large actors (major foundations, governments, AI companies, etc.), risk aversion makes more sense since their failures aren’t offset by others’ successes
For correlated risks (like your animal welfare example where many actors might simultaneously cause harm based on shared wrong beliefs), we need more caution
But for most EA individuals working on diverse, independent projects? Risk-neutral EV maximization is exactly what we want everyone doing. The portfolio effect means we’ll get the best aggregate outcome even though some individual bets will fail.
Are the projects of most EA individuals truly independent in the sense of their EVs being essentially uncorrelated with each other? That would be surprising to me, given that many of those projects are conditional on positive evaluation from a small number of funders, and many arose out of the same meta (so would be expected to have meaningful correlations with other active projects).
So my prediction is that most EA stuff falls into one of your two caveats. What I don’t have a good sense of is how correlated the average EA work is, and thus the degree of caution / risk aversion implied by the caveats.
In theory I agree with this, but in practise I personally think “Risk-neutral EV maximisation” can lead to bets which are far worse than they appear to be. This is because I think we often massively overrate the EV of “hits based approaches”.
Generally I think the lower probability of a bet, the higher chance there is of that EV being wrong and lower than stated. I’m keen to see evidence of high risk bets turning out well once in a while before I’m convinced that they really do have the claimed EVs...
Then your issue is with systemically flawed reasoning overestimating the likelihood of low-probability events. The solution for that would be to adjust by some factor that adjusts for this systemic epistemic bias, and then proceed with risk-neutral EV maximization (again, with the caveats that I had mentioned in my initial comment).
I think this is true as a response in certain cases, but many philanthropic interventions probably aren’t tried enough times to get the sample size and lots of communities are small. It’s pretty easy to imagine a situation like:
You and a handful of other people make some positive EV bets.
The median outcome from doing this is the world is worse, and all of the attempts at these bets end up neutral or negative.
The positive EV is never realized and the world is worse on average, despite both the individuals and the ecosystem being +EV.
It seems like this response would imply you should only do EV maximization if your movement is large (or that its impact is reliably predictable if the movement is large).
But I do think this is a fair point overall — though you could imagine a large system of interventions with the same features I describe that would have the same issues as a whole.
You’re absolutely right in my opinion Abraham. EV is a mathematical convenience and simplification that might work most of the time, but certainly not all.
I feel words like “expected value” are confusing & miselading (nothing expected about it—it’s a misnomer and accident of history/ translation from french that it’s called that), and this community would do well to avoid unnecessary jargon. I need to write up a post about this.
If we substituted the word “expected value” with mean or average, which is literally what it is (and I meet so many people in this community who get this wrong), it would be just as accurate but much more easier to understand, even to technical people. And many people understand that mean is not always the most appropriate statistic to use espescially with skewed distributions, e.g. reporting on incomes is often done with median, and that’s perfectly fine.
Expected value should also not be confused with expected utility (or as I prefer the mean utility). You use the terms interchangeably in your post as do many people in this community, it’s fine in causal conversation but it is worth being specific when necessary in discussions like this. I suspect because the word “value” in expected value implies some humanistic desireable ideal, but that’s what the word utility is for.
You also say use the term “in expectation” to mean the most likeliest outcome (i.e. the mode outcome) or more than 50%. I’m not familiar with that langauge—maybe it is accurate, but to me it is confusing when you can simply say the mode or the likeliest outcome.
So ultimately I suspect your dilemmas boil down to the recognition that there are circumstances where mean, mode, median, (and in fact even more statistics) can be maximised and we get different optimal decisions depending on the statistic we use.
At the end of the day we want to: maximise utility. But if utility has a probability distribution, it’s hard to maxmise without using some statistic (mean, mode, median etc) that maps a probability distribution to a real number. Using mean (EV) is not completely arbitrary I believe (it has some nice preperties), but it’s not fully settled. I personally think it has problems.