Who are the 16 people you surveyed to determine the disvalue of reducing people’s freedom to buy sugary drinks?
[EDITED TO ADD: this comment burys the lede a bit, in a great-grandchild I do some napkin math that, if correct, indicates that the survey results mean that the reduced value of freedom entirely negates the health benefits]
OK I’m really confused—you calculate ~0.001 DALYs (which i guess is ~9 disability-adjusted life-hours) lost per person to eliminating people’s freedom to consume sugary drinks, adjust that down because you’re not eliminating it but just restricting it, make a second adjustment which I don’t understand but which I’ll assume is OK, multiply by the population of the average country to get a total number of DALYs, then:
Fourthly and finally, we divide by overall diabetes mellitus type 2 disease burden, thus creating an estimate −0.001% to be used as a downward adjustment on impact.
But you estimate that taxing sugary drinks won’t eliminate the DMT2 disease burden, but instead reduce it by 0.02%. So shouldn’t this factor instead be 0.001% / 0.02% = 5%?
Let me try to do a rough calculation myself: if you world-wide banned sugary drinks, each person would lose 0.001 DALYs total over the rest of their lives [EDIT: this is wrong, it’s per annum, see this comment for a corrected version of the following calculation].
What’s the disease burden caused by DMT2? Your report says roughly 90 million DALYs, I’m going to assume that’s per year (you probably say this somewhere or it’s probably an obvious convention, but I couldn’t find it easily and don’t know the conventions in this field). The global average age is ~30 and average life expectancy is ~70, so let’s multiply that by 40 remaining years to say 3,600 million DALYs of total DMT2 burden for the present population over the rest of their lives. Banning sugary drinks would reduce that burden by 5%, for a gain of 180 million DALYs. There are ~8 billion people on earth, so that’s 0.02 DALYs per person gained by banning sugary drinks.
So loss of freedom of 0.001 DALYs reduces the benefit of 0.02 DALYs by 0.001 / 0.02 = 5%, agreeing with my guesstimate above (assuming that the ratio is the same for a tax vs an all-out ban, which seems right to first order).
FWIW I’m also suspicious of the 0.001 DALYs per person number.
AFAICT, the way you get it is by combining two methods: method 1 is to ask people a chain of questions like “as a fraction of death, how bad is life imprisonment”, “as a fraction of life imprisonment, how bad is not being able to eat tasty stuff”, “as a fraction of not being able to eat tasty stuff, how bad is not being able to have sugary drinks”, multiply their answers to get how bad losing sugary drinks is as a fraction of dying, and then multiply by the fraction 64⁄74 (for remaining life years? this was opaque to me), to get a DALY loss of 0.016 +/- 0.009 [1]. You then do method 2: ask people how much of their annual income they’d give up to get a 1-year exemption from a ban on drinking sugary drinks, take the binary logarithm of 1 + that fraction, and multiply by 2 to get DALY loss. This gives you a loss of 0.0012 +/- 0.0009 [1]. You then average each respondent’s result from each method to get a per-respondent DALY loss estimate, before aggregating that accross respondents. Because the standard deviation of responses from method 1 is 10 times higher than that of method 2 [2], you weight method 2 10x higher in the per-respondent average, meaning that the overall loss is basically just that of method 2.
But I don’t think you’re right to conclude that method 2 is more accurate than method 1: it’s just that method 2 gives ~10x smaller results for whatever reason, so it makes sense that its error is also ~10x smaller. If you look at the spread in responses as a fraction of the mean response, methods 1 and 2 are pretty close (if anything, it looks like method 1 is a bit more precise). If you instead weighted the methods equally, you would get 3x the per-person DALY loss [3], and if I’m right in the parent comment, that would net out to a 15% reduction in the value of the program.
(also more fundamentally, the fact that the methods give 10x different values suggests that they plausibly are just measuring different things, and we should be unsure which (if either) is actually measuring the disvalue of the loss of freedom to drink sugary drinks)
[1] My error here is the standard error of the mean of each result: basically, how much we’d expect our calculated mean to vary if we resampled. It’s equal to the empirical standard deviation divided by the square root of the number of samples (which is 4).
[2] You also list one benefit of method 1 and one benefit of method 2, which I’m assuming cancel out in your considerations.
[3] Sanity check: the mean of the first method is 10x bigger than the second method, previously we were ~ignoring the first method, now we’re taking the geometric mean, and the geometric mean of 1 and 10 is 3 (because 3^2 is about 10), so this looks right.
Wait: your survey numbers are for DALYs lost per year. So if the disease burden is 90 million DALYs per year, banning sugary drinks gives a benefit of 0.0006 DALYs per person-year, compared to 0.001 DALYs lost per person-year, meaning that the loss of freedom reduces the benefit by 167% (or 500% if you believe this comment). So now I’m really curious how your adjustments are bringing that down so much.
EDIT: reading the full report, the 0.02% reduction in diabetes burden is from eliminating sugary drink consumption in a single country. I’ve updated my comments below to correct for that
Overall, we expect a 100% reduction in sugar-sweetened beverages consumption in a single country to reduce the global disease burden of diabetes mellitus type 2 by 0.02%.
I’m also confused here, but I get different numbers than you do.
My BOTEC:
For 100% elimination of sugary beverages in all countries, the benefits seem like they’d be 0.0002*193*90,000,000 = 3,474,00 DALYs averted /year
For 100% elimination of sugary drinks, the costs in reduced freedom seem like they’d be 8,100,000,000 * 0.001 = 8,100,000 DALYs/year
Sounds like you roughly agree with me − 8.1 / 3.5 = 230%, which is close to 167%. Difference is I use the 5% reduction number for proportion of burden due to sugary drinks, getting 90 mil / 20 = 4.5 mil, 8.1 / 4.5 = 180%, and the rest is error built into these calcs.
Another way to look at this is that according to the report’s numbers, a 100% reduction in sugary beverages would prevent 0.02%*193 = 1 in 26 cases of diabetes. But it would also make life 1/1000th worse for everyone.
To answer the various issues you raised (in order of when they were posted):
(1) We surveyed EAs, via the forum mainly. Obviously not a particularly representative sample, but bigger surveys are more expensive (even if you do them online via e.g. Lucid) and my sense coming in was that this was unlikely to affect the CEA materially (a lot of people don’t have libertarian intuitions, for better or for worse), and so we didn’t commit too much time or money towards this issue. I will say that I suspect EAs – being highly educated and more likely to be concerned with abstract considerations like freedom – the estimate may higher than the population average; this would be consistent with larger panel data in political science on attitudes with respect to abstract concerns like democracy, civil rights, rule of law etc.
As for relative accuracy of the two methods – assuming there’s a true mean both are trying to get at, my view is that for the method with far higher variance, the sample mean we calculate is almost certainly much further off from the true mean, and I would be uncomfortable using it for critical calculations, if that makes sense.
(2) For the adjustments, the first, as you note, is just for the fact that sugary drinks are merely being taxed, rather than being banned. The second is just for diminishing marginal returns to freedom on income – the basic intuition here is that your freedom/ability to (say) read books increases as your income increases and you can buy more books; but the increase from 0->10 is far different from 1000->1010, since we would intuitively say there is a massive increase in your freedom to read books in the first case (you can read at all!), and a fairly marginal one in the latter. On top of this, there’s just the fact that income is no longer the binding constraint at high levels of income (we lack time to read, not money with which to buy books with).
(3) The calculation of an overall downward adjustment (the issue you raised with “But you estimate that taxing sugary drinks won’t eliminate the DMT2 disease burden, but instead reduce it by 0.02%. So shouldn’t this factor instead be 0.001% / 0.02% = 5%?”). I think you’re right! This is almost certainly a mistake we made – I’ll have to go recrunch the numbers myself (also noticed that there are a number of issues – probability of success, and also substitution with diet coke – which should be factored in). Thanks for flagging this out!
(4) On the per annum issue: In our analysis of tractability, we’ve generally stuck to trying to calculate “proportion of problem solved in a single year”, and bring in how said benefits persist in a different section. Similarly, for the adjustments, we stick to the per annum numbers.
(5) On the issue of whether the intervention is potentially net negative given BOTECs if we compare headline numbers of disease burden of diabetes * effect of banning SSBs against global population * 0.001. Beyond potential a prior scepticism as to whether such a significant number of people not dying or suffering ill health from diabetes really is less valuable than loss of freedom to drink sugar drinks, there are a number of adjustments here that are probably important (the massive cardiovascular benefits, economic impacts, plus deliberate geographic prioritization increasing cost-effectiveness).
Beyond potential a prior scepticism as to whether such a significant number of people not dying or suffering ill health from diabetes really is less valuable than loss of freedom to drink sugar drinks
I guess I want to add something here about why one would have the opposite prior: by and large, a decent model of people is that when they make decisions, they roughly weigh costs and benefits to themselves (or follow a policy that they adopted when weighing costs and benefits). Diabetes is mostly a cost to oneself. At least in the US, people are broadly aware that drinking sugary drinks makes you a bit less healthy, via an increased chance of obesity, diabetes, heart disease etc. But people drink them anyway because they’re tasty—that is, in their judgement, the value of being able to drink them is higher than the health risk.
It seems like you have a strong prior that people are wrong about this, and that they significantly underweight the health impacts of soft drinks, such that limiting their intake by 20% is worth it to reduce their risk of getting diabetes by 1%. This isn’t impossible—it could be that people don’t know that sugary drinks are unhealthy (of course, people could also overestimate how unhealthy they are), or that socialized healthcare means that there are massive externalities to diabetes cases—but I didn’t see any arguments to that effect in your executive summary.
(1) Per the formula (1-ABS(((1000/(0.83*1.2))-(1000/0.83))/(1000/0.83)))^0.1, it would appear that we have 98.2% remaining freedom, and a 1.8% reduction?
(2) I think you’re right on the averaging issue—in a number of other areas (e.g. calculating probabilities from a bunch of reference classes), we’ve tended to use the geomean, but that’s for extrapolating from estimates to the true value, as opposed to true individual means to the true population means. The other related issue, however, is whether the sample is representative, and whether we think that highly educated people are more likely to report caring about abstract concerns. Will have to think about this, but thanks for the feedback!
(3) As to your larger point, I’m not sure if it’s reasonable to interpret individual behaviour when it comes to trading off short term against long term gains as maximizing, given time inconsistent preferences (relative to valuing your welfare equally at all times), people just not thinking too much about daily choices, lack of awareness of the precise degree of risk (even if the risk were 10x or 0.1x, it’s not like you would likely shift a meaningful shift in behaviour), and of course motivated reasoning.
(1) Sorry, I was copy-pasting the formula in the spreadsheet, but missed the extra 0.1 factor you added at the end. First of all, I still think the factor of 50 you’re shaving off due to diminishing marginal returns seems quite extreme, given the lack of articulable justification for why it should be so big. I guess the extra factor of 10 is because diet cola exists? I’m not sure why you’re adding that—as I mentioned, the questions you asked people already mentioned that people could substitute to diet sodas. Quoting from the instructions to participants of cell E1 of your survey spreadsheet, “Note that this does not include artificially-sweetened beverages (i.e. Coke Zero, Diet Pepsi, Sprite Zero etc) – you would still be able to drink those.”.
(2) Makes sense. Regarding composition, we could look at polling of soda taxes to get a sense for this. One thing to note is that politicians typically don’t implement these taxes (hence you looking into lobbying for them), suggesting that they’re not very popular. Of the twopolls I could find, it seemed like either there were no significant differences between groups, or that Republicans were more opposed to soda taxes than Democrats. Given that the vast majority of American EAs are Democrats, this suggests that the poll could be underrating the disutility of reduced sugar consumption.
(3) Note that many of these factors go both ways—people could be motivated to appear self-abnegating and healthy, think sugary drinks are less healthy than they are (anecdotally, I’ve asked two people how much diabetes would be reduced if sugary drinks didn’t exist, and they both overestimated relative to your cited number), or not think about how sugary drinks are actually OK. It’s indeed plausible that people value their future selves less than they ought to according to standard utilitarianism, but I don’t think it’s a priori clear that they do so by a massive factor.
(1) Fair enough re sample (altho it obviously limits how much of a conclusion you can draw). Re: the different variances, I basically dispute that the cash value method has a meaningfully lower variance than the hypothetical sequence method, because the relevant error is relevant to the mean. That said, this factor of 3 is the smallest issue I have with your calculation.
(2) Could you show the calculations (perhaps in a simplified format, like I did)? The tax vs ban seems like it should affect the value of freedom similarly to how it affects the disease burden reduction, and it’s very unclear to me what you’re actually doing to adjust for diminishing marginal returns to freedom—and I’m skeptical that the heavy discount relative to the BOTEC model is justified.
(3) Fair enough! Easy to make this sort of slip-up in such a big survey [EDIT: by “such a big survey” I meant “such a big report”—surveys were just on the mind]
(4) Yeah, the per annum issue was just a problem with my BOTEC, I have no particular reason to think you got it wrong (other than the extreme divergence from the BOTEC).
(5) For what it’s worth I don’t share your skepticism that taxing sugary drinks could be net negative. At any rate: I’d guess the factors you first thought of are probably the most important (deferring to your process of creating the initial report), and I suppose that there are probably further effects that go either way—especially since, as you noted, cost effectiveness analyses tend to look less promising as more effort is put into them.
(1) One issue we’ve had issues with is large variance in estimates—and we typically try to use the geometric (rather than arithmetic mean) to better capture the differences in magnitudes. Our previous average of different people’s estimates used a pure arithmetric mean (because of the presence of zeroes—people didn’t value drinking sugary drinks at all, I guess).
Partly because of our discussion on weighing, I tried a different approach of getting a geomean of the non-zeroes, and then creating a weighted average with the zeroes. The results are pretty sensitive − 0.0002 with this mixed method vs 0.001 with a pure arithmetric mean. I do think the former is probably the better method, insofar as arithmetric means are too sensitive to the high-end number, and have updated accordingly—but I’m not sure if there’s a good answer one way or another!
(2) For diminishing marginal returns—I basically think of it in terms of a graph (x axis is number of sugary drinks you can buy with your current income, y-axis is “freedom of choice”), and I take the relationship to be y=x^0.1. The proportional reduction in the number of sugar drinks you can buy is estimated like this: (a) calculating how much you can buy without the tax (an arbitrary USD 1000, divided by prevailing prices sourced from Walmart), (b) calculating how much you can buy after the tax (USD 1000, divided by prices subject to 20% tax), and then (c) taking ABS((b-a)/a) to get the proportional reduction (0.17, for a 20% price rise). (d) 1-c then gives how much you can still buy, proportionally (0.83). (d) Then, using the y=x^0.1 formula, we take (1-d)^0.1 to find that 98% of “freedom of choice” still remains, and correspondingly, there was a 2% reduction.
Obviously, a whole bunch of assumptions are made, most significantly about the curve and DMR—any thoughts you have would be welcome!
Then, using the y=x^0.1 formula, we take (1-d)^0.1 to find that 98% of “freedom of choice” still remains, and correspondingly, there was a 2% reduction.
The number in the sheet is a 0.2% reduction, not a 2% reduction. [EDIT: my bad, it’s a 2% reduction, there’s just another factor of 10 reduction that I mistakenly lumped into that]
I still disagree with your belief that the accuracy of the iterated questions format was lower than the accuracy of the fraction of income format—both questions had standard deviations that were approximately the same multiple of their means.
I think your original strategy of aggregating across the population using the arithmetic mean made sense, and don’t understand what the justification is supposed to be for replacing it with a geometric mean [1]. Concretely, imagine a decision that affects two friends lives, making one 50% worse, and the other 0.005% worse. Presumably you wouldn’t take the geometric mean and say “this basically makes both your lives 0.5% worse, which is not very much”. Instead you might conclude that your friends are different in some way. Similarly, it seems like probably some people like sugary drinks and others don’t, causing significant variation in how much they care about sugary drinks being banned.
As DMR said, that curve seems kind of weird to me—it seems like an unjustified assumption is being used to cut a BOTEC by a factor of 50, which strikes me as suspicious. The real curve is presumably not linear (because otherwise people would buy more sugary drinks on the margin), but intuitively I feel like a factor of 5 adjustment makes way more sense than a factor of 50.
By my analysis of your sheet, if you use a factor of 5 rather than 50 for the decreasing marginal utility, and use the arithmetic mean rather than the geometric mean to aggregate across participants, you get the disutility of freedom as 500% higher than the gains. If you also weight both estimation methods equally, it goes up to 1,100% - which is bigger enough than my BOTEC that I worry you might be making some errors in the opposite direction?
[1] Consider that this analysis is done in the genre of a utilitarian calculation, which usually uses the arithmetic mean of welfare rather than the geometric mean, as is used implicitly in the disease reduction component.
Regarding the choice of means, I personally think that the arithmetic mean may make more sense in this case. Suppose you had 3 respondents who gave responses of 0.1, 0.001, and 0.001 as their DALYs from full banning of sugary drinks. If we assume that the individuals are accurately responding regarding their views and (as you are though the rest of this analysis) that utility gains and losses can be linearly aggregated across different individuals, and then the total harm that occurs to the three individuals in the case of a drink ban is 0.102 DALYs. If you had used the geometric mean instead, you would estimate that the total harm from a drink ban was just 0.014 DALYs.
It seems that an arithmetic mean would make more sense if one thinks that differences between the survey respondents’ views reflected true underlying differences in preferences, whereas a geometric mean would make more sense if one thinks that differences between the survey respondents’ views reflected noise in estimates of similar underlying preferences.
As mentioned to Daniel in the other thread: I think both of you are right on the averaging issue—in a number of other areas (e.g. calculating probabilities from a bunch of reference classes), we’ve tended to use the geomean, but that’s for extrapolating from estimates to the true value, as opposed to true individual means to the true population means. The other related issue, however, is whether the sample is representative, and whether we think that highly educated people are more likely to report caring about abstract concerns. Again, will have to think about this, but grateful for the feedback!
Very interesting, thanks for the detailed explanation! I’ll be curious to see what Daniel says, but my intuition is that y=x^0.1 is a generous assumption for the shape of the freedom of choice utility curve. I do agree that I’d expect utility losses from freedom of choice to be nonlinear, but the function you’re using would imply that 80% of freedom of choice is preserved when sugary drink consumption has been cut by 90%. Moreover, my intuition is that the welfare losses due to reduced sugary drink consumption are also partially due to not getting tasty drinks anymore (thinking for myself, I wouldn’t really mind not having sugary drinks, but I would definitely be sad in the similar case of not having sugary foods). That component of the utility losses seems like it would be closer to linear.
also noticed that there are a number of issues – probability of success, and also substitution with diet coke – which should be factored in
Just reread this—surely these don’t need to be factored in? Probability of success affects the numerator and the denominator equally, and your poll respondents probably already knew about diet coke.
Who are the 16 people you surveyed to determine the disvalue of reducing people’s freedom to buy sugary drinks?
[EDITED TO ADD: this comment burys the lede a bit, in a great-grandchild I do some napkin math that, if correct, indicates that the survey results mean that the reduced value of freedom entirely negates the health benefits]
OK I’m really confused—you calculate ~0.001 DALYs (which i guess is ~9 disability-adjusted life-hours) lost per person to eliminating people’s freedom to consume sugary drinks, adjust that down because you’re not eliminating it but just restricting it, make a second adjustment which I don’t understand but which I’ll assume is OK, multiply by the population of the average country to get a total number of DALYs, then:
But you estimate that taxing sugary drinks won’t eliminate the DMT2 disease burden, but instead reduce it by 0.02%. So shouldn’t this factor instead be 0.001% / 0.02% = 5%?
Let me try to do a rough calculation myself: if you world-wide banned sugary drinks, each person would lose 0.001 DALYs total over the rest of their lives [EDIT: this is wrong, it’s per annum, see this comment for a corrected version of the following calculation].
What’s the disease burden caused by DMT2? Your report says roughly 90 million DALYs, I’m going to assume that’s per year (you probably say this somewhere or it’s probably an obvious convention, but I couldn’t find it easily and don’t know the conventions in this field). The global average age is ~30 and average life expectancy is ~70, so let’s multiply that by 40 remaining years to say 3,600 million DALYs of total DMT2 burden for the present population over the rest of their lives. Banning sugary drinks would reduce that burden by 5%, for a gain of 180 million DALYs. There are ~8 billion people on earth, so that’s 0.02 DALYs per person gained by banning sugary drinks.
So loss of freedom of 0.001 DALYs reduces the benefit of 0.02 DALYs by 0.001 / 0.02 = 5%, agreeing with my guesstimate above (assuming that the ratio is the same for a tax vs an all-out ban, which seems right to first order).
FWIW I’m also suspicious of the 0.001 DALYs per person number.
AFAICT, the way you get it is by combining two methods: method 1 is to ask people a chain of questions like “as a fraction of death, how bad is life imprisonment”, “as a fraction of life imprisonment, how bad is not being able to eat tasty stuff”, “as a fraction of not being able to eat tasty stuff, how bad is not being able to have sugary drinks”, multiply their answers to get how bad losing sugary drinks is as a fraction of dying, and then multiply by the fraction 64⁄74 (for remaining life years? this was opaque to me), to get a DALY loss of 0.016 +/- 0.009 [1]. You then do method 2: ask people how much of their annual income they’d give up to get a 1-year exemption from a ban on drinking sugary drinks, take the binary logarithm of 1 + that fraction, and multiply by 2 to get DALY loss. This gives you a loss of 0.0012 +/- 0.0009 [1]. You then average each respondent’s result from each method to get a per-respondent DALY loss estimate, before aggregating that accross respondents. Because the standard deviation of responses from method 1 is 10 times higher than that of method 2 [2], you weight method 2 10x higher in the per-respondent average, meaning that the overall loss is basically just that of method 2.
But I don’t think you’re right to conclude that method 2 is more accurate than method 1: it’s just that method 2 gives ~10x smaller results for whatever reason, so it makes sense that its error is also ~10x smaller. If you look at the spread in responses as a fraction of the mean response, methods 1 and 2 are pretty close (if anything, it looks like method 1 is a bit more precise). If you instead weighted the methods equally, you would get 3x the per-person DALY loss [3], and if I’m right in the parent comment, that would net out to a 15% reduction in the value of the program.
(also more fundamentally, the fact that the methods give 10x different values suggests that they plausibly are just measuring different things, and we should be unsure which (if either) is actually measuring the disvalue of the loss of freedom to drink sugary drinks)
[1] My error here is the standard error of the mean of each result: basically, how much we’d expect our calculated mean to vary if we resampled. It’s equal to the empirical standard deviation divided by the square root of the number of samples (which is 4).
[2] You also list one benefit of method 1 and one benefit of method 2, which I’m assuming cancel out in your considerations.
[3] Sanity check: the mean of the first method is 10x bigger than the second method, previously we were ~ignoring the first method, now we’re taking the geometric mean, and the geometric mean of 1 and 10 is 3 (because 3^2 is about 10), so this looks right.
Wait: your survey numbers are for DALYs lost per year. So if the disease burden is 90 million DALYs per year, banning sugary drinks gives a benefit of 0.0006 DALYs per person-year, compared to 0.001 DALYs lost per person-year, meaning that the loss of freedom reduces the benefit by 167% (or 500% if you believe this comment). So now I’m really curious how your adjustments are bringing that down so much.
EDIT: reading the full report, the 0.02% reduction in diabetes burden is from eliminating sugary drink consumption in a single country. I’ve updated my comments below to correct for that
I’m also confused here, but I get different numbers than you do.
My BOTEC:
For 100% elimination of sugary beverages in all countries, the benefits seem like they’d be 0.0002*193*90,000,000 = 3,474,00 DALYs averted /year
For 100% elimination of sugary drinks, the costs in reduced freedom seem like they’d be 8,100,000,000 * 0.001 = 8,100,000 DALYs/year
So this looks net negative in expectation
Sounds like you roughly agree with me − 8.1 / 3.5 = 230%, which is close to 167%. Difference is I use the 5% reduction number for proportion of burden due to sugary drinks, getting 90 mil / 20 = 4.5 mil, 8.1 / 4.5 = 180%, and the rest is error built into these calcs.
Gotcha, makes sense!
Another way to look at this is that according to the report’s numbers, a 100% reduction in sugary beverages would prevent 0.02%*193 = 1 in 26 cases of diabetes. But it would also make life 1/1000th worse for everyone.
Hi Daniel!
To answer the various issues you raised (in order of when they were posted):
(1) We surveyed EAs, via the forum mainly. Obviously not a particularly representative sample, but bigger surveys are more expensive (even if you do them online via e.g. Lucid) and my sense coming in was that this was unlikely to affect the CEA materially (a lot of people don’t have libertarian intuitions, for better or for worse), and so we didn’t commit too much time or money towards this issue. I will say that I suspect EAs – being highly educated and more likely to be concerned with abstract considerations like freedom – the estimate may higher than the population average; this would be consistent with larger panel data in political science on attitudes with respect to abstract concerns like democracy, civil rights, rule of law etc.
As for relative accuracy of the two methods – assuming there’s a true mean both are trying to get at, my view is that for the method with far higher variance, the sample mean we calculate is almost certainly much further off from the true mean, and I would be uncomfortable using it for critical calculations, if that makes sense.
(2) For the adjustments, the first, as you note, is just for the fact that sugary drinks are merely being taxed, rather than being banned. The second is just for diminishing marginal returns to freedom on income – the basic intuition here is that your freedom/ability to (say) read books increases as your income increases and you can buy more books; but the increase from 0->10 is far different from 1000->1010, since we would intuitively say there is a massive increase in your freedom to read books in the first case (you can read at all!), and a fairly marginal one in the latter. On top of this, there’s just the fact that income is no longer the binding constraint at high levels of income (we lack time to read, not money with which to buy books with).
(3) The calculation of an overall downward adjustment (the issue you raised with “But you estimate that taxing sugary drinks won’t eliminate the DMT2 disease burden, but instead reduce it by 0.02%. So shouldn’t this factor instead be 0.001% / 0.02% = 5%?”). I think you’re right! This is almost certainly a mistake we made – I’ll have to go recrunch the numbers myself (also noticed that there are a number of issues – probability of success, and also substitution with diet coke – which should be factored in). Thanks for flagging this out!
(4) On the per annum issue: In our analysis of tractability, we’ve generally stuck to trying to calculate “proportion of problem solved in a single year”, and bring in how said benefits persist in a different section. Similarly, for the adjustments, we stick to the per annum numbers.
(5) On the issue of whether the intervention is potentially net negative given BOTECs if we compare headline numbers of disease burden of diabetes * effect of banning SSBs against global population * 0.001. Beyond potential a prior scepticism as to whether such a significant number of people not dying or suffering ill health from diabetes really is less valuable than loss of freedom to drink sugar drinks, there are a number of adjustments here that are probably important (the massive cardiovascular benefits, economic impacts, plus deliberate geographic prioritization increasing cost-effectiveness).
I guess I want to add something here about why one would have the opposite prior: by and large, a decent model of people is that when they make decisions, they roughly weigh costs and benefits to themselves (or follow a policy that they adopted when weighing costs and benefits). Diabetes is mostly a cost to oneself. At least in the US, people are broadly aware that drinking sugary drinks makes you a bit less healthy, via an increased chance of obesity, diabetes, heart disease etc. But people drink them anyway because they’re tasty—that is, in their judgement, the value of being able to drink them is higher than the health risk.
It seems like you have a strong prior that people are wrong about this, and that they significantly underweight the health impacts of soft drinks, such that limiting their intake by 20% is worth it to reduce their risk of getting diabetes by 1%. This isn’t impossible—it could be that people don’t know that sugary drinks are unhealthy (of course, people could also overestimate how unhealthy they are), or that socialized healthcare means that there are massive externalities to diabetes cases—but I didn’t see any arguments to that effect in your executive summary.
Consolidating over the number of comments:
(1) Per the formula (1-ABS(((1000/(0.83*1.2))-(1000/0.83))/(1000/0.83)))^0.1, it would appear that we have 98.2% remaining freedom, and a 1.8% reduction?
(2) I think you’re right on the averaging issue—in a number of other areas (e.g. calculating probabilities from a bunch of reference classes), we’ve tended to use the geomean, but that’s for extrapolating from estimates to the true value, as opposed to true individual means to the true population means. The other related issue, however, is whether the sample is representative, and whether we think that highly educated people are more likely to report caring about abstract concerns. Will have to think about this, but thanks for the feedback!
(3) As to your larger point, I’m not sure if it’s reasonable to interpret individual behaviour when it comes to trading off short term against long term gains as maximizing, given time inconsistent preferences (relative to valuing your welfare equally at all times), people just not thinking too much about daily choices, lack of awareness of the precise degree of risk (even if the risk were 10x or 0.1x, it’s not like you would likely shift a meaningful shift in behaviour), and of course motivated reasoning.
(1) Sorry, I was copy-pasting the formula in the spreadsheet, but missed the extra 0.1 factor you added at the end. First of all, I still think the factor of 50 you’re shaving off due to diminishing marginal returns seems quite extreme, given the lack of articulable justification for why it should be so big. I guess the extra factor of 10 is because diet cola exists? I’m not sure why you’re adding that—as I mentioned, the questions you asked people already mentioned that people could substitute to diet sodas. Quoting from the instructions to participants of cell E1 of your survey spreadsheet, “Note that this does not include artificially-sweetened beverages (i.e. Coke Zero, Diet Pepsi, Sprite Zero etc) – you would still be able to drink those.”.
(2) Makes sense. Regarding composition, we could look at polling of soda taxes to get a sense for this. One thing to note is that politicians typically don’t implement these taxes (hence you looking into lobbying for them), suggesting that they’re not very popular. Of the two polls I could find, it seemed like either there were no significant differences between groups, or that Republicans were more opposed to soda taxes than Democrats. Given that the vast majority of American EAs are Democrats, this suggests that the poll could be underrating the disutility of reduced sugar consumption.
(3) Note that many of these factors go both ways—people could be motivated to appear self-abnegating and healthy, think sugary drinks are less healthy than they are (anecdotally, I’ve asked two people how much diabetes would be reduced if sugary drinks didn’t exist, and they both overestimated relative to your cited number), or not think about how sugary drinks are actually OK. It’s indeed plausible that people value their future selves less than they ought to according to standard utilitarianism, but I don’t think it’s a priori clear that they do so by a massive factor.
(1) Fair enough re sample (altho it obviously limits how much of a conclusion you can draw). Re: the different variances, I basically dispute that the cash value method has a meaningfully lower variance than the hypothetical sequence method, because the relevant error is relevant to the mean. That said, this factor of 3 is the smallest issue I have with your calculation.
(2) Could you show the calculations (perhaps in a simplified format, like I did)? The tax vs ban seems like it should affect the value of freedom similarly to how it affects the disease burden reduction, and it’s very unclear to me what you’re actually doing to adjust for diminishing marginal returns to freedom—and I’m skeptical that the heavy discount relative to the BOTEC model is justified.
(3) Fair enough! Easy to make this sort of slip-up in such a big survey [EDIT: by “such a big survey” I meant “such a big report”—surveys were just on the mind]
(4) Yeah, the per annum issue was just a problem with my BOTEC, I have no particular reason to think you got it wrong (other than the extreme divergence from the BOTEC).
(5) For what it’s worth I don’t share your skepticism that taxing sugary drinks could be net negative. At any rate: I’d guess the factors you first thought of are probably the most important (deferring to your process of creating the initial report), and I suppose that there are probably further effects that go either way—especially since, as you noted, cost effectiveness analyses tend to look less promising as more effort is put into them.
Hi Daniel—thanks for all the previous feedback. I did a tentative update to the CEA (https://docs.google.com/spreadsheets/d/1kdnvaeP5iAUAF_Fgcf_ZgZDci2cYV02M51N7HMTMOBQ/edit#gid=1248986269), taking into account the points you raised plus some other considerations. I also separated out the analysis into its own tab, hopefully for better clarity.
(1) One issue we’ve had issues with is large variance in estimates—and we typically try to use the geometric (rather than arithmetic mean) to better capture the differences in magnitudes. Our previous average of different people’s estimates used a pure arithmetric mean (because of the presence of zeroes—people didn’t value drinking sugary drinks at all, I guess).
Partly because of our discussion on weighing, I tried a different approach of getting a geomean of the non-zeroes, and then creating a weighted average with the zeroes. The results are pretty sensitive − 0.0002 with this mixed method vs 0.001 with a pure arithmetric mean. I do think the former is probably the better method, insofar as arithmetric means are too sensitive to the high-end number, and have updated accordingly—but I’m not sure if there’s a good answer one way or another!
(2) For diminishing marginal returns—I basically think of it in terms of a graph (x axis is number of sugary drinks you can buy with your current income, y-axis is “freedom of choice”), and I take the relationship to be y=x^0.1. The proportional reduction in the number of sugar drinks you can buy is estimated like this: (a) calculating how much you can buy without the tax (an arbitrary USD 1000, divided by prevailing prices sourced from Walmart), (b) calculating how much you can buy after the tax (USD 1000, divided by prices subject to 20% tax), and then (c) taking ABS((b-a)/a) to get the proportional reduction (0.17, for a 20% price rise). (d) 1-c then gives how much you can still buy, proportionally (0.83). (d) Then, using the y=x^0.1 formula, we take (1-d)^0.1 to find that 98% of “freedom of choice” still remains, and correspondingly, there was a 2% reduction.
Obviously, a whole bunch of assumptions are made, most significantly about the curve and DMR—any thoughts you have would be welcome!
The number in the sheet is a 0.2% reduction, not a 2% reduction. [EDIT: my bad, it’s a 2% reduction, there’s just another factor of 10 reduction that I mistakenly lumped into that]
I still disagree with your belief that the accuracy of the iterated questions format was lower than the accuracy of the fraction of income format—both questions had standard deviations that were approximately the same multiple of their means.
I think your original strategy of aggregating across the population using the arithmetic mean made sense, and don’t understand what the justification is supposed to be for replacing it with a geometric mean [1]. Concretely, imagine a decision that affects two friends lives, making one 50% worse, and the other 0.005% worse. Presumably you wouldn’t take the geometric mean and say “this basically makes both your lives 0.5% worse, which is not very much”. Instead you might conclude that your friends are different in some way. Similarly, it seems like probably some people like sugary drinks and others don’t, causing significant variation in how much they care about sugary drinks being banned.
As DMR said, that curve seems kind of weird to me—it seems like an unjustified assumption is being used to cut a BOTEC by a factor of 50, which strikes me as suspicious. The real curve is presumably not linear (because otherwise people would buy more sugary drinks on the margin), but intuitively I feel like a factor of 5 adjustment makes way more sense than a factor of 50.
By my analysis of your sheet, if you use a factor of 5 rather than 50 for the decreasing marginal utility, and use the arithmetic mean rather than the geometric mean to aggregate across participants, you get the disutility of freedom as 500% higher than the gains. If you also weight both estimation methods equally, it goes up to 1,100% - which is bigger enough than my BOTEC that I worry you might be making some errors in the opposite direction?
[1] Consider that this analysis is done in the genre of a utilitarian calculation, which usually uses the arithmetic mean of welfare rather than the geometric mean, as is used implicitly in the disease reduction component.
Regarding the choice of means, I personally think that the arithmetic mean may make more sense in this case. Suppose you had 3 respondents who gave responses of 0.1, 0.001, and 0.001 as their DALYs from full banning of sugary drinks. If we assume that the individuals are accurately responding regarding their views and (as you are though the rest of this analysis) that utility gains and losses can be linearly aggregated across different individuals, and then the total harm that occurs to the three individuals in the case of a drink ban is 0.102 DALYs. If you had used the geometric mean instead, you would estimate that the total harm from a drink ban was just 0.014 DALYs.
It seems that an arithmetic mean would make more sense if one thinks that differences between the survey respondents’ views reflected true underlying differences in preferences, whereas a geometric mean would make more sense if one thinks that differences between the survey respondents’ views reflected noise in estimates of similar underlying preferences.
As mentioned to Daniel in the other thread: I think both of you are right on the averaging issue—in a number of other areas (e.g. calculating probabilities from a bunch of reference classes), we’ve tended to use the geomean, but that’s for extrapolating from estimates to the true value, as opposed to true individual means to the true population means. The other related issue, however, is whether the sample is representative, and whether we think that highly educated people are more likely to report caring about abstract concerns. Again, will have to think about this, but grateful for the feedback!
Very interesting, thanks for the detailed explanation! I’ll be curious to see what Daniel says, but my intuition is that y=x^0.1 is a generous assumption for the shape of the freedom of choice utility curve. I do agree that I’d expect utility losses from freedom of choice to be nonlinear, but the function you’re using would imply that 80% of freedom of choice is preserved when sugary drink consumption has been cut by 90%. Moreover, my intuition is that the welfare losses due to reduced sugary drink consumption are also partially due to not getting tasty drinks anymore (thinking for myself, I wouldn’t really mind not having sugary drinks, but I would definitely be sad in the similar case of not having sugary foods). That component of the utility losses seems like it would be closer to linear.
Just reread this—surely these don’t need to be factored in? Probability of success affects the numerator and the denominator equally, and your poll respondents probably already knew about diet coke.
Also in cell 1F of the results spreadsheet, “salty food” should be “sugary drinks”.