This was very common feedback, I should have added a section on it!
I disagree though. I think your example is perfect, with a denominator that spans many orders of magnitude
mean(cost) / mean(effect) is also wrong unless you have a constant cost. Consider for simplicity a case of constant effect of 1 life saved, and where the cost could be $10, $1000, or $10,000. mean(cost) / mean(effect) = $3670 per life saved, but the correct answer is 0.0337 lives saved per dollar = $29.67 per life saved.
I disagree, let’s say you have N interventions with that distribution of costs and effects and you fund all of them. The total cost/effect would be ∑costi∑effecti=N⋅mean(cost)N⋅1=mean(cost)≈$3670
Not sure how useful it is as an intuition pump, but here is an even more extreme/absurd example: if there is a 0.001% chance that the cost is 0 and a 99.999% chance that the cost is $1T, mean(effect/cost) would be ∞, even if the effect is super tiny, which is definitely not what we care about.
In general, as JoelMcGuire mentioned, it’s “a general issue where your simulation involves ratios”, taking the other ratio helps only insofar as it prevents the denominator from spanning as many orders of magnitude.
Does this make sense? Is there a better way to write it? Is it completely wrong? In general, I think we don’t care about the mean of X/Y, (that indeed can be dominated by cases where Y is really tiny), but about the expected total X / Y.
As an example, let’s say you have three interventions with that distribution, and they turn out to be perfectly distributed, you have total cost=$11,010 and total effect=3 so, as a funder that cares about expected value, $3670 is the value you care about.
That’s true if you spend money that way, but why would you spend money that way? Why would you spend less on the interventions that are more cost-effective? It makes more sense to spend a fixed budget. Given a 1⁄3 chance that the cost per life saved is $10, $1000, or $10,000, and you spend $29.67, then you save 1 life in expectation (= 1⁄3 * (29.67 / 10 + 29.67 / 1000 + 29.67 / 10,000)).
Not sure how useful it is as an intuition pump, but here is an even more extreme/absurd example: if there is a 0.001% chance that the cost is 0 and a 99.999% chance that the cost is $1T, mean(effect/cost) would be ∞
That’s a feature, not a bug. If something has positive value and zero cost, then you should spend zero dollars/resources to invoke the effect infinitely many times and produce infinite value (with probability 0.00001).
Not sure how useful it is as an intuition pump, but here is an even more extreme/absurd example: if there is a 0.001% chance that the cost is 0 and a 99.999% chance that the cost is $1T, mean(effect/cost) would be ∞, even if the effect is super tiny, which is definitely not what we care about.
I agree a non-null probability of null cost implies infinite mean cost-effectiveness (mean ratio between effect and cost). However:
For a null cost, you would have null effect, thus getting an indetermination of the type 0⁄0. Denoting as CEi the possible values of the ratio between effect and cost, for CE1 = 0⁄0, the mean cost-effectiveness would be:
CE_mu = (0/0 + CE2 + CE3 + … CEN)/N = 0⁄0.
In reality, the cost can never be (exactly) null, and numerical errors resulting from the possibility of a very low cost could be handled by increasing the number of Monte Carlo samples.
The effect can be null, but this could be handled by focussing on “effect”/”cost” (what we arguably care about) instead of “cost”/”effect”.
I think one should refer to “effect”/”cost” as cost-effectiveness, since more cost-effective interventions have greater ratios between the effect and cost.
Yeah, it’s a very theoretical example. There are things that could be modeled as very very low cost, like choosing A instead of B where both A and B cost $10, but indeed let’s focus on the other example.
I think we care about the expected marginal cost-effectiveness (i.e. mean(“marginal effect”/”marginal cost”)). Both mean(“total effect”)/mean(“total cost”) and mean(“total effect”/”total cost”) are good approximations if our budget is small, but they might not be if the budget has to be large for some reason.
This was very common feedback, I should have added a section on it!
I disagree though. I think your example is perfect, with a denominator that spans many orders of magnitude
I disagree, let’s say you have N interventions with that distribution of costs and effects and you fund all of them.
The total cost/effect would be ∑costi∑effecti=N⋅mean(cost)N⋅1=mean(cost)≈$3670
As an example, let’s say you have three interventions with that distribution, and they turn out to be perfectly distributed, you have
total cost=$11,010 and total effect=3 lives so, as a funder that cares about expected value, $3670 is the value you care about.
https://docs.google.com/spreadsheets/d/1yfK7J5V4rBUQ7-lWKrdXDNyI3NYjRouub4KwO2PVkuQ/edit?usp=sharing here is a spreadsheet with 100 cases.
Not sure how useful it is as an intuition pump, but here is an even more extreme/absurd example: if there is a 0.001% chance that the cost is 0 and a 99.999% chance that the cost is $1T, mean(effect/cost) would be ∞, even if the effect is super tiny, which is definitely not what we care about.
In general, as JoelMcGuire mentioned, it’s “a general issue where your simulation involves ratios”, taking the other ratio helps only insofar as it prevents the denominator from spanning as many orders of magnitude.
Does this make sense? Is there a better way to write it? Is it completely wrong?
In general, I think we don’t care about the mean of X/Y, (that indeed can be dominated by cases where Y is really tiny), but about the expected total X / Y.
That’s true if you spend money that way, but why would you spend money that way? Why would you spend less on the interventions that are more cost-effective? It makes more sense to spend a fixed budget. Given a 1⁄3 chance that the cost per life saved is $10, $1000, or $10,000, and you spend $29.67, then you save 1 life in expectation (= 1⁄3 * (29.67 / 10 + 29.67 / 1000 + 29.67 / 10,000)).
That’s a feature, not a bug. If something has positive value and zero cost, then you should spend zero dollars/resources to invoke the effect infinitely many times and produce infinite value (with probability 0.00001).
I agree a non-null probability of null cost implies infinite mean cost-effectiveness (mean ratio between effect and cost). However:
For a null cost, you would have null effect, thus getting an indetermination of the type 0⁄0. Denoting as CEi the possible values of the ratio between effect and cost, for CE1 = 0⁄0, the mean cost-effectiveness would be:
CE_mu = (0/0 + CE2 + CE3 + … CEN)/N = 0⁄0.
In reality, the cost can never be (exactly) null, and numerical errors resulting from the possibility of a very low cost could be handled by increasing the number of Monte Carlo samples.
The effect can be null, but this could be handled by focussing on “effect”/”cost” (what we arguably care about) instead of “cost”/”effect”.
I think one should refer to “effect”/”cost” as cost-effectiveness, since more cost-effective interventions have greater ratios between the effect and cost.
Yeah, it’s a very theoretical example.
There are things that could be modeled as very very low cost, like choosing A instead of B where both A and B cost $10, but indeed let’s focus on the other example.
Using “effect”/”cost” helps in many cases, but definitely not in all.
E.g. for policy intervention estimates of costs can vary by orders of magnitude: see https://forum.effectivealtruism.org/posts/h2N9qEbvQ6RHABcae/a-critical-review-of-open-philanthropy-s-bet-on-criminal?commentId=NajaYiQD7KhAJyBcp
I think we care about the expected marginal cost-effectiveness (i.e. mean(“marginal effect”/”marginal cost”)). Both mean(“total effect”)/mean(“total cost”) and mean(“total effect”/”total cost”) are good approximations if our budget is small, but they might not be if the budget has to be large for some reason.