I could be missing something but this sounds wrong to me. I think the actual objective is mean(effect / cost). effect / cost is the thing you care about, and if you’re uncertain, you should take the expectation over the thing you care about.mean(cost / effect) can give the wrong answer because it’s the reciprocal of what you care about.
mean(cost) / mean(effect) is also wrong unless you have a constant cost. Consider for simplicity a case of constant effect of 1 life saved, and where the cost could be $10, $1000, or $10,000. mean(cost) / mean(effect) = $3670 per life saved, but the correct answer is 0.0337 lives saved per dollar = $29.67 per life saved.
Coming back to this a year later, I wanted to thank you for this comment! It took me an embarrassingly long time to realize I was wrong about this, but I was! I’ve now edited the post.
If any new reader is confused by the comments pointing out this mistake, see footnote 1 and the appendix above.
Ok, so say you have a fixed budget. Then you want to maximise mean(total effect), which is equal to mean(budget/cost * unit effect)
… I agree.
Also, infinite expected values come from having some chance of doing the thing an infinite number of times, where the problem is clearly the assumption that the effect is equal to budget/cost * unit effect when this is actually true only in the limit of small numbers of additional interventions.
Also, Lorenzo’s proposal is ok when cost and effect are independent, while the error he identifies is still an error in this case.
The below is a reply to a previous version of the above comment.
I do not think we want to maximise mean(“effect”—“cost”).
Note “effect” and “cost” have different units, so they cannot be combined in that way. “Effect” refers to the outcome, whereas “cost” corresponds to the amount of resources we have to spend.
One might want to include “-cost” due to the desire of accounting for the counterfactual, but this is supposed to be included in “effect” = “factual effect”—“counterfactual effect”.
We want to maximise mean(“effect”) for “cost” ⇐ “maximum cost” (see this comment).
This was very common feedback, I should have added a section on it!
I disagree though. I think your example is perfect, with a denominator that spans many orders of magnitude
mean(cost) / mean(effect) is also wrong unless you have a constant cost. Consider for simplicity a case of constant effect of 1 life saved, and where the cost could be $10, $1000, or $10,000. mean(cost) / mean(effect) = $3670 per life saved, but the correct answer is 0.0337 lives saved per dollar = $29.67 per life saved.
I disagree, let’s say you have N interventions with that distribution of costs and effects and you fund all of them. The total cost/effect would be ∑costi∑effecti=N⋅mean(cost)N⋅1=mean(cost)≈$3670
Not sure how useful it is as an intuition pump, but here is an even more extreme/absurd example: if there is a 0.001% chance that the cost is 0 and a 99.999% chance that the cost is $1T, mean(effect/cost) would be ∞, even if the effect is super tiny, which is definitely not what we care about.
In general, as JoelMcGuire mentioned, it’s “a general issue where your simulation involves ratios”, taking the other ratio helps only insofar as it prevents the denominator from spanning as many orders of magnitude.
Does this make sense? Is there a better way to write it? Is it completely wrong? In general, I think we don’t care about the mean of X/Y, (that indeed can be dominated by cases where Y is really tiny), but about the expected total X / Y.
As an example, let’s say you have three interventions with that distribution, and they turn out to be perfectly distributed, you have total cost=$11,010 and total effect=3 so, as a funder that cares about expected value, $3670 is the value you care about.
That’s true if you spend money that way, but why would you spend money that way? Why would you spend less on the interventions that are more cost-effective? It makes more sense to spend a fixed budget. Given a 1⁄3 chance that the cost per life saved is $10, $1000, or $10,000, and you spend $29.67, then you save 1 life in expectation (= 1⁄3 * (29.67 / 10 + 29.67 / 1000 + 29.67 / 10,000)).
Not sure how useful it is as an intuition pump, but here is an even more extreme/absurd example: if there is a 0.001% chance that the cost is 0 and a 99.999% chance that the cost is $1T, mean(effect/cost) would be ∞
That’s a feature, not a bug. If something has positive value and zero cost, then you should spend zero dollars/resources to invoke the effect infinitely many times and produce infinite value (with probability 0.00001).
Not sure how useful it is as an intuition pump, but here is an even more extreme/absurd example: if there is a 0.001% chance that the cost is 0 and a 99.999% chance that the cost is $1T, mean(effect/cost) would be ∞, even if the effect is super tiny, which is definitely not what we care about.
I agree a non-null probability of null cost implies infinite mean cost-effectiveness (mean ratio between effect and cost). However:
For a null cost, you would have null effect, thus getting an indetermination of the type 0⁄0. Denoting as CEi the possible values of the ratio between effect and cost, for CE1 = 0⁄0, the mean cost-effectiveness would be:
CE_mu = (0/0 + CE2 + CE3 + … CEN)/N = 0⁄0.
In reality, the cost can never be (exactly) null, and numerical errors resulting from the possibility of a very low cost could be handled by increasing the number of Monte Carlo samples.
The effect can be null, but this could be handled by focussing on “effect”/”cost” (what we arguably care about) instead of “cost”/”effect”.
I think one should refer to “effect”/”cost” as cost-effectiveness, since more cost-effective interventions have greater ratios between the effect and cost.
Yeah, it’s a very theoretical example. There are things that could be modeled as very very low cost, like choosing A instead of B where both A and B cost $10, but indeed let’s focus on the other example.
I think we care about the expected marginal cost-effectiveness (i.e. mean(“marginal effect”/”marginal cost”)). Both mean(“total effect”)/mean(“total cost”) and mean(“total effect”/”total cost”) are good approximations if our budget is small, but they might not be if the budget has to be large for some reason.
Haha, I came up with that example as well. You’re thinking about this in the same way I did!
I think to say that one is the “actual objective” is not very rigorous. Although I’m saying this from a place of making that same argument. It does answer a valid question of “how much money should one donate to get an expected 1 unit of good” (which is also really easy to communicate, dollars per life saved is much easier to talk about than lives saved per dollar). I’ve been thinking about it for a while and put a comment under Edo Arad’s one.
As for the second point about simple going E(cost)E(effect). I agree that this is likely an error, and you have a good counterexample.
I could be missing something but this sounds wrong to me. I think the actual objective is
mean(effect / cost).effect / costis the thing you care about, and if you’re uncertain, you should take the expectation over the thing you care about.mean(cost / effect)can give the wrong answer because it’s the reciprocal of what you care about.mean(cost) / mean(effect)is also wrong unless you have a constant cost. Consider for simplicity a case of constant effect of 1 life saved, and where the cost could be $10, $1000, or $10,000.mean(cost) / mean(effect)= $3670 per life saved, but the correct answer is 0.0337 lives saved per dollar = $29.67 per life saved.Coming back to this a year later, I wanted to thank you for this comment! It took me an embarrassingly long time to realize I was wrong about this, but I was! I’ve now edited the post.
If any new reader is confused by the comments pointing out this mistake, see footnote 1 and the appendix above.
Ok, so say you have a fixed budget. Then you want to maximise mean(total effect), which is equal to mean(budget/cost * unit effect)
… I agree.
Also, infinite expected values come from having some chance of doing the thing an infinite number of times, where the problem is clearly the assumption that the effect is equal to budget/cost * unit effect when this is actually true only in the limit of small numbers of additional interventions.
Also, Lorenzo’s proposal is ok when cost and effect are independent, while the error he identifies is still an error in this case.
The below is a reply to a previous version of the above comment.
I do not think we want to maximise mean(“effect”—“cost”).
Note “effect” and “cost” have different units, so they cannot be combined in that way. “Effect” refers to the outcome, whereas “cost” corresponds to the amount of resources we have to spend.
One might want to include “-cost” due to the desire of accounting for the counterfactual, but this is supposed to be included in “effect” = “factual effect”—“counterfactual effect”.
We want to maximise mean(“effect”) for “cost” ⇐ “maximum cost” (see this comment).
Yeah, I was mentally substituting “effect” for “good” and “cost” for “bad”
This was very common feedback, I should have added a section on it!
I disagree though. I think your example is perfect, with a denominator that spans many orders of magnitude
I disagree, let’s say you have N interventions with that distribution of costs and effects and you fund all of them.
The total cost/effect would be ∑costi∑effecti=N⋅mean(cost)N⋅1=mean(cost)≈$3670
As an example, let’s say you have three interventions with that distribution, and they turn out to be perfectly distributed, you have
total cost=$11,010 and total effect=3 lives so, as a funder that cares about expected value, $3670 is the value you care about.
https://docs.google.com/spreadsheets/d/1yfK7J5V4rBUQ7-lWKrdXDNyI3NYjRouub4KwO2PVkuQ/edit?usp=sharing here is a spreadsheet with 100 cases.
Not sure how useful it is as an intuition pump, but here is an even more extreme/absurd example: if there is a 0.001% chance that the cost is 0 and a 99.999% chance that the cost is $1T, mean(effect/cost) would be ∞, even if the effect is super tiny, which is definitely not what we care about.
In general, as JoelMcGuire mentioned, it’s “a general issue where your simulation involves ratios”, taking the other ratio helps only insofar as it prevents the denominator from spanning as many orders of magnitude.
Does this make sense? Is there a better way to write it? Is it completely wrong?
In general, I think we don’t care about the mean of X/Y, (that indeed can be dominated by cases where Y is really tiny), but about the expected total X / Y.
That’s true if you spend money that way, but why would you spend money that way? Why would you spend less on the interventions that are more cost-effective? It makes more sense to spend a fixed budget. Given a 1⁄3 chance that the cost per life saved is $10, $1000, or $10,000, and you spend $29.67, then you save 1 life in expectation (= 1⁄3 * (29.67 / 10 + 29.67 / 1000 + 29.67 / 10,000)).
That’s a feature, not a bug. If something has positive value and zero cost, then you should spend zero dollars/resources to invoke the effect infinitely many times and produce infinite value (with probability 0.00001).
I agree a non-null probability of null cost implies infinite mean cost-effectiveness (mean ratio between effect and cost). However:
For a null cost, you would have null effect, thus getting an indetermination of the type 0⁄0. Denoting as CEi the possible values of the ratio between effect and cost, for CE1 = 0⁄0, the mean cost-effectiveness would be:
CE_mu = (0/0 + CE2 + CE3 + … CEN)/N = 0⁄0.
In reality, the cost can never be (exactly) null, and numerical errors resulting from the possibility of a very low cost could be handled by increasing the number of Monte Carlo samples.
The effect can be null, but this could be handled by focussing on “effect”/”cost” (what we arguably care about) instead of “cost”/”effect”.
I think one should refer to “effect”/”cost” as cost-effectiveness, since more cost-effective interventions have greater ratios between the effect and cost.
Yeah, it’s a very theoretical example.
There are things that could be modeled as very very low cost, like choosing A instead of B where both A and B cost $10, but indeed let’s focus on the other example.
Using “effect”/”cost” helps in many cases, but definitely not in all.
E.g. for policy intervention estimates of costs can vary by orders of magnitude: see https://forum.effectivealtruism.org/posts/h2N9qEbvQ6RHABcae/a-critical-review-of-open-philanthropy-s-bet-on-criminal?commentId=NajaYiQD7KhAJyBcp
I think we care about the expected marginal cost-effectiveness (i.e. mean(“marginal effect”/”marginal cost”)). Both mean(“total effect”)/mean(“total cost”) and mean(“total effect”/”total cost”) are good approximations if our budget is small, but they might not be if the budget has to be large for some reason.
Haha, I came up with that example as well. You’re thinking about this in the same way I did!
I think to say that one is the “actual objective” is not very rigorous. Although I’m saying this from a place of making that same argument. It does answer a valid question of “how much money should one donate to get an expected 1 unit of good” (which is also really easy to communicate, dollars per life saved is much easier to talk about than lives saved per dollar). I’ve been thinking about it for a while and put a comment under Edo Arad’s one.
As for the second point about simple going E(cost)E(effect). I agree that this is likely an error, and you have a good counterexample.
I still don’t think it’s an error, added a comment with my perspective, curious to hear your thoughts!
Indeed it was common feedback, but I don’t understand it fully, maybe we add a section on it to the post if we reach an agreement.