Not sure how useful it is as an intuition pump, but here is an even more extreme/absurd example: if there is a 0.001% chance that the cost is 0 and a 99.999% chance that the cost is $1T, mean(effect/cost) would be ∞, even if the effect is super tiny, which is definitely not what we care about.
I agree a non-null probability of null cost implies infinite mean cost-effectiveness (mean ratio between effect and cost). However:
For a null cost, you would have null effect, thus getting an indetermination of the type 0⁄0. Denoting as CEi the possible values of the ratio between effect and cost, for CE1 = 0⁄0, the mean cost-effectiveness would be:
CE_mu = (0/0 + CE2 + CE3 + … CEN)/N = 0⁄0.
In reality, the cost can never be (exactly) null, and numerical errors resulting from the possibility of a very low cost could be handled by increasing the number of Monte Carlo samples.
The effect can be null, but this could be handled by focussing on “effect”/”cost” (what we arguably care about) instead of “cost”/”effect”.
I think one should refer to “effect”/”cost” as cost-effectiveness, since more cost-effective interventions have greater ratios between the effect and cost.
Yeah, it’s a very theoretical example. There are things that could be modeled as very very low cost, like choosing A instead of B where both A and B cost $10, but indeed let’s focus on the other example.
I think we care about the expected marginal cost-effectiveness (i.e. mean(“marginal effect”/”marginal cost”)). Both mean(“total effect”)/mean(“total cost”) and mean(“total effect”/”total cost”) are good approximations if our budget is small, but they might not be if the budget has to be large for some reason.
I agree a non-null probability of null cost implies infinite mean cost-effectiveness (mean ratio between effect and cost). However:
For a null cost, you would have null effect, thus getting an indetermination of the type 0⁄0. Denoting as CEi the possible values of the ratio between effect and cost, for CE1 = 0⁄0, the mean cost-effectiveness would be:
CE_mu = (0/0 + CE2 + CE3 + … CEN)/N = 0⁄0.
In reality, the cost can never be (exactly) null, and numerical errors resulting from the possibility of a very low cost could be handled by increasing the number of Monte Carlo samples.
The effect can be null, but this could be handled by focussing on “effect”/”cost” (what we arguably care about) instead of “cost”/”effect”.
I think one should refer to “effect”/”cost” as cost-effectiveness, since more cost-effective interventions have greater ratios between the effect and cost.
Yeah, it’s a very theoretical example.
There are things that could be modeled as very very low cost, like choosing A instead of B where both A and B cost $10, but indeed let’s focus on the other example.
Using “effect”/”cost” helps in many cases, but definitely not in all.
E.g. for policy intervention estimates of costs can vary by orders of magnitude: see https://forum.effectivealtruism.org/posts/h2N9qEbvQ6RHABcae/a-critical-review-of-open-philanthropy-s-bet-on-criminal?commentId=NajaYiQD7KhAJyBcp
I think we care about the expected marginal cost-effectiveness (i.e. mean(“marginal effect”/”marginal cost”)). Both mean(“total effect”)/mean(“total cost”) and mean(“total effect”/”total cost”) are good approximations if our budget is small, but they might not be if the budget has to be large for some reason.