Nice find, but I think there might be a subtle mistake there in the interpretation.
I think that the problem is with taking the expectation over 1effect, instead of multiplying the expected values mean(1effect)⋅mean(cost). It’s reasonable to expect that the cost and the effect (of some specifically defined intervention) are uncorrelated, so the latter is actually the same as mean(costeffect).
However, taking the mean value of 1effect is definitely not the same as the reciprocal of the expected effect. In fact, at the start of the post you have computed a cost effectiveness of 100$ per life, but the way you have done it is by looking at the expected utility for the intervention (which is mean(effect)=100 lives ) and dividing the cost by this amount.
In GiveWell’s CEAs (at least, I’ve verified for Deworm The World), they calculate the average value per constant constant cost. So this shouldn’t affect their analysis.
I think that the problem is with taking the expectation over 1effect, instead of multiplying the expected values mean(1effect)⋅mean(cost).
I agree! It’s the division that’s not linear, not the multiplication. How do you think we could make it clearer in the post?
Edit: on more thought, I’m not sure I understand your point.mean(1/effect)⋅mean(cost) gives the “wrong” result (e.g. if effect can be 0 with non 0 probability, mean(1/effect) will be +∞)
People are asking the question “How much money do you have to donate to get an expected value of 1 unit of good”.
I think the question is:
How can I do as much good as possible with C units of cost?
This corresponds to the problem of maximising E(U(C)), where U(c) is the utility achieved (via a certain intervention) for the cost c (which must not exceed C). If the budget C is small enough (thinking at the margin):
U(C) = U’(0)*C, where U’(c) is the derivate of U with respect to cost.
Assuming U’(0) and C are independent, mean(“effect”/”cost”) equals mean(“effect”)/mean(“cost”):
mean(“effect”)/mean(“cost”) = E(U(C))/E(C) = E(U’(0))*E(C)/E(C) (assuming independence between U’(0) and C) = E(U’(0)).
So, it seems that, regardless of the metric we choose, we should maximise E(U’(0)), i.e. the expected marginal cost-effectiveness. However, U’(0) and C will not be independent for large C, so I think it is better to maximise mean(“effect”/”cost”).
I think Michael Dickens explains this better than me in a more recent comment. The point is that we usually care about effect/cost rather than the other way around (although I’d love to understand more clearly why and when exactly that’s true). In your example, you have implicitly computed this and then compare it to Guesstimate’s model computing cost/effect.
How do you think we could make it clearer in the post?
I think your example should focus on the mean of the ratio between the effect and cost, not on the mean of the ratio between the cost and effect. The latter is a bad metric because:
A very small “cost”/”effect” could correspond to interventions that are either quite bad or good (since “cost”/∞ = 0). This means small numerical errors could lead to large differences in mean(“cost”/”effect”), which is bad.
When changing from negative values of “cost”/”effect” to positive ones, the goodness of the intervention increases (changing from harmful to beneficial). However, for negative and positive values, a higher “cost”/”effect” corresponds to a worse intervention.
The metric “effect”/”cost” has good properties:
A higher value always implies a better intervention (at least in theory).
Nice find, but I think there might be a subtle mistake there in the interpretation.
I think that the problem is with taking the expectation over 1effect, instead of multiplying the expected values mean(1effect)⋅mean(cost). It’s reasonable to expect that the cost and the effect (of some specifically defined intervention) are uncorrelated, so the latter is actually the same as mean(costeffect).
However, taking the mean value of 1effect is definitely not the same as the reciprocal of the expected effect. In fact, at the start of the post you have computed a cost effectiveness of 100$ per life, but the way you have done it is by looking at the expected utility for the intervention (which is mean(effect)=100 lives ) and dividing the cost by this amount.
In GiveWell’s CEAs (at least, I’ve verified for Deworm The World), they calculate the average value per constant constant cost. So this shouldn’t affect their analysis.
I agree! It’s the division that’s not linear, not the multiplication.How do you think we could make it clearer in the post?
Edit: on more thought, I’m not sure I understand your point.mean(1/effect)⋅mean(cost) gives the “wrong” result (e.g. if effect can be 0 with non 0 probability, mean(1/effect) will be +∞)
Thank you so much for the post! I might communicate it as:
People are asking the question “How much money do you have to donate to get an expected value of 1 unit of good” Which could be formulated as:
E(good(x))=1
where x is the amount you donate and good(x) is the amount of utility you get out of it.
In most cases, this is linear, so: good(x)=goodcost∗x. And E(goodcostx)=1.
Solving for x in this case gets x=E(goodcost)−1, but the mistake is to solve it and get x=E(costgood).
Please correct me if this is a bad way to formulate the problem! Can’t wait to see your future work as well
nice explanation :)
I think the question is:
How can I do as much good as possible with C units of cost?
This corresponds to the problem of maximising E(U(C)), where U(c) is the utility achieved (via a certain intervention) for the cost c (which must not exceed C). If the budget C is small enough (thinking at the margin):
U(C) = U’(0)*C, where U’(c) is the derivate of U with respect to cost.
Assuming U’(0) and C are independent, mean(“effect”/”cost”) equals mean(“effect”)/mean(“cost”):
mean(“effect”/”cost”) = E(U(C)/C) = E(U’(0)*C/C) = E(U’(0)).
mean(“effect”)/mean(“cost”) = E(U(C))/E(C) = E(U’(0))*E(C)/E(C) (assuming independence between U’(0) and C) = E(U’(0)).
So, it seems that, regardless of the metric we choose, we should maximise E(U’(0)), i.e. the expected marginal cost-effectiveness. However, U’(0) and C will not be independent for large C, so I think it is better to maximise mean(“effect”/”cost”).
Is that a typo on your final bullet point? Should say mean(“effect”)/mean(“cost”).
Hi Stan,
Yes, it was a typo, which I have now corrected. Thanks for catching it! I have also added one extra sentence at the end:
Thanks for commenting here, and thanks again for your initial feedback!
I don’t really have anything planned in this area, what would you be excited to see?
I think Michael Dickens explains this better than me in a more recent comment. The point is that we usually care about effect/cost rather than the other way around (although I’d love to understand more clearly why and when exactly that’s true). In your example, you have implicitly computed this and then compare it to Guesstimate’s model computing cost/effect.
Replied to Michael Dickens, curious about your thoughts! We should definitely add a section on this
I think your example should focus on the mean of the ratio between the effect and cost, not on the mean of the ratio between the cost and effect. The latter is a bad metric because:
A very small “cost”/”effect” could correspond to interventions that are either quite bad or good (since “cost”/∞ = 0). This means small numerical errors could lead to large differences in mean(“cost”/”effect”), which is bad.
When changing from negative values of “cost”/”effect” to positive ones, the goodness of the intervention increases (changing from harmful to beneficial). However, for negative and positive values, a higher “cost”/”effect” corresponds to a worse intervention.
The metric “effect”/”cost” has good properties:
A higher value always implies a better intervention (at least in theory).
A null value correspond to neutrality.