I do not think anyone with a math background would find this obvious. Judging by the comments on this post and the feedback you said you received, I think you should update your beliefs on whether this claim is obvious at all.
I was completely wrong, indeed!
Will think about the comments for a few hours and write an appendix tonight.
Do you agree that the main practical takeaway for non-experts reading this post should be “Be very careful using mean(cost/effect), especially if the effect can be small”?
I think the focus on examples detracts from the post. Examples can be misleading.
I disagree, the first example is exaggerated, but it’s a very common issue, I think like a third of guesstimate models have some version of it. (see the three recent examples in the post)
Will respond to the other parts of the comments in the appendix, since many other commenters raised similar points.
I think the main practical takeaway should be to use mean(effect/cost) unless you have a really good reason not to. I agree mean(cost/effect) is a bad metric because it would be unreasonable for our effect distribution to not include zero or negative values—which is the only way mean(cost/effect) is even defined.
I was completely wrong, indeed!
Will think about the comments for a few hours and write an appendix tonight.
Do you agree that the main practical takeaway for non-experts reading this post should be “Be very careful using mean(cost/effect), especially if the effect can be small”?
I disagree, the first example is exaggerated, but it’s a very common issue, I think like a third of guesstimate models have some version of it. (see the three recent examples in the post)
Will respond to the other parts of the comments in the appendix, since many other commenters raised similar points.
I found this comment https://forum.effectivealtruism.org/posts/SesLZfeYsqjRxM6gq/probability-distributions-of-cost-effectiveness-can-be?commentId=nA3mJoj2fToXtX7pY from Jérémy particularly clear
I think the main practical takeaway should be to use mean(effect/cost) unless you have a really good reason not to. I agree mean(cost/effect) is a bad metric because it would be unreasonable for our effect distribution to not include zero or negative values—which is the only way mean(cost/effect) is even defined.