As far as I can tell, ex-ante cost-effectiveness is not the most important figure for someone considering whether to fund a future project. I think the expected benefit per unit cost is more important.
For reference, you give this definition:
We define the ex-ante cost-effectiveness of an R&D project as the expected value of its ex-post cost-effectiveness, given only the information that is available before the project is conducted.
I think I understand what this means, but I am going to attempt to show why it’s not that useful using a simple example.
Scenario 1: Suppose we know that a project will have benefit B=1 and that the projected cost C has distribution
P(C=1)=12
P(C=10)=12
Then the ex-post cost-effectiveness CE of the project will have distribution
P(CE=1)=12
P(CE=110)=12
and thus has expected value
E(CE)=12⋅1+12⋅110=0.505
Why is this not useful? It does not reflect the expected return-on-investment, and is not sensitive to high-cost scenarios. Consider Scenario 2, a similar project with known benefit B=1 and cost with distribution
P(C=1)=12
P(C=10000)=12
Scenario 2 is clearly much less cost-effective than Scenario 1. But the ex-ante cost-effectiveness is 0.500005, very close to 0.505.
What a decision-maker really wants to know is the amount of benefit they can expect from each unit of investment. This can be given byE(B)E(C).
Scenario 1: E(B)E(C)=15.5≈0.18
Scenario 2:E(B)E(C)=15000.5≈0.00020
We can see that this does appropriately reflect the difference in cost-effectiveness between the two scenarios. What I’m not so sure about is how we might give the expected benefit per unit cost as a distribution, rather than just a point-estimate.
It seems likely that I’m missing something.
What is your rationale for focusing on expected value of ex-post cost-effectiveness ?
Could you use an adapted method to make an ex-ante prediction of the benefit per unit cost of Baumsteiger’s R&D project?
I think the appropriateness of E[CE] as a prioritization criterion depends on the nature of the decision problem.
I think the expected value of the cost-effectiveness ratio is the appropriate prioritization criterion for the following scenario: i) a decision-maker is considering which organization should receive a given fixed amount of money (m), and ii) each organization (i) turns every dollar it receives into some uncertain amount of value (CE_i). In that case, the expected utility of giving the money to organization i is E[U_i]= m*E[CE_i]. Therefore, the way to maximize expected utility is to give the money to the organization with the highest expected cost-effectiveness. In this scenario, the consequences of contributing $1 to a project with an expected cost-effectiveness of 1 WELLBY/$ are almost identical in both scenarios. Most of the expected utility comes from the possibility that the project might be highly cost-effective. If the project is not highly cost-effective, then the $1 contribution accomplishes very little, regardless of whether the project costs $10,000, $100,000, or $1,000,000.
In my view, your example illustrates that the expected cost-effectiveness ratio is an inappropriate prioritization criterion if the funder has to decide whether to pay 100% of the project’s costs without knowing how much that will be. In that scenario, I think the appropriate prioritization criterion would be E[B]-E[CE_alt]*E[C], where E[CE_alt] is the expected cost-effectiveness of the most promising project that the funder could fund instead.
I think the second decision problem describes the situation of a researcher or funder who is committed to seeing their project through until the end. By contrast, the first decision problem corresponds to a researcher/funder intending to allocate a fixed amount of time/money to one project or another (e.g., 3 years of personal time or 1 million dollars) and then move on to another project after that.
As far as I can tell, ex-ante cost-effectiveness is not the most important figure for someone considering whether to fund a future project. I think the expected benefit per unit cost is more important.
For reference, you give this definition:
I think I understand what this means, but I am going to attempt to show why it’s not that useful using a simple example.
Scenario 1: Suppose we know that a project will have benefit B=1 and that the projected cost C has distribution
P(C=1)=12
P(C=10)=12
Then the ex-post cost-effectiveness CE of the project will have distribution
P(CE=1)=12
P(CE=110)=12
and thus has expected value
E(CE)=12⋅1+12⋅110=0.505
Why is this not useful? It does not reflect the expected return-on-investment, and is not sensitive to high-cost scenarios. Consider Scenario 2, a similar project with known benefit B=1 and cost with distribution
P(C=1)=12
P(C=10000)=12
Scenario 2 is clearly much less cost-effective than Scenario 1. But the ex-ante cost-effectiveness is 0.500005, very close to 0.505.
What a decision-maker really wants to know is the amount of benefit they can expect from each unit of investment. This can be given byE(B)E(C).
Scenario 1: E(B)E(C)=15.5≈0.18
Scenario 2: E(B)E(C)=15000.5≈0.00020
We can see that this does appropriately reflect the difference in cost-effectiveness between the two scenarios. What I’m not so sure about is how we might give the expected benefit per unit cost as a distribution, rather than just a point-estimate.
It seems likely that I’m missing something.
What is your rationale for focusing on expected value of ex-post cost-effectiveness ?
Could you use an adapted method to make an ex-ante prediction of the benefit per unit cost of Baumsteiger’s R&D project?
Thank you for your feedback, Stan!
I think the appropriateness of E[CE] as a prioritization criterion depends on the nature of the decision problem.
I think the expected value of the cost-effectiveness ratio is the appropriate prioritization criterion for the following scenario: i) a decision-maker is considering which organization should receive a given fixed amount of money (m), and ii) each organization (i) turns every dollar it receives into some uncertain amount of value (CE_i). In that case, the expected utility of giving the money to organization i is E[U_i]= m*E[CE_i]. Therefore, the way to maximize expected utility is to give the money to the organization with the highest expected cost-effectiveness. In this scenario, the consequences of contributing $1 to a project with an expected cost-effectiveness of 1 WELLBY/$ are almost identical in both scenarios. Most of the expected utility comes from the possibility that the project might be highly cost-effective. If the project is not highly cost-effective, then the $1 contribution accomplishes very little, regardless of whether the project costs $10,000, $100,000, or $1,000,000.
In my view, your example illustrates that the expected cost-effectiveness ratio is an inappropriate prioritization criterion if the funder has to decide whether to pay 100% of the project’s costs without knowing how much that will be. In that scenario, I think the appropriate prioritization criterion would be E[B]-E[CE_alt]*E[C], where E[CE_alt] is the expected cost-effectiveness of the most promising project that the funder could fund instead.
I think the second decision problem describes the situation of a researcher or funder who is committed to seeing their project through until the end. By contrast, the first decision problem corresponds to a researcher/funder intending to allocate a fixed amount of time/money to one project or another (e.g., 3 years of personal time or 1 million dollars) and then move on to another project after that.
Regardless thereof, I can rerun the analyses for E[B]/E[C] as a robustness check and let you know what I find.