Hey! Love the post. Just putting my comments here as they go.
Tldr This seems to be a special case of the more general theory of Value of Information. There’s a lot to be said about value of information, and there are a couple of parameter choices I would question.
The EA Forum supports both Math and Footnotes now! Would be lovely to see them included for readability.
I’m sure you’re familiar with Value of Information. It has a tag on the EA Forum. It seems as if you have presumed the calculations around value of information (For instance, you have given a probability p and better-than-top-charity ratio n, both of which can be explicitly calculated with Value of Information). The rest of the calculations seem valid and interesting.
For instance, when the total budget is 1 billion dollars, then this equation entails that a research project that costs 1 million dollars (c/(m-c)=0.001) is worth funding if it has at least a 1% chance (p=0.01) of producing an intervention that is at least 10% more cost-effective (n=1.1) than the best existing intervention. This is a surprisingly low bar relative to how hard it is to get funding for EA-aligned academic research
I might be wrong, but I think this is assuming that this is the only research project that is happening. I could easily assume that EA spends more than 0.1% of it’s resources on identifying/evaluting new interventions. Although, I’m yet to know of how to do the math with multiple research projects. It’s currently a bit beyond me.
There’s a common bias to choose numbers within, say 104 and 10−3 that may bias this investigation. For instance, when I calculated the value of information on GiveDirectly, when n>1 was p<10−12. If you are unsure about whether a charity is cost-effective, often the tails of your certainty can drop fast.
Your “lower bound” is entirely of your own construction. It’s derived from your decleration at the start that p is the chance that you find a “investing cdollars into the research generates an intervention that is at leastn times as effective as the best existing intervention with probability p. If I was to call your construction the “Minimum value of information”, it’s possible to calculate the “Expected value of [Perfect|imperfect] information”, which I feel like might be a more useful number. Guesstimate can do this as well, I could provide an example if you’d like.
We have to remember that we are still uncertain about the cost-effectiveness of the new intervention, which means it would need to be expected to be more cost-effective even after considering all priors. This may increase c or decrease p. However, this is probably irrelevant to the argument.
Amusingly, we seem to come at this at two very different angles, I have a bias that I’d like EA to spend less on research (or less on research in specific directions) and you’re here to try and convince EA to spend more on research! Love you’re work, I’ll get onto your next post and we’ll chat soon.
Yes, I am familiar with the Value of Information, and I am building on it in this project. I have added the “Value of Information” tag.
I might be wrong, but I think this is assuming that this is the only >research project that is happening. I could easily assume that EA >spends more than 0.1% of it's resources on identifying/evaluting >new interventions. Although, I'm yet to know of how to do the >math with multiple research projects. It's currently a bit beyond >me.
Yes, this argument assumes that the alternative to investing some funds into R&D is that all the funds are invested into existing interventions/charities. I intended to answer the question ”When is it worthwhile for a grant maker, such as GiveWell, that currently does not fund any R&D projects to invest into the development of new interventions at all?”.
Your "lower bound" is entirely of your own construction. It's derived from your decleration at the start that p is the chance that you find a "investing c dollars into the research generates an intervention that is at least n times as effective as the best existing intervention with probability
p. If I was to call your construction the "Minimum value of information", it's possible to calculate the "Expected value of [Perfect|imperfect] information", which I feel like might be a more useful number. Guesstimate can do this as well, I could provide an example if you'd like.
I have done that. Those analyses are reported farther down in this post and in follow-up posts.
We have to remember that we are still uncertain about the cost-effectiveness of the new intervention, which means it would need to be expected to be more cost-effective even after considering all priors. This may increase or decrease . However, this is probably irrelevant to the argument.
Good point! One way to accommodate this is to add the cost of determining whether the new intervention is more cost-effective than the previous to the research cost c.
Hey! Love the post. Just putting my comments here as they go.
Tldr This seems to be a special case of the more general theory of Value of Information. There’s a lot to be said about value of information, and there are a couple of parameter choices I would question.
The EA Forum supports both Math and Footnotes now! Would be lovely to see them included for readability.
I’m sure you’re familiar with Value of Information. It has a tag on the EA Forum. It seems as if you have presumed the calculations around value of information (For instance, you have given a probability p and better-than-top-charity ratio n, both of which can be explicitly calculated with Value of Information). The rest of the calculations seem valid and interesting.
I might be wrong, but I think this is assuming that this is the only research project that is happening. I could easily assume that EA spends more than 0.1% of it’s resources on identifying/evaluting new interventions. Although, I’m yet to know of how to do the math with multiple research projects. It’s currently a bit beyond me.
There’s a common bias to choose numbers within, say 104 and 10−3 that may bias this investigation. For instance, when I calculated the value of information on GiveDirectly, when n>1 was p<10−12. If you are unsure about whether a charity is cost-effective, often the tails of your certainty can drop fast.
Your “lower bound” is entirely of your own construction. It’s derived from your decleration at the start that p is the chance that you find a “investing c dollars into the research generates an intervention that is at least n times as effective as the best existing intervention with probability p. If I was to call your construction the “Minimum value of information”, it’s possible to calculate the “Expected value of [Perfect|imperfect] information”, which I feel like might be a more useful number. Guesstimate can do this as well, I could provide an example if you’d like.
We have to remember that we are still uncertain about the cost-effectiveness of the new intervention, which means it would need to be expected to be more cost-effective even after considering all priors. This may increase c or decrease p. However, this is probably irrelevant to the argument.
Amusingly, we seem to come at this at two very different angles, I have a bias that I’d like EA to spend less on research (or less on research in specific directions) and you’re here to try and convince EA to spend more on research! Love you’re work, I’ll get onto your next post and we’ll chat soon.
Thank you, Sam!
Yes, I am familiar with the Value of Information, and I am building on it in this project. I have added the “Value of Information” tag.
Yes, this argument assumes that the alternative to investing some funds into R&D is that all the funds are invested into existing interventions/charities. I intended to answer the question ”When is it worthwhile for a grant maker, such as GiveWell, that currently does not fund any R&D projects to invest into the development of new interventions at all?”.
I have done that. Those analyses are reported farther down in this post and in follow-up posts.
Good point! One way to accommodate this is to add the cost of determining whether the new intervention is more cost-effective than the previous to the research cost c.