Interesting Analysis!
Since you already have confidence intervals for a lot of your models factors, using the guesstimate web tool to get a more detailed idea of the uncertainty in the final estimate might be helpful, since some bayesian discounting based on estimate’s uncertainty might be a sensible thing to do. (https://www.lesswrong.com/posts/5gQLrJr2yhPzMCcni/the-optimizer-s-curse-and-how-to-beat-it)
Here how I would reason about moral weights in this case:
In this case the definition of a “life saved” is pretty different than what normally means. Normally a life saved means 30 to 80 DALYs averted, depending if the intervention is on adults or children. In this case we are talking about potentially thousands of DALYs averted, so a life saved should count more. On the other hand there’s also to take into consideration that when saving, for example, children who would have died of malaria, you are also giving them a chance of reaching LEV. It’s not a full chance as in the present evaluation, but something probably ranging from 30% to 70%.
Additional consideration: some people may want to consider children more important to save than adults. Introducing age weighting and time discounting could seem reasonable in this case, since even if you save 5000 DALYs you are only saving one person, so you might want to discount DALYs saved later in life. On the other hand there are reasons to disagree with this approach: Saving an old person and guaranteeing him/her to reach LEV means also “saving a library”. A vast amount of knowledge and experience, especially future experience would have been otherwise completely destroyed. In fact I am not so sure I would apply time discounting myself for this reason.
I would also like to introduce probability distributions in the whole analysis and turn some arguments made in the explanations of some variables in variables in their own right, and I would like to add some more informations (for example the safety profile and history of metformin and the value of information of the trial) based on feedback I’m receiving. This would mean rewriting many sections though, and this will require time.
For now I put an “Edit” at the beginning in order to warn readers not to take the numbers reached too seriously, but I invited them to delve in some more broadly applicable ideas I presented in the analysis that could be useful for evaluating many interventions in the cause area of aging.
I think, it might be best to just report confidence intervals for your final estimates (guesstimate should give you those). Then everyone can combine your estimates with their own priors on general intervention’s effectiveness and thereby potentially correct for the high levels of uncertainty (at least in a crude way by estimating the variance from the confidence intervals).
The variance of X can be defined as E[X^2]-E[X]^2, which should not be hard to implement in Guesstimate. However, i am not sure, whether or not having the variance yields to more accurate updating, than having a confidence interval. Optimally you’d have the full distribution, but i am not sure, whether anyone will actually do the maths to update from there. (But they could get it roughly from your guesstimate model).
I might comment more on some details and the moral assumptions, if i find the time for it soon.
Thank you, I applied your suggestion by modifying the text. I just noticed that Guesstimate gives you the standard deviation. I guess I had to familiarise with the tool.
I’m still learning and comments really help me to be more accurate and they steepen my learning curve. I set up a Guesstimate model (https://www.getguesstimate.com/models/10848). I didn’t know about this tool, it is really helpful!
Tomorrow I will improve the guesstimate and get back to you with another comment regarding the bayesian discounting you proposed and the moral weights. I also might make other changes to the evaluation together with the ones you suggested, especially considering that Guesstimate lets me toy with probability distributions.
Interesting Analysis! Since you already have confidence intervals for a lot of your models factors, using the guesstimate web tool to get a more detailed idea of the uncertainty in the final estimate might be helpful, since some bayesian discounting based on estimate’s uncertainty might be a sensible thing to do. (https://www.lesswrong.com/posts/5gQLrJr2yhPzMCcni/the-optimizer-s-curse-and-how-to-beat-it)
It might also make sense to make your ethical assumptions more explicit in the beginning (https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/comparing-moral-weights), especially since the case against aging seems to be less intuitive than most of givewells interventions.
Here how I would reason about moral weights in this case:
In this case the definition of a “life saved” is pretty different than what normally means. Normally a life saved means 30 to 80 DALYs averted, depending if the intervention is on adults or children. In this case we are talking about potentially thousands of DALYs averted, so a life saved should count more. On the other hand there’s also to take into consideration that when saving, for example, children who would have died of malaria, you are also giving them a chance of reaching LEV. It’s not a full chance as in the present evaluation, but something probably ranging from 30% to 70%.
Additional consideration: some people may want to consider children more important to save than adults. Introducing age weighting and time discounting could seem reasonable in this case, since even if you save 5000 DALYs you are only saving one person, so you might want to discount DALYs saved later in life. On the other hand there are reasons to disagree with this approach: Saving an old person and guaranteeing him/her to reach LEV means also “saving a library”. A vast amount of knowledge and experience, especially future experience would have been otherwise completely destroyed. In fact I am not so sure I would apply time discounting myself for this reason.
Regarding bayesian discounting:
I just read how GiveWell would go about this (https://blog.givewell.org/2011/08/18/why-we-cant-take-expected-value-estimates-literally-even-when-theyre-unbiased/). To account for it I would need a prior distribution (or more than one?). I also have difficulty making the calculation, since Guesstimate doesn’t let me calculate the variance of the random variables. I will try with other means… maybe with smaller data sets and proceeding by hand or using online calculators.
I would also like to introduce probability distributions in the whole analysis and turn some arguments made in the explanations of some variables in variables in their own right, and I would like to add some more informations (for example the safety profile and history of metformin and the value of information of the trial) based on feedback I’m receiving. This would mean rewriting many sections though, and this will require time.
For now I put an “Edit” at the beginning in order to warn readers not to take the numbers reached too seriously, but I invited them to delve in some more broadly applicable ideas I presented in the analysis that could be useful for evaluating many interventions in the cause area of aging.
I think, it might be best to just report confidence intervals for your final estimates (guesstimate should give you those). Then everyone can combine your estimates with their own priors on general intervention’s effectiveness and thereby potentially correct for the high levels of uncertainty (at least in a crude way by estimating the variance from the confidence intervals).
The variance of X can be defined as E[X^2]-E[X]^2, which should not be hard to implement in Guesstimate. However, i am not sure, whether or not having the variance yields to more accurate updating, than having a confidence interval. Optimally you’d have the full distribution, but i am not sure, whether anyone will actually do the maths to update from there. (But they could get it roughly from your guesstimate model).
I might comment more on some details and the moral assumptions, if i find the time for it soon.
Thank you, I applied your suggestion by modifying the text. I just noticed that Guesstimate gives you the standard deviation. I guess I had to familiarise with the tool.
Thank you for the feedback!
I’m still learning and comments really help me to be more accurate and they steepen my learning curve. I set up a Guesstimate model (https://www.getguesstimate.com/models/10848). I didn’t know about this tool, it is really helpful!
Tomorrow I will improve the guesstimate and get back to you with another comment regarding the bayesian discounting you proposed and the moral weights. I also might make other changes to the evaluation together with the ones you suggested, especially considering that Guesstimate lets me toy with probability distributions.