Note that Charity Entrepreneurship (CE) has now rebranded to AIM to reflect our widening scope of programs
[Edited for tone]
Thank you so much for engaging with our work in this level of detail. It is great to get critical feedback and analysis like this. I have made a note of this point on my long list of things to improve about how we do our CEAs, although for the reasons I explain below it is fairly low down on that list.
Ultimately what we are using now is a very crude approximation. That said it is one I am extremely loath to start fiddling without putting the effort in to do this well.
You are right that the numbers used for comparing deaths and disability are a fairly crude approximation. A reasonable change in moral weights can lead to a large change in the comparison between YLDs and YLLs. Consider that when GiveWell last reviewed their moral weights (between 2019 and 2020) they increased the value of an under-5 life saved compared to YLDs by +68% (from 100⁄3.3 to 116.9/2.3). Another very valid criticism is that (as you point out) the current numbers we are using are calculated with a 3% discount rate, yet we are now using a 1.5% discount rate for health effects, so perhaps to ensure consistency we should increase the numbers by +42%ish. Or taking the HLI work on the value of death seriously could suggest a huge decrease of −50% or more. The change you suggest would be nice but I think getting this right really needs a lot of work.
Right now I am uncertain how best to update these numbers. A minus −10% change is reasonable but so are many other changes. I would very much like AIM to have our own calculated moral weightings that account for various factors, including life expectancy, a range of ethical views, quality of life, beneficiary preferences, etc. However getting this correct is a complicated and lengthy process. This is on the to-do list but has not happened yet unfortunately.
So what do we do in the meantime:
We use numbers that seem justifiable, close to what I understand as standard and reasonably acceptable within Global Health and Development (from here table 5.1, I believe have been used by GW DCP2 and GBD etc). These numbers are also close to (but somewhat below) a very crude staff survey we did on the moral weight of saving a life. That said I admit I would be interested in updates of what organisations are currently using.
We are aware of the limits of our CEAs and use them cautiously in our decision making process, and would encourage others to be cautious about over relying on them. We have written about this here: https://www.charityentrepreneurship.com/cea. We are well aware in making decisions that some of the numbers used to compare different kinds of interventions rest on a lot of shaky assumptions.
We tend to try to pick a range of interventions across reasonable moral weights and moral views. We will try to pick some interventions that save lives, some that improve health, some that improve lives in other ways. That said I expect that maybe we have over (or under) valued lives saved.
Ultimately I believe that this is sufficient for the level of decision making we need to make.
I hope that someday soon we have the time to work this out in detail.
ACTIONS. • [Edited: I wont change anything straight away as the model as a bunch of modelling in this research round has already been done, and for now I would rather use numbers I can back up with a source than numbers that are tweaked for one reason but not another reason.] • I have added a note about the point you raise to our internal list of ways to improve our CEAs. [Edit: I really would like to make some changes here going forward. I expect that if I put a few hours into this the number is more likely to go up than down given the discount rate difference (and the staff survey).] • I might also do some extra sensitivity analysis on our CEAs to highlight the uncertainty around this factor and ensure it is flagged to decision makers.
I don’t buy your line of argument that just because what you do “crude approximation” or “scratching the surface” (which I agree with and many commentators have pointed out) is a reason not to include new variables to your model. That seems like conflating two issues. Obviously assumptions and errors are massive, but that’s not a reason not to continuously be improving models.
Like @Larks says its not about a constant 10% reduction, but taking something extra into account which makes sense could meaningfully be added to the model.
Just because it might not be the most important thing right now or “low down your list” isn’t a reason from my perspective not to include it.
I agree that you are probably underestimating rather than overstimating in general, and I actually prefer AIM/CEs approach to GiveWell’s partly for the reason that this post exists—it is a bit easier to scrutinise and criticise, and therefore potentially easier to improve.
As a side note your response (probably unintentionally) feels just a little unnecesarily defensive but I might be reading too much into it.
Hi Nick, Thank you very much for the comment. These are all good points.
I fully agree with you and Larks that where a specific intervention will have reduced impact due to long run health effects this should be included in our models and I will check this is happening.
I apologise for the defensiveness and made a few minor edits to the post trying to keep content the same.
That’s not a reason not to continuously be improving models.
To be clear, we are always always improving our CEA models. This is an ongoing iterative process, and my hope is they get better year upon year. However, I guess I don’t have confidence right now that a −10% change to this number is actually improving the model or affecting our decision making.
If we dive into these numbers just a bit, I immediately notice that the discount rate in the GBD data is higher than ours and that should suggest that, if we are adjusting these numbers, that probably we want a significant +increase not decrease. But that then raises the question of what discount rate we are using and why, which has a huge effect on some of the models – and this is something there are currently internal debates in the team about, and we are looking at changing. But this then raises a question about how to represent the uncertainty about these numbers in our models and ensure the decision makers and readers are more aware of the inherent estimations that can have big effect on CEA outputs – and improving this is probably towards the top of my list.
I just want to flag that I’ve raised the issue of the inconsistencies in the use of discount rate (if by “the discount rate in the GBD data” you mean the 3% or 4% discount rate in the standard inputs table) in an email sent a few days ago to one of the CE employees. Unfortunately, we failed to have a productive discussion, as the conversation died quickly when CE stopped responding. Here is one of the emails I sent:
Hi [name],
I might be wrong but you are using 1.4% rate in the CEA but the value of life saved at various ages is copied from GiveWell standard inputs that uses 4% discount rate to calculate the value. Isn’t this an inconsistency?
Makes sense that you aren’t confident about this particular change yet, are discussing improvement with the team and that you are concerned about the overall situation that on balance you think your numbers are too low and you probably want a increase rather than a decrease.
To reiterate I love that you have a model which can actually be scrutinised and meaningfully iterated—I still can’t really get my head around GiveWell’s but maybe I haven’t tried hard enough.
No matter what system is used error’s are going to be massive, so why not make it more understandable and editable?
I think the issue isn’t so much a constant −10%, but that some specific life-saving interventions might saves lives yet leave people with unusually low quality of life, and for those interventions the error term might be much larger than 10%.
Thank you Larks. This is a very good point and I fully agree.
In any cases where this happens it should be incorporated into our current model. That said I will check this for our current research and make sure that in any such cases (such as say pulmonary rehabilitation for COPD where patients are expected to have a lower quality of life if they survive) this is accounted for.
I might have been too directive when writing this post. I lack the organizational context and knowledge of how CEAs are used to say definitively that this should be changed. I ultimately agree that this is a small change that might not affect the decisions made, and it’s up to you to decide whether to account for it. However, some of the points you raised against updating this are incorrect.
I might have focused too much on the 10% reduction, while the real issue, as Elliot mentioned, is that you ignore two variables in the formula for DALYs averted:
Missing out on three 10% reductions in error X results in a difference of 0.1^3 = 27.1% which could be significant. I generally view organizations as growing through small iterative changes and optimization rather than big leaps.
My critique is only valid if you are trying to measure DALYs averted. If you choose to do something similar to GiveWell, which is more arbitrary, then it might not make sense to adjust for this anymore.
The three changes to the value of life saved come from different frameworks:
GiveWell values don’t represent DALYs averted but are mixed with other factors such as survey results.
HLI’s work is based on the assumption that death isn’t the worst possible state and that there is a baseline quality of life that must be met for a life to be worth living.
The change I’m suggesting is compatible with your current method of estimating the value of life saved. It doesn’t introduce any new assumptions; it simply makes some assumptions explicit. Unless you state something like, “We used those values initially but then detached them from their original formulas and now we will update them in another way,” my suggestion should fit within your existing framework.
EDIT:
I can’t say much about the GiveWell 1.5% rate, but I’ve heard it comes from the Rethink Priorities review, but it suggests 4.3% discount rate: can you direct me somewhere where I can read more about it?
Hi, Thank you. All good points. Fully agree with ongoing iterative improvement to our CEAs and hopefully you will see such improvements happening over the various research rounds (see also my reply to Nick). I also agree with picking up on specific cases where this might be a bigger issue (see my reply to Larks). I don’t think it is fair to say that we treat those two numbers as zero but it is fair to say we are currently using a fairly crude approximation to get at what those numbers are getting it in our lives saved calculations.
Hi there. I am Research Director at CE/AIM
Note that Charity Entrepreneurship (CE) has now rebranded to AIM to reflect our widening scope of programs
[Edited for tone]
Thank you so much for engaging with our work in this level of detail. It is great to get critical feedback and analysis like this. I have made a note of this point on my long list of things to improve about how we do our CEAs, although for the reasons I explain below it is fairly low down on that list.
Ultimately what we are using now is a very crude approximation. That said it is one I am extremely loath to start fiddling without putting the effort in to do this well.
You are right that the numbers used for comparing deaths and disability are a fairly crude approximation. A reasonable change in moral weights can lead to a large change in the comparison between YLDs and YLLs. Consider that when GiveWell last reviewed their moral weights (between 2019 and 2020) they increased the value of an under-5 life saved compared to YLDs by +68% (from 100⁄3.3 to 116.9/2.3). Another very valid criticism is that (as you point out) the current numbers we are using are calculated with a 3% discount rate, yet we are now using a 1.5% discount rate for health effects, so perhaps to ensure consistency we should increase the numbers by +42%ish. Or taking the HLI work on the value of death seriously could suggest a huge decrease of −50% or more. The change you suggest would be nice but I think getting this right really needs a lot of work.
Right now I am uncertain how best to update these numbers. A minus −10% change is reasonable but so are many other changes. I would very much like AIM to have our own calculated moral weightings that account for various factors, including life expectancy, a range of ethical views, quality of life, beneficiary preferences, etc. However getting this correct is a complicated and lengthy process. This is on the to-do list but has not happened yet unfortunately.
So what do we do in the meantime:
We use numbers that seem justifiable, close to what I understand as standard and reasonably acceptable within Global Health and Development (from here table 5.1, I believe have been used by GW DCP2 and GBD etc). These numbers are also close to (but somewhat below) a very crude staff survey we did on the moral weight of saving a life. That said I admit I would be interested in updates of what organisations are currently using.
We are aware of the limits of our CEAs and use them cautiously in our decision making process, and would encourage others to be cautious about over relying on them. We have written about this here: https://www.charityentrepreneurship.com/cea. We are well aware in making decisions that some of the numbers used to compare different kinds of interventions rest on a lot of shaky assumptions.
We tend to try to pick a range of interventions across reasonable moral weights and moral views. We will try to pick some interventions that save lives, some that improve health, some that improve lives in other ways. That said I expect that maybe we have over (or under) valued lives saved.
Ultimately I believe that this is sufficient for the level of decision making we need to make.
I hope that someday soon we have the time to work this out in detail.
ACTIONS.
• [Edited: I wont change anything straight away as the model as a bunch of modelling in this research round has already been done, and for now I would rather use numbers I can back up with a source than numbers that are tweaked for one reason but not another reason.]
• I have added a note about the point you raise to our internal list of ways to improve our CEAs. [Edit: I really would like to make some changes here going forward. I expect that if I put a few hours into this the number is more likely to go up than down given the discount rate difference (and the staff survey).]
• I might also do some extra sensitivity analysis on our CEAs to highlight the uncertainty around this factor and ensure it is flagged to decision makers.
So thank you for raising this.
I don’t buy your line of argument that just because what you do “crude approximation” or “scratching the surface” (which I agree with and many commentators have pointed out) is a reason not to include new variables to your model. That seems like conflating two issues. Obviously assumptions and errors are massive, but that’s not a reason not to continuously be improving models.
Like @Larks says its not about a constant 10% reduction, but taking something extra into account which makes sense could meaningfully be added to the model.
Just because it might not be the most important thing right now or “low down your list” isn’t a reason from my perspective not to include it.
I agree that you are probably underestimating rather than overstimating in general, and I actually prefer AIM/CEs approach to GiveWell’s partly for the reason that this post exists—it is a bit easier to scrutinise and criticise, and therefore potentially easier to improve.
As a side note your response (probably unintentionally) feels just a little unnecesarily defensive but I might be reading too much into it.
Hi Nick, Thank you very much for the comment. These are all good points.
I fully agree with you and Larks that where a specific intervention will have reduced impact due to long run health effects this should be included in our models and I will check this is happening.
I apologise for the defensiveness and made a few minor edits to the post trying to keep content the same.
To be clear, we are always always improving our CEA models. This is an ongoing iterative process, and my hope is they get better year upon year. However, I guess I don’t have confidence right now that a −10% change to this number is actually improving the model or affecting our decision making.
If we dive into these numbers just a bit, I immediately notice that the discount rate in the GBD data is higher than ours and that should suggest that, if we are adjusting these numbers, that probably we want a significant +increase not decrease. But that then raises the question of what discount rate we are using and why, which has a huge effect on some of the models – and this is something there are currently internal debates in the team about, and we are looking at changing. But this then raises a question about how to represent the uncertainty about these numbers in our models and ensure the decision makers and readers are more aware of the inherent estimations that can have big effect on CEA outputs – and improving this is probably towards the top of my list.
I just want to flag that I’ve raised the issue of the inconsistencies in the use of discount rate (if by “the discount rate in the GBD data” you mean the 3% or 4% discount rate in the standard inputs table) in an email sent a few days ago to one of the CE employees. Unfortunately, we failed to have a productive discussion, as the conversation died quickly when CE stopped responding. Here is one of the emails I sent:
Nice one I love that response.
Makes sense that you aren’t confident about this particular change yet, are discussing improvement with the team and that you are concerned about the overall situation that on balance you think your numbers are too low and you probably want a increase rather than a decrease.
To reiterate I love that you have a model which can actually be scrutinised and meaningfully iterated—I still can’t really get my head around GiveWell’s but maybe I haven’t tried hard enough.
No matter what system is used error’s are going to be massive, so why not make it more understandable and editable?
I think the issue isn’t so much a constant −10%, but that some specific life-saving interventions might saves lives yet leave people with unusually low quality of life, and for those interventions the error term might be much larger than 10%.
Thank you Larks. This is a very good point and I fully agree.
In any cases where this happens it should be incorporated into our current model. That said I will check this for our current research and make sure that in any such cases (such as say pulmonary rehabilitation for COPD where patients are expected to have a lower quality of life if they survive) this is accounted for.
I might have been too directive when writing this post. I lack the organizational context and knowledge of how CEAs are used to say definitively that this should be changed. I ultimately agree that this is a small change that might not affect the decisions made, and it’s up to you to decide whether to account for it. However, some of the points you raised against updating this are incorrect.
I might have focused too much on the 10% reduction, while the real issue, as Elliot mentioned, is that you ignore two variables in the formula for DALYs averted:
Missing out on three 10% reductions in error X results in a difference of 0.1^3 = 27.1% which could be significant. I generally view organizations as growing through small iterative changes and optimization rather than big leaps.
My critique is only valid if you are trying to measure DALYs averted. If you choose to do something similar to GiveWell, which is more arbitrary, then it might not make sense to adjust for this anymore.
The three changes to the value of life saved come from different frameworks:
GiveWell values don’t represent DALYs averted but are mixed with other factors such as survey results.
HLI’s work is based on the assumption that death isn’t the worst possible state and that there is a baseline quality of life that must be met for a life to be worth living.
The change I’m suggesting is compatible with your current method of estimating the value of life saved. It doesn’t introduce any new assumptions; it simply makes some assumptions explicit. Unless you state something like, “We used those values initially but then detached them from their original formulas and now we will update them in another way,” my suggestion should fit within your existing framework.
EDIT:
I can’t say much about the GiveWell 1.5% rate, but I’ve heard it comes from the Rethink Priorities review, but it suggests 4.3% discount rate: can you direct me somewhere where I can read more about it?
Hi, Thank you. All good points. Fully agree with ongoing iterative improvement to our CEAs and hopefully you will see such improvements happening over the various research rounds (see also my reply to Nick). I also agree with picking up on specific cases where this might be a bigger issue (see my reply to Larks). I don’t think it is fair to say that we treat those two numbers as zero but it is fair to say we are currently using a fairly crude approximation to get at what those numbers are getting it in our lives saved calculations.
For a source on discounting see here: https://rethinkpriorities.org/publications/a-review-of-givewells-discount-rate#we-recommend-that-givewell-continue-discounting-health-at-a-lower-rate-than-consumption-but-we-are-uncertain-about-the-precise-discount-rate
“Discounting consumption vs. health benefits | Discount health benefits using only the temporal uncertainty component”