I don’t buy your line of argument that just because what you do “crude approximation” or “scratching the surface” (which I agree with and many commentators have pointed out) is a reason not to include new variables to your model. That seems like conflating two issues. Obviously assumptions and errors are massive, but that’s not a reason not to continuously be improving models.
Like @Larks says its not about a constant 10% reduction, but taking something extra into account which makes sense could meaningfully be added to the model.
Just because it might not be the most important thing right now or “low down your list” isn’t a reason from my perspective not to include it.
I agree that you are probably underestimating rather than overstimating in general, and I actually prefer AIM/CEs approach to GiveWell’s partly for the reason that this post exists—it is a bit easier to scrutinise and criticise, and therefore potentially easier to improve.
As a side note your response (probably unintentionally) feels just a little unnecesarily defensive but I might be reading too much into it.
Hi Nick, Thank you very much for the comment. These are all good points.
I fully agree with you and Larks that where a specific intervention will have reduced impact due to long run health effects this should be included in our models and I will check this is happening.
I apologise for the defensiveness and made a few minor edits to the post trying to keep content the same.
That’s not a reason not to continuously be improving models.
To be clear, we are always always improving our CEA models. This is an ongoing iterative process, and my hope is they get better year upon year. However, I guess I don’t have confidence right now that a −10% change to this number is actually improving the model or affecting our decision making.
If we dive into these numbers just a bit, I immediately notice that the discount rate in the GBD data is higher than ours and that should suggest that, if we are adjusting these numbers, that probably we want a significant +increase not decrease. But that then raises the question of what discount rate we are using and why, which has a huge effect on some of the models – and this is something there are currently internal debates in the team about, and we are looking at changing. But this then raises a question about how to represent the uncertainty about these numbers in our models and ensure the decision makers and readers are more aware of the inherent estimations that can have big effect on CEA outputs – and improving this is probably towards the top of my list.
I just want to flag that I’ve raised the issue of the inconsistencies in the use of discount rate (if by “the discount rate in the GBD data” you mean the 3% or 4% discount rate in the standard inputs table) in an email sent a few days ago to one of the CE employees. Unfortunately, we failed to have a productive discussion, as the conversation died quickly when CE stopped responding. Here is one of the emails I sent:
Hi [name],
I might be wrong but you are using 1.4% rate in the CEA but the value of life saved at various ages is copied from GiveWell standard inputs that uses 4% discount rate to calculate the value. Isn’t this an inconsistency?
Makes sense that you aren’t confident about this particular change yet, are discussing improvement with the team and that you are concerned about the overall situation that on balance you think your numbers are too low and you probably want a increase rather than a decrease.
To reiterate I love that you have a model which can actually be scrutinised and meaningfully iterated—I still can’t really get my head around GiveWell’s but maybe I haven’t tried hard enough.
No matter what system is used error’s are going to be massive, so why not make it more understandable and editable?
I don’t buy your line of argument that just because what you do “crude approximation” or “scratching the surface” (which I agree with and many commentators have pointed out) is a reason not to include new variables to your model. That seems like conflating two issues. Obviously assumptions and errors are massive, but that’s not a reason not to continuously be improving models.
Like @Larks says its not about a constant 10% reduction, but taking something extra into account which makes sense could meaningfully be added to the model.
Just because it might not be the most important thing right now or “low down your list” isn’t a reason from my perspective not to include it.
I agree that you are probably underestimating rather than overstimating in general, and I actually prefer AIM/CEs approach to GiveWell’s partly for the reason that this post exists—it is a bit easier to scrutinise and criticise, and therefore potentially easier to improve.
As a side note your response (probably unintentionally) feels just a little unnecesarily defensive but I might be reading too much into it.
Hi Nick, Thank you very much for the comment. These are all good points.
I fully agree with you and Larks that where a specific intervention will have reduced impact due to long run health effects this should be included in our models and I will check this is happening.
I apologise for the defensiveness and made a few minor edits to the post trying to keep content the same.
To be clear, we are always always improving our CEA models. This is an ongoing iterative process, and my hope is they get better year upon year. However, I guess I don’t have confidence right now that a −10% change to this number is actually improving the model or affecting our decision making.
If we dive into these numbers just a bit, I immediately notice that the discount rate in the GBD data is higher than ours and that should suggest that, if we are adjusting these numbers, that probably we want a significant +increase not decrease. But that then raises the question of what discount rate we are using and why, which has a huge effect on some of the models – and this is something there are currently internal debates in the team about, and we are looking at changing. But this then raises a question about how to represent the uncertainty about these numbers in our models and ensure the decision makers and readers are more aware of the inherent estimations that can have big effect on CEA outputs – and improving this is probably towards the top of my list.
I just want to flag that I’ve raised the issue of the inconsistencies in the use of discount rate (if by “the discount rate in the GBD data” you mean the 3% or 4% discount rate in the standard inputs table) in an email sent a few days ago to one of the CE employees. Unfortunately, we failed to have a productive discussion, as the conversation died quickly when CE stopped responding. Here is one of the emails I sent:
Nice one I love that response.
Makes sense that you aren’t confident about this particular change yet, are discussing improvement with the team and that you are concerned about the overall situation that on balance you think your numbers are too low and you probably want a increase rather than a decrease.
To reiterate I love that you have a model which can actually be scrutinised and meaningfully iterated—I still can’t really get my head around GiveWell’s but maybe I haven’t tried hard enough.
No matter what system is used error’s are going to be massive, so why not make it more understandable and editable?