Thanks Benita, really appreciate the field perspective. You’re right that the parameters are a snapshot — the tool takes GiveWell’s November 2025 spreadsheet values as given and doesn’t attempt to model how they change over time. GiveWell updates their spreadsheets periodically as they get new data, and the tool would need to be re-extracted to reflect that.
On within-country variation, this is a real limitation. The model treats each country as a single unit with one set of parameters, but as you note, conditions in Sokoto vs. other parts of Nigeria can be very different. The sensitivity analysis helps show how much the result depends on any single parameter (like counterfactual coverage), but it doesn’t capture the kind of correlated shifts you’re describing where multiple parameters move together as systems evolve.
I built this as a side project to make GiveWell’s existing estimates more explorable, not to improve on their parameter estimation — that’s solidly their domain expertise. But the tool does make it easy to test “what if coverage in Zamfara drops by 10%” type questions, which I think is part of what you’re getting at.
That makes sense, and I think the tool does a great job of making those tradeoffs legible.
One thing I’ve found is that spending time in these settings can change how you think about some of the parameters, especially around counterfactuals and how multiple constraints interact in practice. Certain assumptions that look independent in a model often move together on the ground.
It would be interesting to see how that kind of correlated variation could be explored more systematically over time.
Model covariance in cost-effectiveness analyses is a good call-out, and I don’t know of anything that’s been shared on the EA forum, although apparently in health economics this is a solved problem so there’s an angle of attack there for anyone reading this who’s keen to give it a try. Quoting froolow:
… you’ll be pleased to know that this is basically a solved problem in Health Economics which I just skimmed over in the interests of time. The ‘textbook’ method of solving the problem is to use a ‘Cholesky Decomposition’ on the covariance matrix and sample from that. In recent years I’ve also started experimenting with microsimulating the underlying process which generates the correlated results, with some mixed success (but it is cool when it works!).
Practitioner input, e.g. from folks like you who’ve noticed this and have a sense of how much assumptions move together, would be needed to quantify the model covariance so it jives with what’s being seen.
That’s really interesting, Mo. Appreciate you sharing! The Cholesky approach definitely makes sense conceptually.
From a practitioner perspective, the correlations tend to come from fairly intuitive system dynamics rather than anything formal. So in Northern Nigeria, when outreach improved, you would often see several things move together. Coverage would go up, dropout rates would fall, and supply chains would stabilize as demand became more predictable. The opposite would happen when systems were under strain. Staffing gaps, stockouts, and lower uptake would start reinforcing each other quite quickly.
The tricky part is that those shifts are often uneven and very context specific. Translating them into a stable covariance structure is not straightforward. But I agree there’s probably a useful bridge here between how these dynamics play out operationally and how they could be reflected in models.
Thanks Benita, really appreciate the field perspective. You’re right that the parameters are a snapshot — the tool takes GiveWell’s November 2025 spreadsheet values as given and doesn’t attempt to model how they change over time. GiveWell updates their spreadsheets periodically as they get new data, and the tool would need to be re-extracted to reflect that.
On within-country variation, this is a real limitation. The model treats each country as a single unit with one set of parameters, but as you note, conditions in Sokoto vs. other parts of Nigeria can be very different. The sensitivity analysis helps show how much the result depends on any single parameter (like counterfactual coverage), but it doesn’t capture the kind of correlated shifts you’re describing where multiple parameters move together as systems evolve.
I built this as a side project to make GiveWell’s existing estimates more explorable, not to improve on their parameter estimation — that’s solidly their domain expertise. But the tool does make it easy to test “what if coverage in Zamfara drops by 10%” type questions, which I think is part of what you’re getting at.
That makes sense, and I think the tool does a great job of making those tradeoffs legible.
One thing I’ve found is that spending time in these settings can change how you think about some of the parameters, especially around counterfactuals and how multiple constraints interact in practice. Certain assumptions that look independent in a model often move together on the ground.
It would be interesting to see how that kind of correlated variation could be explored more systematically over time.
Model covariance in cost-effectiveness analyses is a good call-out, and I don’t know of anything that’s been shared on the EA forum, although apparently in health economics this is a solved problem so there’s an angle of attack there for anyone reading this who’s keen to give it a try. Quoting froolow:
Practitioner input, e.g. from folks like you who’ve noticed this and have a sense of how much assumptions move together, would be needed to quantify the model covariance so it jives with what’s being seen.
That’s really interesting, Mo. Appreciate you sharing! The Cholesky approach definitely makes sense conceptually.
From a practitioner perspective, the correlations tend to come from fairly intuitive system dynamics rather than anything formal. So in Northern Nigeria, when outreach improved, you would often see several things move together. Coverage would go up, dropout rates would fall, and supply chains would stabilize as demand became more predictable. The opposite would happen when systems were under strain. Staffing gaps, stockouts, and lower uptake would start reinforcing each other quite quickly.
The tricky part is that those shifts are often uneven and very context specific. Translating them into a stable covariance structure is not straightforward. But I agree there’s probably a useful bridge here between how these dynamics play out operationally and how they could be reflected in models.