That makes sense, and I think the tool does a great job of making those tradeoffs legible.
One thing I’ve found is that spending time in these settings can change how you think about some of the parameters, especially around counterfactuals and how multiple constraints interact in practice. Certain assumptions that look independent in a model often move together on the ground.
It would be interesting to see how that kind of correlated variation could be explored more systematically over time.
Model covariance in cost-effectiveness analyses is a good call-out, and I don’t know of anything that’s been shared on the EA forum, although apparently in health economics this is a solved problem so there’s an angle of attack there for anyone reading this who’s keen to give it a try. Quoting froolow:
… you’ll be pleased to know that this is basically a solved problem in Health Economics which I just skimmed over in the interests of time. The ‘textbook’ method of solving the problem is to use a ‘Cholesky Decomposition’ on the covariance matrix and sample from that. In recent years I’ve also started experimenting with microsimulating the underlying process which generates the correlated results, with some mixed success (but it is cool when it works!).
Practitioner input, e.g. from folks like you who’ve noticed this and have a sense of how much assumptions move together, would be needed to quantify the model covariance so it jives with what’s being seen.
That’s really interesting, Mo. Appreciate you sharing! The Cholesky approach definitely makes sense conceptually.
From a practitioner perspective, the correlations tend to come from fairly intuitive system dynamics rather than anything formal. So in Northern Nigeria, when outreach improved, you would often see several things move together. Coverage would go up, dropout rates would fall, and supply chains would stabilize as demand became more predictable. The opposite would happen when systems were under strain. Staffing gaps, stockouts, and lower uptake would start reinforcing each other quite quickly.
The tricky part is that those shifts are often uneven and very context specific. Translating them into a stable covariance structure is not straightforward. But I agree there’s probably a useful bridge here between how these dynamics play out operationally and how they could be reflected in models.
That makes sense, and I think the tool does a great job of making those tradeoffs legible.
One thing I’ve found is that spending time in these settings can change how you think about some of the parameters, especially around counterfactuals and how multiple constraints interact in practice. Certain assumptions that look independent in a model often move together on the ground.
It would be interesting to see how that kind of correlated variation could be explored more systematically over time.
Model covariance in cost-effectiveness analyses is a good call-out, and I don’t know of anything that’s been shared on the EA forum, although apparently in health economics this is a solved problem so there’s an angle of attack there for anyone reading this who’s keen to give it a try. Quoting froolow:
Practitioner input, e.g. from folks like you who’ve noticed this and have a sense of how much assumptions move together, would be needed to quantify the model covariance so it jives with what’s being seen.
That’s really interesting, Mo. Appreciate you sharing! The Cholesky approach definitely makes sense conceptually.
From a practitioner perspective, the correlations tend to come from fairly intuitive system dynamics rather than anything formal. So in Northern Nigeria, when outreach improved, you would often see several things move together. Coverage would go up, dropout rates would fall, and supply chains would stabilize as demand became more predictable. The opposite would happen when systems were under strain. Staffing gaps, stockouts, and lower uptake would start reinforcing each other quite quickly.
The tricky part is that those shifts are often uneven and very context specific. Translating them into a stable covariance structure is not straightforward. But I agree there’s probably a useful bridge here between how these dynamics play out operationally and how they could be reflected in models.