Sam and I have discussed this project in private and for the record will summarise a few of my comments here in case another reader had the similar thoughts. Overall, I’m supportive, and a big fan of doing analyses like this. At HLI we attempt to quantify uncertainty in our in depth analyses! We’ve also done a GiveDirectly cost-effectiveness analysis where we estimated the uncertainty for every parameter based on empirical estimates of variance. Hence, why I have views on the topic.
Here are the things I mentioned, and what Sam and I will likely continue to discuss.
Why quantify uncertainty in the first place?
We may care about uncertainty because we want to know about the relative uncertainty in the cost-effectiveness of interventions.
Or, we may have a desire to compare the likelihood of harm across two interventions. Both of which adding uncertainty would afford.
The argument we need to make isn’t just about it being better but it being worth the additional time and effort it takes.
Why Squiggle? And not Guesstimate, Causal or R / Python.
Note, you say:
It offers a large collection of advantages over other alternatives:
Ability to write detail inbetween cells
External links
Ability to quantify uncertainty
Ability to create calculators and interactive analysis
But Causal and Guesstimate / R notebooks appear to already do some / most of these? Indeed, in the appendix you go on to say:
In honesty, I could complete all of the research outcomes listed here with either of these two platforms. However, I see possibilities that Squiggle can far exceed the capabilities of these two platforms, mainly through the use of functions and its flexibility as a custom DSL for estimation. This analysis did not use these features, but Ozzie discussed them in a past post on Squiggle.
If functions is what you need, then why not R? Or write a package for R or Python? (I don’t know what DSL means).
For my part, I’m a pretty big fan of using R. Many Effective Altruist organisations are very google docs dependent, HLI included. But we’ve found a pretty satisfying workflow where we go: google sheets → R → back to google sheets → link summary tables to google docs. Hopefully we will make the code for this public if / when we update our analysis in the next year.
I’m all for quantifying uncertainty, but I would like to hear more about the need for a new language that will presumably need support through its life cycle? (Or am I missing something?) Squiggle makes a lot of sense for forecasting (which appears the intended purpose?). I don’t want to open R and write a script when I just want a quick and dirty Monte Carlo simulation to help me answer a metaculus question. For that purpose Squiggle and Guesstimate work well.
What are the implications?
Currently, the meatiest takeaway is “Quantifying uncertainty eventually found that the cost effectiveness of GiveDirectly varies by about an order of magnitude and is approximately lognormally distributed. The cost effectiveness, however doesn’t have particularly large tails.” and to paraphrase: “there are a lot of other possibilities!”
What does this mean practically?
Adding uncertainty allows you to easily do a sensitivity analysis where you find the inputs that explain the most variation in the results. You can then single these out for special attention and scrutiny. An investigation will look weird if 50% of the results are explained by that input that was a subjective guess.
I’d be VERY interested to see how the value of information stuff would be done because I don’t know how to do this.
For my part, I’m a pretty big fan of using R. Many Effective Altruist organisations are very google docs dependent, HLI included. But we’ve found a pretty satisfying workflow where we go: google sheets → R → back to google sheets → link summary tables to google docs. Hopefully we will make the code for this public if / when we update our analysis in the next year.
I love R and maybe there is a way to integrate this project into R? That said, I’d like to see this take on an ‘output’ format with some of the benefits shown in this notebook, and in dashboards like the one for Causal app.
What are the implications?
The connotation of the language “what are the implications” as a header , which I think you didn’t mean, could be “what’s the big deal, not impressed”. If someone (not you :) ) were to say that, I’d suggest also we need to give it a little more time, and in particular, have at least two models to compare, for at least 2 interventions/charities.
Your discussion of ‘sensitivity analysis’ seems like this could lead to a lot of value. I’ve seen a bit about the VOI stuff but I’m also not versed in it; but we could get someone on board who is.
When you wrote
I don’t want to open R and write a script when I just want a quick and dirty Monte Carlo simulation to help me answer a metaculus question. For that purpose Squiggle and Guesstimate work well.
I don’t want to open R and write a script when I just want a quick and dirty Monte Carlo simulation to help me answer a metaculus question. For that purpose Squiggle and Guesstimate work well.
Wikipedia lists some main advantages and disadvantages, I mostly agree with this section and think in most cases we should go for an R, Python or Apps Script/js library or framework, but there are cases of successful DSLs (e.g. LaTeX). I’m curious what the target audience is for squiggle, maybe professional forecasters that are not programmers?
Sam and I have discussed this project in private and for the record will summarise a few of my comments here in case another reader had the similar thoughts. Overall, I’m supportive, and a big fan of doing analyses like this. At HLI we attempt to quantify uncertainty in our in depth analyses! We’ve also done a GiveDirectly cost-effectiveness analysis where we estimated the uncertainty for every parameter based on empirical estimates of variance. Hence, why I have views on the topic.
Here are the things I mentioned, and what Sam and I will likely continue to discuss.
Why quantify uncertainty in the first place?
We may care about uncertainty because we want to know about the relative uncertainty in the cost-effectiveness of interventions.
Or, we may have a desire to compare the likelihood of harm across two interventions. Both of which adding uncertainty would afford.
The argument we need to make isn’t just about it being better but it being worth the additional time and effort it takes.
Why Squiggle? And not Guesstimate, Causal or R / Python.
Note, you say:
But Causal and Guesstimate / R notebooks appear to already do some / most of these? Indeed, in the appendix you go on to say:
If functions is what you need, then why not R? Or write a package for R or Python? (I don’t know what DSL means).
For my part, I’m a pretty big fan of using R. Many Effective Altruist organisations are very google docs dependent, HLI included. But we’ve found a pretty satisfying workflow where we go: google sheets → R → back to google sheets → link summary tables to google docs. Hopefully we will make the code for this public if / when we update our analysis in the next year.
I’m all for quantifying uncertainty, but I would like to hear more about the need for a new language that will presumably need support through its life cycle? (Or am I missing something?) Squiggle makes a lot of sense for forecasting (which appears the intended purpose?). I don’t want to open R and write a script when I just want a quick and dirty Monte Carlo simulation to help me answer a metaculus question. For that purpose Squiggle and Guesstimate work well.
What are the implications?
Currently, the meatiest takeaway is “Quantifying uncertainty eventually found that the cost effectiveness of GiveDirectly varies by about an order of magnitude and is approximately lognormally distributed. The cost effectiveness, however doesn’t have particularly large tails.” and to paraphrase: “there are a lot of other possibilities!”
What does this mean practically?
Adding uncertainty allows you to easily do a sensitivity analysis where you find the inputs that explain the most variation in the results. You can then single these out for special attention and scrutiny. An investigation will look weird if 50% of the results are explained by that input that was a subjective guess.
I’d be VERY interested to see how the value of information stuff would be done because I don’t know how to do this.
I love R and maybe there is a way to integrate this project into R? That said, I’d like to see this take on an ‘output’ format with some of the benefits shown in this notebook, and in dashboards like the one for Causal app.
The connotation of the language “what are the implications” as a header , which I think you didn’t mean, could be “what’s the big deal, not impressed”. If someone (not you :) ) were to say that, I’d suggest also we need to give it a little more time, and in particular, have at least two models to compare, for at least 2 interventions/charities.
Your discussion of ‘sensitivity analysis’ seems like this could lead to a lot of value. I’ve seen a bit about the VOI stuff but I’m also not versed in it; but we could get someone on board who is.
When you wrote
Did you mean ‘Causal and Guesstimate’?
When you wrote
Did you mean ‘Causal and Guesstimate’?
It’s a Domain Specific Language: a programming language that’s optimized for a specific domain.
Wikipedia lists some main advantages and disadvantages, I mostly agree with this section and think in most cases we should go for an R, Python or Apps Script/js library or framework, but there are cases of successful DSLs (e.g. LaTeX).
I’m curious what the target audience is for squiggle, maybe professional forecasters that are not programmers?
I’ll write more of a post about it in a few weeks. Right now it’s not really meant for external use; the API isn’t quite stable.
That said, you can see some older posts which give the main idea: https://www.lesswrong.com/s/rDe8QE5NvXcZYzgZ3