Math Pedantic/Computer Science Honours interested in Cause Prioritization. Currently working with QURI on Squiggle and Pedant.
SamNolan
Karma: 796
Predicting the cost-effectiveness of future R&D projects and academic research
Estimating the cost-effectiveness of previous R&D projects
EA Software Engineers’ Office Hours (option 2)
Predicting the cost-effectiveness of deploying a new intervention
Value of Information
Forecasting the cost-effectiveness of trying something new
Ahh, that makes sense. I think “250 Hens a year” sounds like “250 Hens/year” not “250 hens * year”. That’s probably where I got my mistake
[Question] How many chickens are being saved by corporate campaigns?
Because this comes up when googling street outreach, as President of EA Melbourne (the EA group that ran the above-mentioned event), I’d love to tell you how it went.
Interestingly, people in the public seem open to ideas of effective altruism. However, the conversion rate is truly tiny, no one we saw on that day came to any future event. In the end, we decided that this was not a worthwhile activity.
Some interesting notes however:
People, especially in the current political climate (referring to Russia invading Ukraine here), are actually quite supportive of longtermist ideas! This is probably because longtermist ideas are the only types of problems that people face in developed nations (in comparison to Animal Welfare and Global Health and Development). We ran a giving game between animal welfare, global poverty and longtermist ideas, and the money spread was fairly even.
Almost no one puts any thought into where their money goes, although this may be just because they didn’t want to strike a conversation up with a stranger. Many people followed just what seemed like a good idea, sometimes confusing cause areas (for instance, thinking “Global Health” is about environmentalism, or possibly that “Animal Welfare” is about helping pet animals etc)
As is usual with street outreach, younger people are much more open to discussion.
This sadly, will likely never happen, or at least not for a few years. This was never within Squiggle’s scope. Squiggle currently has much more critical issues before adding such a feature!
Thanks for checking out this post! This is an old one, and I’m no longer as interested in dimensional checking in models. However, I may come back to this project because I have a feeling it could be used to optimize Squiggle code as well as offer dimensional checking.
What do you mean by “Combines the strengths of”? To me, Squiggle is the successor of Guesstimate, and the strengths of Squiggle + the strengths of Guesstimate = the strengths of Squiggle? What features are you looking for?
When I started this project, Squiggle was not in a state where Pedant code could be translated into Squiggle. This is no longer the case, so if there was sufficient interest, I could finish my project to get Pedant to compile to Squiggle for execution. This would likely take a week of work or so.
However, as much as I love this project, I don’t want to do that until I’ve exhausted any easier option of doing the research I want to do + worked out the theory required to create optimizations. There’s a possibility that this need will arise once I try to run more Value of Perfect Partial Information analysis for my GiveWell work which is quite slow in Squiggle, or possibly if I could work with the much larger models funders run.
Hey! Love the post. Just putting my comments here as they go.
Tldr This seems to be a special case of the more general theory of Value of Information. There’s a lot to be said about value of information, and there are a couple of parameter choices I would question.
The EA Forum supports both Math and Footnotes now! Would be lovely to see them included for readability.
I’m sure you’re familiar with Value of Information. It has a tag on the EA Forum. It seems as if you have presumed the calculations around value of information (For instance, you have given a probability and better-than-top-charity ratio , both of which can be explicitly calculated with Value of Information). The rest of the calculations seem valid and interesting.
For instance, when the total budget is 1 billion dollars, then this equation entails that a research project that costs 1 million dollars (c/(m-c)=0.001) is worth funding if it has at least a 1% chance (p=0.01) of producing an intervention that is at least 10% more cost-effective (n=1.1) than the best existing intervention. This is a surprisingly low bar relative to how hard it is to get funding for EA-aligned academic research
I might be wrong, but I think this is assuming that this is the only research project that is happening. I could easily assume that EA spends more than 0.1% of it’s resources on identifying/evaluting new interventions. Although, I’m yet to know of how to do the math with multiple research projects. It’s currently a bit beyond me.
There’s a common bias to choose numbers within, say and that may bias this investigation. For instance, when I calculated the value of information on GiveDirectly, when was . If you are unsure about whether a charity is cost-effective, often the tails of your certainty can drop fast.
Your “lower bound” is entirely of your own construction. It’s derived from your decleration at the start that p is the chance that you find a “investing dollars into the research generates an intervention that is at least times as effective as the best existing intervention with probability . If I was to call your construction the “Minimum value of information”, it’s possible to calculate the “Expected value of [Perfect|imperfect] information”, which I feel like might be a more useful number. Guesstimate can do this as well, I could provide an example if you’d like.
We have to remember that we are still uncertain about the cost-effectiveness of the new intervention, which means it would need to be expected to be more cost-effective even after considering all priors. This may increase or decrease . However, this is probably irrelevant to the argument.
Amusingly, we seem to come at this at two very different angles, I have a bias that I’d like EA to spend less on research (or less on research in specific directions) and you’re here to try and convince EA to spend more on research! Love you’re work, I’ll get onto your next post and we’ll chat soon.
Quantifying Uncertainty in GiveWell Cost-Effectiveness Analyses
And to add to this, very recently there was a post Quantifying the Uncertainty in AMF! Which still seems a bit in the works but I’m super excited for it.
My Hecking Goodness! This is the coolest thing I have ever seen in a long time! You’ve done a great job! I am like literally popping with excitement and joy. There’s a lot you can do once you’ve got this!
I’ll have to go through the model with a finer comb (and look through Nuno’s recommendations) and probably contribute a few changes, but I’m glad you got so much utility out of using Squiggle! I’ve got a couple of ideas on how to manage the multiple demographics problem, but honestly I’d love to have some chats with you about next steps for these models.
Hello! My goodness I love this! You’ve really written this in a super accessible way!
Some citations: I have previously Quantified the Uncertainty in the GiveDirectly CEA (using Squiggle). I believe the Happier Lives Institute has done the same thing, as did cole_haus who didn’t do an analysis but built a framework for uncertainty analysis (much like I think you did). I just posted a simple example of calculating the Value of Information on GiveWell models. There’s a question about why GiveWell doesn’t quantity uncertainty
My partner Hannah currently has a grant where she’s working on quantifying the uncertainty of other GiveWell charities using techniques similar to mine, starting with New Incentives. Hopefully, we’ll have fruit to show for other GiveWell charities! There is a lot of interest in this type of work.
I’d love to chat with you (or anyone else interested in Uncertainty Quantification) about current methods and how we can improve them. You can book me on calendly. I’d still learning a lot about how to do this sort of thing properly, and am mainly learning by trying, so I would love to have a chat about ways to improve.
Value of Infomation, an example with GiveDirectly
Hello! Thanks for showing interest in my post.
First of all, I don’t represent GiveWell or anyone else but myself, so all of this is more or less speculation.
My best guess as why GiveWell does not quantify uncertainty in their estimates is because the technology to do this is still somewhat primitive. The most mature candidate I see is Causal, but even then it’s difficult to identify how one might do something like have multiple parallel analyses of the same program but in different countries. GiveWell has a lot of requirements that their host plaftorm needs t ohave. Google Sheets has the benefit that it can be used, understood, and edited by anyone. I’m currently working on Squiggle with QURI to make sweeten the deal to quantifying uncertainty explicitly, but there’s a long way to go before it becomes somehing that could be readily understood and trusted to be stable like Google Sheets.
On a second note, I would also say that providing lower and upper estimates for cost-effectiveness for its top charities wouldn’t actually be that valuable, in the sense that it doesn’t influence any real world decisions. I know that I decided to spend hours making the GiveDirectly quantification but in truth, the information gained from it directly is extremely little. The main reason I did it is that it makes a great proof of concept for usage in non-GiveWell fields which need it much more.
There are two reasons why there is so little information gained from it:
The uncertainty of GiveDirectly and other GiveWell supported charities is not actually that high (about an order of magnitude for GiveDirectly, I expect over 2-3 orders of magnitude for the others). For instance, I never expected in my quantifaction of uncertainty in GiveDirectly that there would be practically any probability mass of it being more effective than AMF. At least before counting for things like moral uncertainty.
My uncertainty about my chosen uncertainties are really high. If you strip away how fancy my work looks and just look at what I’ve contributed in comparison to what GiveWell has done, I’ve practically copied GiveWell’s work and pulled some numbers out of thin air for uncertanity with the help of Nuno. Some Bayesian Analysis is done under questionable assumptions etc.
I see much more value in quantifying uncertainty when we might expect the uncertainty to be much larger, for instance, when dealing with moral uncertainty, or animal welfare/longtermist interventions.
Haha, I came up with that example as well. You’re thinking about this in the same way I did!
I think to say that one is the “actual objective” is not very rigorous. Although I’m saying this from a place of making that same argument. It does answer a valid question of “how much money should one donate to get an expected 1 unit of good” (which is also really easy to communicate, dollars per life saved is much easier to talk about than lives saved per dollar). I’ve been thinking about it for a while and put a comment under Edo Arad’s one.
As for the second point about simple going . I agree that this is likely an error, and you have a good counterexample.
Sorry for the late comment, but I was wondering:
Why do you think it’s an underestimate?