I appreciate your taking the time to write up your decision process again, Michael. As you have said, by making the process more explicit it makes it easier for others to check and contribute to the process, and produces knowledge that others can use as a public good.
In this case I think the model you are using suffers from some serious flaws, both in the model structure and in the parameter values. I will discuss modifications to the parameters in your app, and readers may wish to open that alongside this comment to examine sensitivities.
To roadmap:
I think you are using a prior that makes false predictions and Charity Doomsday Arguments
The results are extremely sensitive to minor disturbances in variance that are amplified by your prior and modeling choices
In particular, the model frequently treats very good news as very bad news
I see multiple dubious parameter choices that switch the charity recommended by the model for long-run impact
Here’s one example of perverse behavior. Under the GFI model, you give a credence interval for the number of people who would switch to cultured meat if it were available, with a 10th percentile of 500 million and a 90th percentile of 2 billion; but if we raise the 90th percentile to 6 billion people (most of the world population), the posterior value of GFI falls (from 9375 to 1855 to in the ‘direct effects’ section, and from 2.6e+33 to 9.1e+32 in long-run effects).
Evidence that leads us to update to having the same 10th percentile and a higher 90th percentile is good news about GFI, but in the model it drops the value of GFI by 3-5x, enough to change the model’s recommendation away from GFI. And if meat substitutes eventually turn out to be cheaper, healthier, better for the environment, and avert the cruelty of factory farming why think 6 billion is too high for the 90th percentile? If we also allow for a 10th percentile of 100 million the values fall further to 138 (vs 9375 to start, an ~68x decline) and 1e+32 (a 25x decline).
Why does this happen? You have input a prior distribution over cost-effectiveness that assigns vanishingly low and rapidly decreasing probabilities to very large impacts. In particular it penalizes impacts on the long-run future of civilization by 15-20 orders of magnitude. In other words, it might appear that we know a lot about the rate at which dinosaur-killer asteroids impact the Earth (based on craters, meteor remnants, astronomical observations, etc), and that we were able to set up an asteroid watch system to give us sufficient advance warning to deflect such an asteroid for $100MM. The scientific evidence seems to indicate pretty clearly that we can track and stop asteroids, but your prior says with overwhelming confidence that we can’t do so if preventing a dinosaur-killer hitting the Earth would have very large benefits. On the other hand, if we knew we were about to be wiped out by something else, then we could believe the astronomers, since it wouldn’t be that good to stop asteroids.
Likewise with GFI and vegan promotion. In your model, the more long-run flow-through effects of getting people to stop patronizing factory farms, the more your prior predicts it is essentially impossible to convert someone to veganism or develop cultured meat, since it would be too beneficial. This Charity Doomsday Argument lets us make all sorts of empirically wrong predictions about the world we live in.
Essentially the prior in the model acts like an adversarial agent: whenever there is any uncertainty about anything that could affect cost-effectiveness the model overwhelmingly favors the worst case. Good news about the upper bound reach of GFI makes your model consider it much worse because you treat this as evidence of increased variance and of the extreme worst case being much worse. Many of the conclusions generated by your app stem from artifacts of this effect.
In addition, I continue to find many of the parameters problematic.
For example, in the model most of the value of GFI comes from advanced civilizations converting resources into well-being at peak efficiency (ecstatic dense AIs, etc); setting the probability of hedonium and dolorium to 0 lowers the posterior for GFI to 8e+20 from 2.6e+33. In the model GFI affects this by developing meat substitutes, and people who do not eat meat are then more likely to produce hedonium and avoid producing dolorium. But any society capable of interstellar colonization and producing hedonium would be perfectly capable of producing meat without animals, or powering themselves and enjoying sensory pleasures without either. Earlier availability would be relevant for the shape of later periods only insofar as the later availability doesn’t substitute. The model does not appear to adequately account for this (although it has some relevant parameters).
We could shoehorn this consideration into the ‘memetically relevant humans’ parameter under Veg Advocacy, which currently has a 10th percentile of 1 billion and 90th percentile of 2 billion. If we increase the 90th percentile there to 200 billion to reflect future generations having more of the control over the beings they create, that alone switches the recommendation of your model away from GFI.
[Disclosure: I work at the Future of Humanity Institute, and consult for the Open Philanthropy Project. I have previously worked at the Machine Intelligence Research Institute (and my wife is on its board), and volunteered and consulted for the Center for Effective Altruism, particularly 80,000 Hours. I am writing only for myself.]
Also, it would be helpful if you said more about how you think I should do things. Should I not use a Bayesian prior at all? Should I use a wider prior or a different distribution? Should I model interventions in a different way? How do you think I could do better?
Right now all I know is that any approach has lots of problems, and my current approach seems the least problematic. If you think something else would be better, please say what it is and why you prefer it.
Compared to this, I would use something that looks more like standard cost-effectiveness analysis. Rather than use the Doomsday Prior and variance approach to assess robustness (which is ultra-sensitive to errors about variance for ranking options, at least comparably to errors about EV in cost-effectiveness analysis) my recommendationsI would include the following:
Do multiple cost-effectiveness analyses using different approaches, methodologies, and people (overall and of local variables); seek robustness in the analysis by actually doing it differently
Use empirical and intuitive priors about more local variables (e.g. make use of data about the success rates of past scientific research and startups in forming your prior about the success of meat substitute research, without hugely penalizing the success rate because the topic has more utilitarian value by your lights) rather than global
Assess the size of the future separately from our ability to prevent short-run catastrophic risks or produce short-run value changes (like adding vegans)
Focus on relative impact of different actions, rather than absolute impact in QALYs (the size of the future mostly cancels out, and even a future as large as Earth’s past is huge relative to the present, hundreds of millions of years; actions like cash transfers also have long-run effects, although a lot less than actions optimized for long-run effects)
Vigorously seek out criticism, missing pieces, and improvements for the models and their components
Part of what I’m getting at is a desire to see you defend the wacky claims implicit in your model posteriors. You present arguments for how the initial estimates could make sense, but not for how the posteriors could make sense. And as I discuss above, it’s hard to make them make sense, and that counts against the outputs.
So I’d like to see some account of why your best picture of the world is in so much tension with your prior, and how we could have an understanding of the world that is consistent with your posterior.
Thanks, this is exactly the sort of thing I was looking for.
Slightly unrelated but:
Part of what I’m getting at is a desire to see you defend the wacky claims implicit in your model posteriors.
The wacky claims you’ve talked about here relate to far-future posteriors. Do you also mean the direct effect posteriors imply wacky claims? I know you’ve said before you think the way I set a prior is arbitrary, is there anything else?
I generally agree with this. My model has lots of problems and certainly shouldn’t be taken 100% seriously, Perhaps I should have said more about this earlier but I wasn’t really thinking about it until you brought it up, so it’s probably good that you did.
It sounds like you’re raising 2-3 main issues:
Posteriors have weird behavior.
1a. Increasing the upper bound on a confidence interval reduces the posterior.
1b. Far future effects are penalized in a way that has some counterintuitive implications.
The inputs for GFI’s far future effects look wrong.
For (2), I don’t think they’re as wrong as you think they are, but nonetheless I don’t really rely on these and I wouldn’t suggest that people do. I could get more into this but it doesn’t seem that important to me.
For (1a), widening a confidence interval generally makes a posterior get worse. This sort of makes intuitive sense because if your cost-effectiveness estimate has a wider distribution, it’s more likely you’re making a mistake and your expected value estimate is too high. Maybe the current implementation of my model exaggerates this too much; I’m not really sure what the correct behavior here should be.
For (1b), this is a big problem and I’m not sure what to do about it. The obvious other thing to do is to essentially take claims about the far future literally—if calculations suggest that things you do now affect 10^55 QALYs in the far future, then that’s totally reasonable. (Obviously these aren’t the only things you can do, but you can move in the direction of “take calculations more seriously” or “take calculations less seriously”, and there are different ways to do that.) I don’t believe we should take those sorts of claims entirely seriously. People have a history of over-estimating cost-effectiveness of interventions, and we should expect this to be even more overstated for really big far future effects; so I believe there’s a good case for heavily discounting these estimates. I don’t know how to avoid both the “affecting anything is impossible” problem and the “take everything too seriously” problem, and I’d be happy to hear suggestions about how to fix it. My current strategy is to combine quantitative models with qualitative reasoning, but I’d like to have better quantitative models that I could rely on more heavily.
So, how should this model be used? I’m generally a fan of cost-effectiveness calculations. I calculated expected values for how MFA, GFI, and ACE directly reduce suffering, and my prior adjusts these based on robustness of evidence. The way I calculate posteriors has problems, but I believe the direct-effect posteriors better reflect reality than the raw estimates (I have no idea about the far-future posteriors). If you don’t like the way I use priors, you can just look at the raw estimates. I wouldn’t take these estimates too seriously since my calculations are pretty rough, but it at least gives some idea of how they compare in quantitative terms, and I believe it doesn’t make much sense to decide where to donate without doing something like this.
or (1b), this is a big problem and I’m not sure what to do about it. The obvious other thing to do is to essentially take claims about the far future literally—if calculations suggest that things you do now affect 10^55 QALYs in the far future, then that’s totally reasonable. (Obviously these aren’t the only things you can do, but you can move in the direction of “take calculations more seriously” or “take calculations less seriously”, and there are different ways to do that.)
I think you would benefit a lot from separating out ‘can we make this change in the world, e.g. preventing an asteroid from hitting the Earth, answering this scientific question, convincing one person to be vegan’ from the size of the future. A big future (as big as the past, the fossil records shows billions of years of life) doesn’t reach backwards in time to warp all of these ordinary empirical questions about life today.
It doesn’t even have much of an efficient market effect, because essentially no actors are allocating resources in a way that depends on whether the future is 1,000,000x as important as the past 100 years or 10^50. Indeed almost no one is allocating resources as though the next 100 million years are 10x as important as the past 100 years.
The things that come closest are generic Doomsday Arguments, and the Charity Doomsday Prior can be seen as falling into this class. The best cashed out relies on the simulation argument:
Let the size of the universe be X
Our apparent position seems to let us affect a big future, with value that grows with X
The larger X is, the greater the expected number of simulations of people in seemingly pivotal positions like ours
So X appears on both sides of the equation, cancels out, and the value of future relative to past is based on the ratio of the size and density of value in simulations vs basement reality
You then get figures like 10^12 or 10^20, not 10^50 or infinity, for the value of local helping (multiplied by the number of simulations) vs the future
Your Charity Doomsday Prior might force you into a view something like that, and I think it bounds the differences in expected value between actions. Now this isn’t satisfying for you because you (unlike Holden in the thread where you got the idea for this prior) don’t want to assess charities in their relative effects, but in absolute effects in QALY-like units. But it does mean that you don’t have to worry about 10^55x differences.
In any case, in your model the distortions get worse as you move further out from the prior, and much of that is being driven by the estimates of the size of the future (making the other problems worse). You could try reformulating with an empirically-informed prior over success at political campaigns, scientific discovery, and whatnot, and then separately assess the value of the different goals according to your total utilitarianish perspective. Then have a separate valuation of the future.
The way I calculate posteriors has problems, but I believe the direct-effect posteriors better reflect reality than the raw estimates (I have no idea about the far-future posteriors). For (2), I don’t think they’re as wrong as you think they are, but nonetheless I don’t really rely on these and I wouldn’t suggest that people do. I could get more into this but it doesn’t seem that important to me.
I’m not sure I fully understand what you are relying on. I think your model goes awry on estimating the relative long-run fruit of the things you consider, and in particular on the bottom-line conclusion/ranking. If I wanted to climb the disagreement hierarchy with you, and engage with and accept or reject your key point, how would you suggest I do it?
You then get figures like 10^12 or 10^20, not 10^50 or infinity, for the value of local helping (multiplied by the number of simulations) vs the future
Are the adjusted lower numbers based on calculations such as these?
I appreciate your taking the time to write up your decision process again, Michael. As you have said, by making the process more explicit it makes it easier for others to check and contribute to the process, and produces knowledge that others can use as a public good.
In this case I think the model you are using suffers from some serious flaws, both in the model structure and in the parameter values. I will discuss modifications to the parameters in your app, and readers may wish to open that alongside this comment to examine sensitivities.
To roadmap:
I think you are using a prior that makes false predictions and Charity Doomsday Arguments
The results are extremely sensitive to minor disturbances in variance that are amplified by your prior and modeling choices
In particular, the model frequently treats very good news as very bad news
I see multiple dubious parameter choices that switch the charity recommended by the model for long-run impact
Here’s one example of perverse behavior. Under the GFI model, you give a credence interval for the number of people who would switch to cultured meat if it were available, with a 10th percentile of 500 million and a 90th percentile of 2 billion; but if we raise the 90th percentile to 6 billion people (most of the world population), the posterior value of GFI falls (from 9375 to 1855 to in the ‘direct effects’ section, and from 2.6e+33 to 9.1e+32 in long-run effects).
Evidence that leads us to update to having the same 10th percentile and a higher 90th percentile is good news about GFI, but in the model it drops the value of GFI by 3-5x, enough to change the model’s recommendation away from GFI. And if meat substitutes eventually turn out to be cheaper, healthier, better for the environment, and avert the cruelty of factory farming why think 6 billion is too high for the 90th percentile? If we also allow for a 10th percentile of 100 million the values fall further to 138 (vs 9375 to start, an ~68x decline) and 1e+32 (a 25x decline).
Why does this happen? You have input a prior distribution over cost-effectiveness that assigns vanishingly low and rapidly decreasing probabilities to very large impacts. In particular it penalizes impacts on the long-run future of civilization by 15-20 orders of magnitude. In other words, it might appear that we know a lot about the rate at which dinosaur-killer asteroids impact the Earth (based on craters, meteor remnants, astronomical observations, etc), and that we were able to set up an asteroid watch system to give us sufficient advance warning to deflect such an asteroid for $100MM. The scientific evidence seems to indicate pretty clearly that we can track and stop asteroids, but your prior says with overwhelming confidence that we can’t do so if preventing a dinosaur-killer hitting the Earth would have very large benefits. On the other hand, if we knew we were about to be wiped out by something else, then we could believe the astronomers, since it wouldn’t be that good to stop asteroids.
Likewise with GFI and vegan promotion. In your model, the more long-run flow-through effects of getting people to stop patronizing factory farms, the more your prior predicts it is essentially impossible to convert someone to veganism or develop cultured meat, since it would be too beneficial. This Charity Doomsday Argument lets us make all sorts of empirically wrong predictions about the world we live in.
Essentially the prior in the model acts like an adversarial agent: whenever there is any uncertainty about anything that could affect cost-effectiveness the model overwhelmingly favors the worst case. Good news about the upper bound reach of GFI makes your model consider it much worse because you treat this as evidence of increased variance and of the extreme worst case being much worse. Many of the conclusions generated by your app stem from artifacts of this effect.
In addition, I continue to find many of the parameters problematic.
For example, in the model most of the value of GFI comes from advanced civilizations converting resources into well-being at peak efficiency (ecstatic dense AIs, etc); setting the probability of hedonium and dolorium to 0 lowers the posterior for GFI to 8e+20 from 2.6e+33. In the model GFI affects this by developing meat substitutes, and people who do not eat meat are then more likely to produce hedonium and avoid producing dolorium. But any society capable of interstellar colonization and producing hedonium would be perfectly capable of producing meat without animals, or powering themselves and enjoying sensory pleasures without either. Earlier availability would be relevant for the shape of later periods only insofar as the later availability doesn’t substitute. The model does not appear to adequately account for this (although it has some relevant parameters).
We could shoehorn this consideration into the ‘memetically relevant humans’ parameter under Veg Advocacy, which currently has a 10th percentile of 1 billion and 90th percentile of 2 billion. If we increase the 90th percentile there to 200 billion to reflect future generations having more of the control over the beings they create, that alone switches the recommendation of your model away from GFI.
[Disclosure: I work at the Future of Humanity Institute, and consult for the Open Philanthropy Project. I have previously worked at the Machine Intelligence Research Institute (and my wife is on its board), and volunteered and consulted for the Center for Effective Altruism, particularly 80,000 Hours. I am writing only for myself.]
Also, it would be helpful if you said more about how you think I should do things. Should I not use a Bayesian prior at all? Should I use a wider prior or a different distribution? Should I model interventions in a different way? How do you think I could do better?
Right now all I know is that any approach has lots of problems, and my current approach seems the least problematic. If you think something else would be better, please say what it is and why you prefer it.
Compared to this, I would use something that looks more like standard cost-effectiveness analysis. Rather than use the Doomsday Prior and variance approach to assess robustness (which is ultra-sensitive to errors about variance for ranking options, at least comparably to errors about EV in cost-effectiveness analysis) my recommendationsI would include the following:
Do multiple cost-effectiveness analyses using different approaches, methodologies, and people (overall and of local variables); seek robustness in the analysis by actually doing it differently
Use empirical and intuitive priors about more local variables (e.g. make use of data about the success rates of past scientific research and startups in forming your prior about the success of meat substitute research, without hugely penalizing the success rate because the topic has more utilitarian value by your lights) rather than global
Assess the size of the future separately from our ability to prevent short-run catastrophic risks or produce short-run value changes (like adding vegans)
Focus on relative impact of different actions, rather than absolute impact in QALYs (the size of the future mostly cancels out, and even a future as large as Earth’s past is huge relative to the present, hundreds of millions of years; actions like cash transfers also have long-run effects, although a lot less than actions optimized for long-run effects)
Vigorously seek out criticism, missing pieces, and improvements for the models and their components
Part of what I’m getting at is a desire to see you defend the wacky claims implicit in your model posteriors. You present arguments for how the initial estimates could make sense, but not for how the posteriors could make sense. And as I discuss above, it’s hard to make them make sense, and that counts against the outputs.
So I’d like to see some account of why your best picture of the world is in so much tension with your prior, and how we could have an understanding of the world that is consistent with your posterior.
Thanks, this is exactly the sort of thing I was looking for.
Slightly unrelated but:
The wacky claims you’ve talked about here relate to far-future posteriors. Do you also mean the direct effect posteriors imply wacky claims? I know you’ve said before you think the way I set a prior is arbitrary, is there anything else?
I generally agree with this. My model has lots of problems and certainly shouldn’t be taken 100% seriously, Perhaps I should have said more about this earlier but I wasn’t really thinking about it until you brought it up, so it’s probably good that you did.
It sounds like you’re raising 2-3 main issues:
Posteriors have weird behavior. 1a. Increasing the upper bound on a confidence interval reduces the posterior. 1b. Far future effects are penalized in a way that has some counterintuitive implications.
The inputs for GFI’s far future effects look wrong.
For (2), I don’t think they’re as wrong as you think they are, but nonetheless I don’t really rely on these and I wouldn’t suggest that people do. I could get more into this but it doesn’t seem that important to me.
For (1a), widening a confidence interval generally makes a posterior get worse. This sort of makes intuitive sense because if your cost-effectiveness estimate has a wider distribution, it’s more likely you’re making a mistake and your expected value estimate is too high. Maybe the current implementation of my model exaggerates this too much; I’m not really sure what the correct behavior here should be.
For (1b), this is a big problem and I’m not sure what to do about it. The obvious other thing to do is to essentially take claims about the far future literally—if calculations suggest that things you do now affect 10^55 QALYs in the far future, then that’s totally reasonable. (Obviously these aren’t the only things you can do, but you can move in the direction of “take calculations more seriously” or “take calculations less seriously”, and there are different ways to do that.) I don’t believe we should take those sorts of claims entirely seriously. People have a history of over-estimating cost-effectiveness of interventions, and we should expect this to be even more overstated for really big far future effects; so I believe there’s a good case for heavily discounting these estimates. I don’t know how to avoid both the “affecting anything is impossible” problem and the “take everything too seriously” problem, and I’d be happy to hear suggestions about how to fix it. My current strategy is to combine quantitative models with qualitative reasoning, but I’d like to have better quantitative models that I could rely on more heavily.
So, how should this model be used? I’m generally a fan of cost-effectiveness calculations. I calculated expected values for how MFA, GFI, and ACE directly reduce suffering, and my prior adjusts these based on robustness of evidence. The way I calculate posteriors has problems, but I believe the direct-effect posteriors better reflect reality than the raw estimates (I have no idea about the far-future posteriors). If you don’t like the way I use priors, you can just look at the raw estimates. I wouldn’t take these estimates too seriously since my calculations are pretty rough, but it at least gives some idea of how they compare in quantitative terms, and I believe it doesn’t make much sense to decide where to donate without doing something like this.
I think you would benefit a lot from separating out ‘can we make this change in the world, e.g. preventing an asteroid from hitting the Earth, answering this scientific question, convincing one person to be vegan’ from the size of the future. A big future (as big as the past, the fossil records shows billions of years of life) doesn’t reach backwards in time to warp all of these ordinary empirical questions about life today.
It doesn’t even have much of an efficient market effect, because essentially no actors are allocating resources in a way that depends on whether the future is 1,000,000x as important as the past 100 years or 10^50. Indeed almost no one is allocating resources as though the next 100 million years are 10x as important as the past 100 years.
The things that come closest are generic Doomsday Arguments, and the Charity Doomsday Prior can be seen as falling into this class. The best cashed out relies on the simulation argument:
Let the size of the universe be X
Our apparent position seems to let us affect a big future, with value that grows with X
The larger X is, the greater the expected number of simulations of people in seemingly pivotal positions like ours
So X appears on both sides of the equation, cancels out, and the value of future relative to past is based on the ratio of the size and density of value in simulations vs basement reality
You then get figures like 10^12 or 10^20, not 10^50 or infinity, for the value of local helping (multiplied by the number of simulations) vs the future
Your Charity Doomsday Prior might force you into a view something like that, and I think it bounds the differences in expected value between actions. Now this isn’t satisfying for you because you (unlike Holden in the thread where you got the idea for this prior) don’t want to assess charities in their relative effects, but in absolute effects in QALY-like units. But it does mean that you don’t have to worry about 10^55x differences.
In any case, in your model the distortions get worse as you move further out from the prior, and much of that is being driven by the estimates of the size of the future (making the other problems worse). You could try reformulating with an empirically-informed prior over success at political campaigns, scientific discovery, and whatnot, and then separately assess the value of the different goals according to your total utilitarianish perspective. Then have a separate valuation of the future.
I’m not sure I fully understand what you are relying on. I think your model goes awry on estimating the relative long-run fruit of the things you consider, and in particular on the bottom-line conclusion/ranking. If I wanted to climb the disagreement hierarchy with you, and engage with and accept or reject your key point, how would you suggest I do it?
Hi Carl,
Are the adjusted lower numbers based on calculations such as these?