Love the ambition, and I like that it could fail gracefully (since steps 1 and 2 are tractable and valuable on their own, and just require people to regain their courage to make up numbers, after learning Saulius’ important lessons). Step 1-3 are worth millions of dollars, maybe more, even if we just do em for the empirical causes.
And longtermism needs evaluation. I have made only the most basic estimate of my own impact. (One narrow unit I’ve used is the ‘basis point reduction in xrisk’.) It could also help precisify what ‘robustly’ good means (95% of mass with positive effect?).
I probably don’t need to emphasise how hard step 4 is. But you don’t emphasise it, so I will.
It’s hard, hard like solving a chunk of philosophy. In fact the crucial considerations you list seem relatively tame: they’re continuous. I take the real horror of crucial considerations to lie with the strongest ones: inverters, those that flip the sign of your evaluation (making you do the opposite action) - then another comes and flips it again—with no particular reason to think they will stop coming. This headache seems like the default situation for crazy-town (that is, strict) longtermists. And it doesn’t seem easy to analyse with your heroic method.*
(I’m not suggesting that we give up—we should whack-a-mole crucial considerations, in the hope of eventually running out of moles.)
Now instead assume victory. Here’s what seems likely on the user end:
An honest longtermist quantification will spit out something spanning 7+ orders of magnitude (with some chunk of it covering negative value land), all relatively flat.
This will make most people’s distributions look very similar, despite them presumably being based on extremely different premises and weightings.
Struggling to manage multiple apparently uninformative intervention distributions, most people will just decide based on the mean, wasting your subtle development and philosophical work.
* You could concoct a prior on how many rungs you expect the deliberation ladder to have, and act once you’ve got past most of it. But this wouldn’t help, because being just one rung off means you’re reducing value.
I actually don’t know a practical way to handle inverters. Do you slightly reduce your investment in all interventions, to hedge against causing harm? Sounds absurd.
With regards to your grim picture of victory, yeah, estimates will span multiple orders of magnitude. But at the very least, we could cut off the paths that “the market”, or “the forecasting system” knows to be dead-ends from the start. Because people have a bunch of biases that better forecasting/evaluation systems could just make apparent in excruciating detail.
For example, consider Open Philanthropy’s spinning off of Just Impact, “[a]fter hundreds of grants totaling more than $130 million over six years” because “we think the top global aid charities recommended by GiveWell (which we used to be part of and remain closely affiliated with) present an opportunity to give away large amounts of money at higher cost-effectiveness than we can achieve in many programs, including CJR, that seek to benefit citizens of wealthy countries”.
I have some pet theories about what happened here, but I’m pretty sure that any decent forecasting/evaluation system would have seen this* coming a mile ahead.
*: this = criminal justice reform being less effective than global health.
Thanks for the thoughtful comment, Gavin. Note that the inverter problem also exists in the case where you are not quantifying at all, so quantification just brings it to the forefront.
In offline conversation, you mentioned that people are bad at overriding shitty models with their initially superior intuition, which is a problem if quantified models start out as being shitty. To this the answer from my part was that yeah, at this point, I would just posit or demand grantmakers who have the skill of combining models which have some error with their own intuitions. Otherwise, the situation would be pretty hopeless.
It’s not that intuition is superior: it is broad, latent, all-things-considered (where all formal models are some-things-considered). The smell test it enables is all we have against model error. (And inverters are just a nasty kind of model error.)
I consider naming particular [AGI timeline median] years to be a cognitively harmful sort of activity; I have refrained from trying to translate my brain’s native intuitions about this into probabilities, for fear that my verbalized probabilities will be stupider than my intuitions if I try to put weight on them. What feelings I do have, I worry may be unwise to voice; AGI timelines, in my own experience, are not great for one’s mental health, and I worry that other people seem to have weaker immune systems than even my own. But I suppose I cannot but acknowledge that my outward behavior seems to reveal a distribution whose median seems to fall well before 2050.
Love the ambition, and I like that it could fail gracefully (since steps 1 and 2 are tractable and valuable on their own, and just require people to regain their courage to make up numbers, after learning Saulius’ important lessons). Step 1-3 are worth millions of dollars, maybe more, even if we just do em for the empirical causes.
And longtermism needs evaluation. I have made only the most basic estimate of my own impact. (One narrow unit I’ve used is the ‘basis point reduction in xrisk’.) It could also help precisify what ‘robustly’ good means (95% of mass with positive effect?).
I probably don’t need to emphasise how hard step 4 is. But you don’t emphasise it, so I will.
It’s hard, hard like solving a chunk of philosophy. In fact the crucial considerations you list seem relatively tame: they’re continuous. I take the real horror of crucial considerations to lie with the strongest ones: inverters, those that flip the sign of your evaluation (making you do the opposite action) - then another comes and flips it again—with no particular reason to think they will stop coming. This headache seems like the default situation for crazy-town (that is, strict) longtermists. And it doesn’t seem easy to analyse with your heroic method.*
(I’m not suggesting that we give up—we should whack-a-mole crucial considerations, in the hope of eventually running out of moles.)
Now instead assume victory. Here’s what seems likely on the user end:
An honest longtermist quantification will spit out something spanning 7+ orders of magnitude (with some chunk of it covering negative value land), all relatively flat.
This will make most people’s distributions look very similar, despite them presumably being based on extremely different premises and weightings.
Struggling to manage multiple apparently uninformative intervention distributions, most people will just decide based on the mean, wasting your subtle development and philosophical work.
* You could concoct a prior on how many rungs you expect the deliberation ladder to have, and act once you’ve got past most of it. But this wouldn’t help, because being just one rung off means you’re reducing value.
I actually don’t know a practical way to handle inverters. Do you slightly reduce your investment in all interventions, to hedge against causing harm? Sounds absurd.
With regards to your grim picture of victory, yeah, estimates will span multiple orders of magnitude. But at the very least, we could cut off the paths that “the market”, or “the forecasting system” knows to be dead-ends from the start. Because people have a bunch of biases that better forecasting/evaluation systems could just make apparent in excruciating detail.
For example, consider Open Philanthropy’s spinning off of Just Impact, “[a]fter hundreds of grants totaling more than $130 million over six years” because “we think the top global aid charities recommended by GiveWell (which we used to be part of and remain closely affiliated with) present an opportunity to give away large amounts of money at higher cost-effectiveness than we can achieve in many programs, including CJR, that seek to benefit citizens of wealthy countries”.
I have some pet theories about what happened here, but I’m pretty sure that any decent forecasting/evaluation system would have seen this* coming a mile ahead.
*: this = criminal justice reform being less effective than global health.
Thanks for the thoughtful comment, Gavin. Note that the inverter problem also exists in the case where you are not quantifying at all, so quantification just brings it to the forefront.
In offline conversation, you mentioned that people are bad at overriding shitty models with their initially superior intuition, which is a problem if quantified models start out as being shitty. To this the answer from my part was that yeah, at this point, I would just posit or demand grantmakers who have the skill of combining models which have some error with their own intuitions. Otherwise, the situation would be pretty hopeless.
It’s not that intuition is superior: it is broad, latent, all-things-considered (where all formal models are some-things-considered). The smell test it enables is all we have against model error. (And inverters are just a nasty kind of model error.)
Here’s Yudkowsky, even: