[edit: a day after posting, I think this perhaps reads more combative that I intended? It was meant to be more âcrisis of faith, looking for reassurance if it existsâ than âdunk on those crazy longtermistsâ. Iâll leave the quick take as-is, but maybe clarification of my intentions might be useful to others]
Warning! Hot Take! đ„đ„đ„ (Also v rambly and not rigorous)
A creeping thought has entered my head recently that I havenât been able to get rid ofâŠ
Is most/âall longtermist spending unjustified?
The EA move toward AI Safety and Longtermism is often based on EV calculations that show the long term future is overwhelmingly valuable, and thus is the intervention that is most cost-effective.
However, more in-depth looks at the EV of x-risk prevention (1, 2) cast significant doubt on those EV calculations, which might make longtermist interventions much less cost-effective than the most effective âneartermistâ ones.
But my doubts get worse...
GiveWell estimates around $5k to save a life. So I went looking for some longtermist calculations, and I really couldnât fund any robust ones![1] Can anyone point me in some robust calculations for longtermist funds/âorganisations where they go âyep, under our assumptions and data, our interventions are at least competitive with top Global Health charitiesâ?
Because it seems to me like that hasnât been done. But if weâre being EA, people with high and intractable p(doom) from AI shouldnât work on AI, they should probably EtG for Global Health instead (if weâre going to be full maximise EV about everything). Like, if weâre taking EA seriously, shouldnât MIRI shut down all AI operations and become a Global Health org? Wouldnât that be a strong +EV move given their pessimistic assessments of reducing xRisk and their knowledge of +EV global health interventions?
But it gets worse...
Suppose that we go, âok, letâs take EV estimates seriously but not literallyâ. In which case fine, but that undermines the whole âlongtermist interventions overwhelmingly dominate EVâ move that many people feel has happened to EA causes over recent years. I donât think that move grounds longtermism, instead I think it grounds a more pluralistic approach to all of EA that values a large variety of approaches to doing good, but acknowledges the uncertainty and limits on each, and even actively prevents the movement from being focused on any one area. That still seems incompatible with the monetary and social rise of longtermism and AI Safety.
tl;dr: Have any longtermist orgs done robust and defensible EV caculations for their interventions? If not, isnât that⊠overwhelmingly sus?
I found some posts arguing that under set of assumptions X then longtermist interventions would be more effective, often by reducing the overall xrisk rate by 1% or something like that. But very little if no work had been done to actually show that was a likely result of the interventions
which might make longtermist interventions much less cost-effective than the most effective âneartermistâ ones.
Why do you think this?
For some very rough maths (appologies in advance for any errors), even Thorstadâs paper (with a 2 century long time of perils, a 0.1% post-peril risk rate, no economic/âpopulation growth, no moral progress, people live for 100 years) suggests that reducing p(doom) by 2% is worth as much as saving 16x8billion livesâi.e. each microdoom is worth 6.4million lives. I think we can buy microdooms more cheaply than $5,000*6.4million = $32billion each.
(I canât actually find those calculations in Thorstadâs paper, could you point them out to me? afaik he mostly looks at the value of fractional reduction in x-risk, while microdooms are an absolute reduction if I understand correctly? happy to be corrected or shown in the right direction!)
My concerns here are twofold:
1 - epistemological: Letâs say those numbers are correct from the Thorstad paper, that a microdoom has to cost â $32bn to be GiveWell cost-effective. The question is, how would we know this. In his recent post Paul Cristiano thinks that RSPs could lead to a â10x reductionâ in AI risk. How does he know this? Is this just a risk reduction this century? This decade? Is it a permanent reduction?
Itâs one thing to argue that under set of conditions X work on x-risk reduction is cost-effective as youâve done here. But Iâm more interested in the question of whether conditions X hold, because thatâs where the rubber hits the road. If those conditions donât hold, then thatâs why longtermism might not ground x-risk work.[1]
Thereâs also the question of persistence. I think the Thorstad model either assumes the persistence of x-risk reduction, or the persistence of a low-risk post-peril rate. But how could we possibly know that such considerations hold? It seems to me that cluelessness just obliterates grounding strong longtermist interventions here.
2 - practical: Letâs say that longtermist organisations/âfunds have actually answered criticisms in 1, and under their own lights have made these calculations. I want to see those calculations! How have they calculated the post-risk peril rate? Where is the support for their views on population growth? Do they assume that we colonise the galaxy by default or not? How do they get feedback that overcomes cluelessness objections?
And critically, I want to see why theyâve argued or assumed these things. afaict I canât find these calculations.
Iâm not being snarky here, I really want to be wrong on this and be shown robust calculations on longtermist work, but I havenât found any. I think the risk is that longtermist EA looks like any other charity/âmovement that funds âwhat they kinda they think is goodâ rather than what we actually know does good, which was meant to be the whole point of EA in the first place.
It could still be grounded for other reasons though, for example if x-risk reduction is highly correlated with GCR reduction, and the latter is of high value
(I canât actually find those calculations in Thorstadâs paper, could you point them out to me? afaik he mostly looks at the value of fractional reduction in x-risk, while microdooms are an absolute reduction if I understand correctly? happy to be corrected or shown in the right direction!)
He assumes 20% risk and a 10% relative risk reduction, which I translate into 2% absolute risk of doom, and then see the table on p12.
Isnât the move here something like, âIf doom soon, then all pre-doom value nets to zeroâ?
Which tbh Iâm not sure is wrong. If I expect doom tomorrow, all efforts today should be to reduce it; one nightâs sleep not being bitten by mosquitoes doesnât matter. Stretching this outward in time, doesnât change the calculus much for a while, maybe about a lifetime or a few lifetimes or so. And a huge chunk of xrisk is concentrated in this century.
The x-risk models actually support the opposite conclusion though. They are generally focused on the balance of two values, v and râwhere v the value of a time period and r is the risk of extinction in that period.
If r is sufficiently high, then it operates as a de facto discount rate on the future, which means that the most effective way to increase good is to increase the present v rather than reduce r. For an analogy, if a patient has an incredibly high risk of succumbing to terminal cancer, the way to increase wellbeing may be to give them morphine and palliative care rather than perscribe them risky treatments that may or may not work (and might only be temporary)
Now one could argue against this by saying âdo not get go gentle into that good night, in the face of destruction we should still do our bestâ. I have sympathy with that view, but itâs not grounded in the general âfollow the EVâ framework of EA, and it would have consequences beyond supporting longtermism
[edit: a day after posting, I think this perhaps reads more combative that I intended? It was meant to be more âcrisis of faith, looking for reassurance if it existsâ than âdunk on those crazy longtermistsâ. Iâll leave the quick take as-is, but maybe clarification of my intentions might be useful to others]
Warning! Hot Take! đ„đ„đ„ (Also v rambly and not rigorous)
A creeping thought has entered my head recently that I havenât been able to get rid ofâŠ
The EA move toward AI Safety and Longtermism is often based on EV calculations that show the long term future is overwhelmingly valuable, and thus is the intervention that is most cost-effective.
However, more in-depth looks at the EV of x-risk prevention (1, 2) cast significant doubt on those EV calculations, which might make longtermist interventions much less cost-effective than the most effective âneartermistâ ones.
But my doubts get worse...
GiveWell estimates around $5k to save a life. So I went looking for some longtermist calculations, and I really couldnât fund any robust ones![1] Can anyone point me in some robust calculations for longtermist funds/âorganisations where they go âyep, under our assumptions and data, our interventions are at least competitive with top Global Health charitiesâ?
Because it seems to me like that hasnât been done. But if weâre being EA, people with high and intractable p(doom) from AI shouldnât work on AI, they should probably EtG for Global Health instead (if weâre going to be full maximise EV about everything). Like, if weâre taking EA seriously, shouldnât MIRI shut down all AI operations and become a Global Health org? Wouldnât that be a strong +EV move given their pessimistic assessments of reducing xRisk and their knowledge of +EV global health interventions?
But it gets worse...
Suppose that we go, âok, letâs take EV estimates seriously but not literallyâ. In which case fine, but that undermines the whole âlongtermist interventions overwhelmingly dominate EVâ move that many people feel has happened to EA causes over recent years. I donât think that move grounds longtermism, instead I think it grounds a more pluralistic approach to all of EA that values a large variety of approaches to doing good, but acknowledges the uncertainty and limits on each, and even actively prevents the movement from being focused on any one area. That still seems incompatible with the monetary and social rise of longtermism and AI Safety.
tl;dr: Have any longtermist orgs done robust and defensible EV caculations for their interventions? If not, isnât that⊠overwhelmingly sus?
I found some posts arguing that under set of assumptions X then longtermist interventions would be more effective, often by reducing the overall xrisk rate by 1% or something like that. But very little if no work had been done to actually show that was a likely result of the interventions
Why do you think this?
For some very rough maths (appologies in advance for any errors), even Thorstadâs paper (with a 2 century long time of perils, a 0.1% post-peril risk rate, no economic/âpopulation growth, no moral progress, people live for 100 years) suggests that reducing p(doom) by 2% is worth as much as saving 16x8billion livesâi.e. each microdoom is worth 6.4million lives. I think we can buy microdooms more cheaply than $5,000*6.4million = $32billion each.
(I canât actually find those calculations in Thorstadâs paper, could you point them out to me? afaik he mostly looks at the value of fractional reduction in x-risk, while microdooms are an absolute reduction if I understand correctly? happy to be corrected or shown in the right direction!)
My concerns here are twofold:
1 - epistemological: Letâs say those numbers are correct from the Thorstad paper, that a microdoom has to cost â $32bn to be GiveWell cost-effective. The question is, how would we know this. In his recent post Paul Cristiano thinks that RSPs could lead to a â10x reductionâ in AI risk. How does he know this? Is this just a risk reduction this century? This decade? Is it a permanent reduction?
Itâs one thing to argue that under set of conditions X work on x-risk reduction is cost-effective as youâve done here. But Iâm more interested in the question of whether conditions X hold, because thatâs where the rubber hits the road. If those conditions donât hold, then thatâs why longtermism might not ground x-risk work.[1]
Thereâs also the question of persistence. I think the Thorstad model either assumes the persistence of x-risk reduction, or the persistence of a low-risk post-peril rate. But how could we possibly know that such considerations hold? It seems to me that cluelessness just obliterates grounding strong longtermist interventions here.
2 - practical: Letâs say that longtermist organisations/âfunds have actually answered criticisms in 1, and under their own lights have made these calculations. I want to see those calculations! How have they calculated the post-risk peril rate? Where is the support for their views on population growth? Do they assume that we colonise the galaxy by default or not? How do they get feedback that overcomes cluelessness objections?
And critically, I want to see why theyâve argued or assumed these things. afaict I canât find these calculations.
Iâm not being snarky here, I really want to be wrong on this and be shown robust calculations on longtermist work, but I havenât found any. I think the risk is that longtermist EA looks like any other charity/âmovement that funds âwhat they kinda they think is goodâ rather than what we actually know does good, which was meant to be the whole point of EA in the first place.
It could still be grounded for other reasons though, for example if x-risk reduction is highly correlated with GCR reduction, and the latter is of high value
He assumes 20% risk and a 10% relative risk reduction, which I translate into 2% absolute risk of doom, and then see the table on p12.
Isnât the move here something like, âIf doom soon, then all pre-doom value nets to zeroâ?
Which tbh Iâm not sure is wrong. If I expect doom tomorrow, all efforts today should be to reduce it; one nightâs sleep not being bitten by mosquitoes doesnât matter. Stretching this outward in time, doesnât change the calculus much for a while, maybe about a lifetime or a few lifetimes or so. And a huge chunk of xrisk is concentrated in this century.
The x-risk models actually support the opposite conclusion though. They are generally focused on the balance of two values, v and râwhere v the value of a time period and r is the risk of extinction in that period.
If r is sufficiently high, then it operates as a de facto discount rate on the future, which means that the most effective way to increase good is to increase the present v rather than reduce r. For an analogy, if a patient has an incredibly high risk of succumbing to terminal cancer, the way to increase wellbeing may be to give them morphine and palliative care rather than perscribe them risky treatments that may or may not work (and might only be temporary)
Now one could argue against this by saying âdo not get go gentle into that good night, in the face of destruction we should still do our bestâ. I have sympathy with that view, but itâs not grounded in the general âfollow the EVâ framework of EA, and it would have consequences beyond supporting longtermism