This is absolutely fantastic work! One of the Forum posts of the year so far! A really good step towards getting robust estimates of xRisk work, would be great to see other work following up on this research agenda (both OAT and your own).[1]
Some thoughts:
If I understand correctly, the value gained from action M is always the same if the fractional reduction in xRisk is the same, ceteris paribus? That still means that there seems to be a tradeoff between assuming a high rate of xRisk and believing in astronomical value, assuming that the cost of an intervention is linear in size relative to the amount decrease (i.e. decrease xrisk from 50% to 40% in a given t is 10 times as hard than reducing from 50% to 49% - would be interesting to see this worked out robustly) I think thatās a robust finding which seems to be unintuitive both for EAs and EA critics
If you had to (and think itās appropriate to do so), what do you think the default assumptions of xRisk mitigation efforts in EA currently believe to be true? Iād guess itād be ātime of perilsā and maybe quadratic or cubic growth? But as you point out, the difference between quadratic/ācubic is immense, and could easily flip whether it would be the best marginal option for altruistic funding.
Iād be interested to see what BOTEC EVs look like under this model and some assumptions. Thorstad has done something like this, but itād be good to get a more robust sense of what parameter configurations would be needed to make xRisk reduction become competitive with top-rated GiveWell Charities
Your finding on convergence is I think very important, not least because it undercuts one of the most common criticisms of xRisk/ālongtermist work āthis assigns infinite value to future people which justifies arbitrary moral harm to current peopleā which just turns out to not hold under your models here. Not going to hold my breath for this critics to update though.
Great work sharing the notebook <3 really love the transparency, I think something like this should become more standard (not just in EA, but everywhere) so wanted to give you big props for exposing your model/ācode/āparameters for anyone to check.
So yeah, great work, love it! Would love to see and support more work along these lines.
Thank you for all the comments JWS, I found your excitement contagious.
Some thoughts on your thoughts:
I couldnāt agree more that thereād be a lot of value from laying out parameter configurations. We have some more work coming out as part of this sequence that aims to help fill this gap!
I think itād be great to see some survey data on what the commonly assumed risk patterns and valued trajectories are in the EA community. Iāve made a push from my little corner to hopefully get some data on common views. Whichever they are, youāre right to point out the immense differences in what could they imply.
Iām really happy that you found the notebook useful. Iāll make sure to update the GitHub with any new features and code discussions.
If you had to (and think itās appropriate to do so), what do you think the default assumptions of xRisk mitigation efforts in EA currently believe to be true?
My guess would be Time of Perils, but with a risk decaying exponentially to 0 after it (instead of a low constant risk).
Your finding on convergence is I think very important, not least because it undercuts one of the most common criticisms of xRisk/ālongtermist work āthis assigns infinite value to future people which justifies arbitrary moral harm to current peopleā which just turns out to not hold under your models here.
Something similar to that critique (replacing infinite by astronomically large, and arbitrary by significant) could still hold if the risk decays to 0.
Itās true there are other scenarios that would recover infinite value. And the proof fails, as mentioned in the convergence section, with changes like rā=0, or when the logistic cap cāā and we end up in the exponential case.
All that said, it is plausible that the universe has a finite length after all, which would provide that finite upper bound. Heat death, proton decay or even just the amount of accessible matter could provide physical limits. Itād be great to see more discussions on this informed by updated astrophysical theories.
Personally, I do not think allowing the risk to decay to 0 is problematic. For a sufficiently long timeframe, there will be evidential symmetry between the risk profiles of any 2 actions (e.g. maybe everything that is bound together will dissolve), so the expected value of mitigation will eventually reach 0. As a result, the expected cumulative value of mitigation always converges.
This is absolutely fantastic work! One of the Forum posts of the year so far! A really good step towards getting robust estimates of xRisk work, would be great to see other work following up on this research agenda (both OAT and your own).[1]
Some thoughts:
If I understand correctly, the value gained from action M is always the same if the fractional reduction in xRisk is the same, ceteris paribus? That still means that there seems to be a tradeoff between assuming a high rate of xRisk and believing in astronomical value, assuming that the cost of an intervention is linear in size relative to the amount decrease (i.e. decrease xrisk from 50% to 40% in a given t is 10 times as hard than reducing from 50% to 49% - would be interesting to see this worked out robustly) I think thatās a robust finding which seems to be unintuitive both for EAs and EA critics
If you had to (and think itās appropriate to do so), what do you think the default assumptions of xRisk mitigation efforts in EA currently believe to be true? Iād guess itād be ātime of perilsā and maybe quadratic or cubic growth? But as you point out, the difference between quadratic/ācubic is immense, and could easily flip whether it would be the best marginal option for altruistic funding.
Iād be interested to see what BOTEC EVs look like under this model and some assumptions. Thorstad has done something like this, but itād be good to get a more robust sense of what parameter configurations would be needed to make xRisk reduction become competitive with top-rated GiveWell Charities
Your finding on convergence is I think very important, not least because it undercuts one of the most common criticisms of xRisk/ālongtermist work āthis assigns infinite value to future people which justifies arbitrary moral harm to current peopleā which just turns out to not hold under your models here. Not going to hold my breath for this critics to update though.
Great work sharing the notebook <3 really love the transparency, I think something like this should become more standard (not just in EA, but everywhere) so wanted to give you big props for exposing your model/ācode/āparameters for anyone to check.
So yeah, great work, love it! Would love to see and support more work along these lines.
The new acronym could be ATOM perhaps? ;)
Thank you for all the comments JWS, I found your excitement contagious.
Some thoughts on your thoughts:
I couldnāt agree more that thereād be a lot of value from laying out parameter configurations. We have some more work coming out as part of this sequence that aims to help fill this gap!
I think itād be great to see some survey data on what the commonly assumed risk patterns and valued trajectories are in the EA community. Iāve made a push from my little corner to hopefully get some data on common views. Whichever they are, youāre right to point out the immense differences in what could they imply.
Iām really happy that you found the notebook useful. Iāll make sure to update the GitHub with any new features and code discussions.
Nice comments!
My guess would be Time of Perils, but with a risk decaying exponentially to 0 after it (instead of a low constant risk).
Something similar to that critique (replacing infinite by astronomically large, and arbitrary by significant) could still hold if the risk decays to 0.
Itās true there are other scenarios that would recover infinite value. And the proof fails, as mentioned in the convergence section, with changes like rā=0, or when the logistic cap cāā and we end up in the exponential case.
All that said, it is plausible that the universe has a finite length after all, which would provide that finite upper bound. Heat death, proton decay or even just the amount of accessible matter could provide physical limits. Itād be great to see more discussions on this informed by updated astrophysical theories.
Thanks for following up!
Personally, I do not think allowing the risk to decay to 0 is problematic. For a sufficiently long timeframe, there will be evidential symmetry between the risk profiles of any 2 actions (e.g. maybe everything that is bound together will dissolve), so the expected value of mitigation will eventually reach 0. As a result, the expected cumulative value of mitigation always converges.