Thanks a bunch for this report! I haven’t had the time to read it very carefully, but I’ve already really enjoyed it and am curating the post.
I’m also sharing some questions I have, my highlights, and my rough understanding of the basic model setup (pulled from my notes as I was skimming the report).
A couple of questions / follow-up discussions
I’m curious about why you chose to focus specifically on biological risks.
I expect that it’s usually good to narrow the scope of reports like this and you do outline the scope at the beginning,[1] but I’d be interested in hearing more about why you didn’t, for instance, focus on risks from AI. (I guess
(For context, in the XPT, risks from AI are believed to be higher than risks from engineered pathogens.)
I’d be interested in a follow-up discussion[2] on this: “My preferred policy stance … is to separately and in parallel pursue reforms that accelerate science and reforms that reduce risks from new technologies, without worrying too much about their interaction (with some likely rare exceptions).”
In particular, I mostly like the proposal, but have some worries. It makes sense to me that it’s generally good to pursue different goals separately[3] But sometimes[4] it does turn out that at-first-seemingly-unrelated side-considerations (predictably) swamp the original consideration. Your report is an update against this being the case for scientific progress and biological risk, but I haven’t tried to estimate what your models (and XPT forecasts I suppose) would predict for AI risk.
I also have the intuition that there’s some kind of ~collaborativeness consideration like: if you have goals A and B (and, unlike in this report, you don’t have an agreed-on exchange rate between them), then you should decide to pursue A and B separately only if the difference in outcomes from A’s perspective between B-optimized actions and A-and-B-compromise actions is comparable to or smaller than the outcomes of A-optimized actions.
To use an example: if I want to have a career that involves traveling while also minimizing my contribution to CO2 levels or something, then I should probably just fly and donate to clean tech or verified carbon offsets or something, because even from the POV of CO2 levels that’s better. But if it turns out that flying does more damage than the change I can make by actively improving CO2 levels, then maybe I should find a way to travel less or err on the side of trains or the like. (I think you can take this too far, but maybe there’s a reasonable ground here.)
More specifically I’m wondering if we can estimate the impact of AI/biosafety interventions, compared to possible harms.
I’m somewhat uncertain about why you model the (biological) time of perils (ToP) the way you do (same with the impact of “pausing science” on ToP).
I was initially most confused about why only the start date of the ToP moves back because of a science-pause, and assumed it was either because you assumed that ToP was indefinite or because it would end for a reason not very affected by the rate of scientific progress. Based on the discussion in one of the sections, I think that’s not quite right? (You also explore the possibility of ToP contracting due to safety-boosting scientific advances, which also seems to contradict my earlier interpretation.)
This also led to me wondering what would happen if the risk grew over the course of ToP a growing risk (d), as opposed to going from 0 to d (e.g. there’s some chance of a new unlocked dangerous biotechnology at any point once ToP starts) — and how that would affect the results? (Maybe you do something like this somewhere!)
Some things that were highlights to me
In the comparison of superforecaster and expert answers in XPT, the “Correlated pessimism” consideration was particularly interesting and more compelling than I expected it to be before I read it!
″...general pessimism and optimism among groups in ways that imply biases. [....] We also see a high degree of correlation in beliefs about catastrophic risk even for categories that seem likely to be uncorrelated. For example, [for the probability that non-anthropogenic causes (e.g. asteroids) cause catastrophes] the third of respondents most concerned about AI risk [...] foresaw a 0.14% chance of such a catastrophe by 2100. The third of respondent least concerned foresaw a 0.01% chance, more than an order of magnitude less. [...]” Then there’s discussion of selection effects — pessimists about catastrophic risks might become experts in the field — and an argument that overly optimistic superforecasters would get corrective feedback that too-pessimistic risk experts might lack.
Also in the XPT discussions (S 4.1), I liked the extrapolation for the current/future ~engineered pandemic peril rates and for a date for the onset of the time of perils.[5]
I thought the “leveling the playing field” consideration was useful and something I probably haven’t considered enough, particularly in bio: ”… the faster is scientific progress, the greater is the set of defensive capabilities, relative to offensive ones. Conversely, a slowdown in the rate of scientific progress (which is arguably underway!) reduces safety by “leveling the playing field” between large and small organizations.” (Related: multipolar AI scenarios)
Factoids/estimates I thought were interesting independent of the rest of the report:
56% of life expectancy increases are attributed to science effects in this paper (see S4.8), which, together with average increases in life expectancy means that a year of science increases our lifespans by around 0.261% (i.e. multiply by ~1.0026 every year). (For the impacts of science on utility via increased incomes, the paper estimates a per-year increase of 0.125%.)
You conclude that: “Roughly half the value comes from technologies that make these people just a tiny bit richer, in every year, for a long time to come. The other half comes from technologies that make people just a tiny bit healthier, again in every year, for a long time to come.” I don’t know if I would have expected the model to predict such an even split.
“Omberg and Tabarrok (2022) [examine] the efficacy of different methods of preparing for a biocatastrophe; specifically, the covid-19 pandemic. The study takes as its starting point the Global Health Security Index, which was completed in 2019, shortly before the onset of the pandemic. This index was designed by a large panel of experts to rate countries on their capacity to prevent and mitigate epidemics and pandemics. Omberg and Tabarrok examine how the index, and various sub-indices of it, are or are not correlated with various metrics of success in the covid-19 pandemic, mostly excess deaths per capita. The main conclusion is that almost none of the indices were correlated with covid-19 responses, whether those metrics related to disease prevention, detection, response, or the capacity of the health system.”
My rough understanding of the basic setup of the main(?) model
(Please let me know if you see an error! I didn’t check this carefully.)
Broad notes:
“Time of (biological) perils” is a distinct period, which starts when the annual probability a biocatastrophe jumps.
The primary benefits of science that are considered are income and health boosts (more income means more utility per person per year, better health means lower mortality rates).
I think the quality of science that happens or doesn’t happen at different times is assumed to be ~always average.
The “more sophisticated model” described in S3.1 compares total utility (across everyone on earth, from now until eternity) under two scenarios: “status quo” and a year-long science “pause.”
Future utility is discounted relative to present/near-future utility,mostly for epistemic reasons (the further out we’re looking, the more uncertainty we have about what the outcomes will be). This is modeled by assuming a constant annual probability that the world totally stops being predictable (it enters a “new epistemic regime”); utility past that point does not depend on whether we accelerate science or not and can be set aside for the purpose of this comparison— see S3.2 and 4.2[6]
Here’s how outcomes differ in the two scenarios, given that context:
In the status quo scenario, current trends continue (this is the baseline).
In the “pause science” scenario, science is “turned off” for a year, which has a delayed impact by:
Delaying the start of the time of perils by a year (without affecting the end-date of the time of perils, if there is one)
Slowing growth for the duration of a year starting time T
I.e. after the pause things continue normally for a while, then slow down for a year, and then go back up after the year has passed. In the model, pausing science slows the decline in mortality for a year but doesn’t affect birth rates. Given some assumptions, this means that pausing science pushes the decline in mortality permanently behind where it should be, so population growth rates slow (forever, with the discount rate p as a caveat) after the effects kick in.
Note that T is also taken to be the time at which the time of perils would start by default — there’s a section discussing what happens if the time of perils starts earlier or later than T that argues that we should expect pausing science to look worse in either scenario.
S 2.2. “Synthetic biology is not the only technology with the potential to destroy humanity—a short list could also include nuclear weapons, nanotechnology, and geoengineering. But synthetic biology appears to be the most salient at the moment. [...]”
Most actions targeted at some goal A affect a separate goal B way less than an action that was taken because it targeted goal B would have affected B. (I think this effect is probably stronger if you filter by “top 10% most effective actions targeted at these goals, assuming we believe that there are huge differences in impact.) If you want to spend 100 resource units for goals A and B you should probably just split the resources and target the two things separately instead of trying to find things that look fine for both A and B.
(I think the “barbell strategy” is a related concept, although I haven’t read much about it.)
Seems like around the year 2038, the superforecasters expect a doubling of annual mortality rates from engineered pandemics from 0.0021% to 0.0041% — around 1 COVID every 48 years — and a shift from ~0% to ~0.0002%/year extinction risk. The increases are assumed to persist (although there were only forecasts until 2100?).
4.2 is a cool collection of different approaches to identifying a discount rate. Ultimately the author assumes p=0.98, which is on the slightly lower end and which he flags will put more weight on near-term events.
I think p can also be understood to incorporate a kind of potential “washout” aspect of scientific progress today (if we don’t discover what we would have in 2024, maybe we still mostly catch up in the next few years), although I haven’t thought carefully about it.
Thanks a bunch for this report! I haven’t had the time to read it very carefully, but I’ve already really enjoyed it and am curating the post.
I’m also sharing some questions I have, my highlights, and my rough understanding of the basic model setup (pulled from my notes as I was skimming the report).
A couple of questions / follow-up discussions
I’m curious about why you chose to focus specifically on biological risks.
I expect that it’s usually good to narrow the scope of reports like this and you do outline the scope at the beginning,[1] but I’d be interested in hearing more about why you didn’t, for instance, focus on risks from AI. (I guess
(For context, in the XPT, risks from AI are believed to be higher than risks from engineered pathogens.)
I’d be interested in a follow-up discussion[2] on this: “My preferred policy stance … is to separately and in parallel pursue reforms that accelerate science and reforms that reduce risks from new technologies, without worrying too much about their interaction (with some likely rare exceptions).”
In particular, I mostly like the proposal, but have some worries. It makes sense to me that it’s generally good to pursue different goals separately[3] But sometimes[4] it does turn out that at-first-seemingly-unrelated side-considerations (predictably) swamp the original consideration. Your report is an update against this being the case for scientific progress and biological risk, but I haven’t tried to estimate what your models (and XPT forecasts I suppose) would predict for AI risk.
I also have the intuition that there’s some kind of ~collaborativeness consideration like: if you have goals A and B (and, unlike in this report, you don’t have an agreed-on exchange rate between them), then you should decide to pursue A and B separately only if the difference in outcomes from A’s perspective between B-optimized actions and A-and-B-compromise actions is comparable to or smaller than the outcomes of A-optimized actions.
To use an example: if I want to have a career that involves traveling while also minimizing my contribution to CO2 levels or something, then I should probably just fly and donate to clean tech or verified carbon offsets or something, because even from the POV of CO2 levels that’s better. But if it turns out that flying does more damage than the change I can make by actively improving CO2 levels, then maybe I should find a way to travel less or err on the side of trains or the like. (I think you can take this too far, but maybe there’s a reasonable ground here.)
More specifically I’m wondering if we can estimate the impact of AI/biosafety interventions, compared to possible harms.
I’m somewhat uncertain about why you model the (biological) time of perils (ToP) the way you do (same with the impact of “pausing science” on ToP).
I was initially most confused about why only the start date of the ToP moves back because of a science-pause, and assumed it was either because you assumed that ToP was indefinite or because it would end for a reason not very affected by the rate of scientific progress. Based on the discussion in one of the sections, I think that’s not quite right? (You also explore the possibility of ToP contracting due to safety-boosting scientific advances, which also seems to contradict my earlier interpretation.)
This also led to me wondering what would happen if the risk grew over the course of ToP a growing risk (d), as opposed to going from 0 to d (e.g. there’s some chance of a new unlocked dangerous biotechnology at any point once ToP starts) — and how that would affect the results? (Maybe you do something like this somewhere!)
Some things that were highlights to me
In the comparison of superforecaster and expert answers in XPT, the “Correlated pessimism” consideration was particularly interesting and more compelling than I expected it to be before I read it!
″...general pessimism and optimism among groups in ways that imply biases. [....] We also see a high degree of correlation in beliefs about catastrophic risk even for categories that seem likely to be uncorrelated. For example, [for the probability that non-anthropogenic causes (e.g. asteroids) cause catastrophes] the third of respondents most concerned about AI risk [...] foresaw a 0.14% chance of such a catastrophe by 2100. The third of respondent least concerned foresaw a 0.01% chance, more than an order of magnitude less. [...]” Then there’s discussion of selection effects — pessimists about catastrophic risks might become experts in the field — and an argument that overly optimistic superforecasters would get corrective feedback that too-pessimistic risk experts might lack.
Also in the XPT discussions (S 4.1), I liked the extrapolation for the current/future ~engineered pandemic peril rates and for a date for the onset of the time of perils.[5]
I thought the “leveling the playing field” consideration was useful and something I probably haven’t considered enough, particularly in bio: ”… the faster is scientific progress, the greater is the set of defensive capabilities, relative to offensive ones. Conversely, a slowdown in the rate of scientific progress (which is arguably underway!) reduces safety by “leveling the playing field” between large and small organizations.” (Related: multipolar AI scenarios)
Factoids/estimates I thought were interesting independent of the rest of the report:
56% of life expectancy increases are attributed to science effects in this paper (see S4.8), which, together with average increases in life expectancy means that a year of science increases our lifespans by around 0.261% (i.e. multiply by ~1.0026 every year). (For the impacts of science on utility via increased incomes, the paper estimates a per-year increase of 0.125%.)
“In 2020, for example, roughly 1 in 20-25 papers published was related to Covid-19 in some way (from essentially none in 2019).”
You conclude that: “Roughly half the value comes from technologies that make these people just a tiny bit richer, in every year, for a long time to come. The other half comes from technologies that make people just a tiny bit healthier, again in every year, for a long time to come.” I don’t know if I would have expected the model to predict such an even split.
“Omberg and Tabarrok (2022) [examine] the efficacy of different methods of preparing for a biocatastrophe; specifically, the covid-19 pandemic. The study takes as its starting point the Global Health Security Index, which was completed in 2019, shortly before the onset of the pandemic. This index was designed by a large panel of experts to rate countries on their capacity to prevent and mitigate epidemics and pandemics. Omberg and Tabarrok examine how the index, and various sub-indices of it, are or are not correlated with various metrics of success in the covid-19 pandemic, mostly excess deaths per capita. The main conclusion is that almost none of the indices were correlated with covid-19 responses, whether those metrics related to disease prevention, detection, response, or the capacity of the health system.”
My rough understanding of the basic setup of the main(?) model
(Please let me know if you see an error! I didn’t check this carefully.)
Broad notes:
“Time of (biological) perils” is a distinct period, which starts when the annual probability a biocatastrophe jumps.
The primary benefits of science that are considered are income and health boosts (more income means more utility per person per year, better health means lower mortality rates).
I think the quality of science that happens or doesn’t happen at different times is assumed to be ~always average.
The “more sophisticated model” described in S3.1 compares total utility (across everyone on earth, from now until eternity) under two scenarios: “status quo” and a year-long science “pause.”
Future utility is discounted relative to present/near-future utility, mostly for epistemic reasons (the further out we’re looking, the more uncertainty we have about what the outcomes will be). This is modeled by assuming a constant annual probability that the world totally stops being predictable (it enters a “new epistemic regime”); utility past that point does not depend on whether we accelerate science or not and can be set aside for the purpose of this comparison— see S3.2 and 4.2[6]
Here’s how outcomes differ in the two scenarios, given that context:
In the status quo scenario, current trends continue (this is the baseline).
In the “pause science” scenario, science is “turned off” for a year, which has a delayed impact by:
Delaying the start of the time of perils by a year (without affecting the end-date of the time of perils, if there is one)
Slowing growth for the duration of a year starting time T
I.e. after the pause things continue normally for a while, then slow down for a year, and then go back up after the year has passed. In the model, pausing science slows the decline in mortality for a year but doesn’t affect birth rates. Given some assumptions, this means that pausing science pushes the decline in mortality permanently behind where it should be, so population growth rates slow (forever, with the discount rate p as a caveat) after the effects kick in.
Note that T is also taken to be the time at which the time of perils would start by default — there’s a section discussing what happens if the time of perils starts earlier or later than T that argues that we should expect pausing science to look worse in either scenario.
S 2.2. “Synthetic biology is not the only technology with the potential to destroy humanity—a short list could also include nuclear weapons, nanotechnology, and geoengineering. But synthetic biology appears to be the most salient at the moment. [...]”
Although I might not be able to respond much/fast in the near future
Most actions targeted at some goal A affect a separate goal B way less than an action that was taken because it targeted goal B would have affected B. (I think this effect is probably stronger if you filter by “top 10% most effective actions targeted at these goals, assuming we believe that there are huge differences in impact.) If you want to spend 100 resource units for goals A and B you should probably just split the resources and target the two things separately instead of trying to find things that look fine for both A and B.
(I think the “barbell strategy” is a related concept, although I haven’t read much about it.)
(for some reason the thing that comes to mind is this SSC post about marijuana legalization from 2014 — I haven’t read it in forever but remember it striking a chord)
Seems like around the year 2038, the superforecasters expect a doubling of annual mortality rates from engineered pandemics from 0.0021% to 0.0041% — around 1 COVID every 48 years — and a shift from ~0% to ~0.0002%/year extinction risk. The increases are assumed to persist (although there were only forecasts until 2100?).
4.2 is a cool collection of different approaches to identifying a discount rate. Ultimately the author assumes p=0.98, which is on the slightly lower end and which he flags will put more weight on near-term events.
I think p can also be understood to incorporate a kind of potential “washout” aspect of scientific progress today (if we don’t discover what we would have in 2024, maybe we still mostly catch up in the next few years), although I haven’t thought carefully about it.