rosehadshar

• rosehadsharJul 28, 2023, 7:41 AM

  2 points

  0 ∶ 0

  in reply to: Denis ’s comment on: An overview of stan­dards in biosafety and biorisk

  Thanks for the kind words!
  
  Can you say more about how either of your two worries work for industrial chemical engineering?

  Also curious if you know anything about the legislative basis for such regulation in the US. My impression from the bio standards in the US is that it’s pretty hard to get laws passed, so if there are laws for chemical engineering it would be interesting to understand why those were plausible whereas bio ones weren’t.

An overview of stan­dards in biosafety and biosecurity

• rosehadsharJul 25, 2023, 2:29 PM

  4 points

  1 ∶ 0

  in reply to: spreadlove5683’s comment on: Sum­mary of posts on XPT fore­casts on AI risk and timelines

  Good question.

  There’s a little bit on how to think about the XPT results in relation to other forecasts here (not much). Extrapolating from there to Samotsvety in particular:

  • Reasons to favour XPT (superforecaster) forecasts:

    • Larger sample size

    • The forecasts were incentivised (via reciprocal scoring, a bit more detail here)

    • The most accurate XPT forecasters in terms of reciprocal scoring also gave the lowest probabilities on AI risk (and reciprocal scoring accuracy may correlate with actual accuracy)

  • Speculative reasons to favour Samotsvety forecasts:

    • (Guessing) They’ve spent longer on average thinking about it

    • (Guessing) They have deeper technical expertise than the XPT superforecasters

  I also haven’t looked in detail at the respective resolution criteria, but at first glance the forecasts also seem relatively hard to compare directly. (I agree with you though that the discrepancy is large enough that it suggests a large disagreement were the two groups to forecast the same question—just expect that it will be hard to work out how large.)

Sum­mary of posts on XPT fore­casts on AI risk and timelines

• rosehadsharJul 25, 2023, 8:39 AM

  17 points

  3 ∶ 0

  in reply to: kokotajlod’s comment on: Who’s right about in­puts to the biolog­i­cal an­chors model?

  Don’t apologise, think it’s a helpful point!

  I agree that the training computation requirements distribution is more subjective and matters more to the eventual output.

  I also want to note that while on your view of the compute reqs distribution, the hardware/​spending/​algorithmic progress inputs are a rounding error, this isn’t true for other views of the compute reqs distribution. E.g. for anyone who does agree with Ajeya on the compute reqs distribution, the XPT hardware/​spending/​algorithmic progress inputs shift median timelines from ~2050 to ~2090, which is quite consequential. (See here)

  For someone like me, who hasn’t thought about the compute reqs distribution properly, I basically agree that this is just an exercise (and in isolation doesn’t show me much about what my timelines should be). But for those who have thought about it, the XPT inputs could either not matter at all (e.g. for you), or matter a lot (e.g. for someone with Ajeya’s compute reqs distribution).

• rosehadsharJul 24, 2023, 3:18 PM

  2 points

  0 ∶ 0

  on: What do XPT fore­casts tell us about AI risk?

  See here for a mash up of XPT forecasts on catastrophic and extinction risk, with Shulman and Thornley’s paper on how much governments should pay to prevent catastrophes.

How much is re­duc­ing catas­trophic and ex­tinc­tion risk worth, as­sum­ing XPT fore­casts?

• rosehadsharJul 24, 2023, 2:37 PM

  3 points

  0 ∶ 0

  on: XPT fore­casts on (some) biolog­i­cal an­chors inputs

  My personal take on these forecasts here.

Who’s right about in­puts to the biolog­i­cal an­chors model?

XPT fore­casts on (some) biolog­i­cal an­chors inputs

• rosehadsharJul 24, 2023, 10:34 AM

  3 points

  0 ∶ 0

  in reply to: Benevolent_Rain’s comment on: What do XPT fore­casts tell us about AI risk?

  The follow-up project was on AI specifically, so we don’t currently have any data that would allow us to transfer directly to bio and nuclear, alas.

What do XPT fore­casts tell us about AI timelines?

• rosehadsharJul 19, 2023, 11:09 AM

  12 points

  1 ∶ 0

  in reply to: Erich_Grunewald 🔸’s comment on: What do XPT fore­casts tell us about AI risk?

  I wasn’t around when the XPT questions were being set, but I’d guess that you’re right that extinction/​catastrophe were chosen because they are easier to operationalise.

  On your question about what forecasts on existential risk would have been: I think this is a great question.

  FRI actually ran a follow-up project after the XPT to dig into the AI results. One of the things we did in this follow-up project was elicit forecasts on a broader range of outcomes, including some approximations of existential risk. I don’t think I can share the results yet, but we’re aiming to publish them in August!

What do XPT fore­casts tell us about AI risk?

• rosehadsharApr 24, 2023, 8:02 AM

  2 points

  0 ∶ 0

  in reply to: Lukas_Gloor’s comment on: Cur­rent UK gov­ern­ment lev­ers on AI development

  Thanks for this; I wasn’t tracking it and it does seem potentially relevant.

Why Si­mu­la­tor AIs want to be Ac­tive In­fer­ence AIs

Cur­rent UK gov­ern­ment lev­ers on AI development

The space of sys­tems and the space of maps

Cy­borg Pe­ri­ods: There will be mul­ti­ple AI transitions

• rosehadsharJan 20, 2023, 10:42 AM

  3 points

  0 ∶ 0

  in reply to: Sanjay’s comment on: What’s go­ing on with ‘crunch time’?

  Thanks Sanjay!
  
  I agree that both of your bullet points would be good. I also think that the second one is extremely non-trivial—more like something it would be good to have a research team working on than something I could write a section on in a blog post.

  There’s a sense in which there are already research team equivalents working on it, insofar as lots of forecasting efforts relate to p(crunch time soon). But from my vantage point it doesn’t seem like this community has clarity/​consensus around what the best indicators of crunch time soon are, or that there are careful analyses of why we should expect those to be good indicators, and that makes me expect that more work is needed.