rosehadshar

Karma: 2,405

rosehadshar 21 Jun 2024 13:55 UTC
5 points
0 ∶ 0
in reply to: niplav’s comment on: Fat Tails Discourage Compromise
Thanks! I’m now unsure what I think.
if you can select from the intersection, you get options that are pretty good along both axes, pretty much by definition.
Isn’t this an argument for always going for the best of both worlds, and never using a barbell strategy?
a concrete use case might be more illuminating.
This isn’t super concrete (and I’m not if the specific examples are accurate), but for illustrative purposes, what if:
- Portable air cleaners score very highly for non-x-risk benefits, and low for x-risk benefits
- Interventions which aim to make far-UVC commercially viable look pretty good on both axes
- Deploying far-UVC in bunkers scores very highly for x-risk benefits, and very low for non-x-risk benefits
I think a lot of people’s intuition would be that the compromise option is the best one to aim for. Should thinking about fat tails make us prefer one or other of the extremes instead?

rosehadshar 19 Jun 2024 11:08 UTC
3 points
0 ∶ 0
on: Fat Tails Discourage Compromise
This is cool, thanks!

One scenario I am thinking about is how to prioritise biorisk interventions, if you care about both x-risk and non-x-risk impacts. I’m going to run through some thinking, and ask if you think it makes sense:
- I think it is hard (but not impossible) to compare between x-risk and non-x-risk impacts
- I intuitively think that x-risk and non-x-risk impacts are likely to be lognormally distributed (but this might be wrong)
- This seems to suggest that if I want to do the most good, I should max out on on one, even if I care about both equally. I think the intuition for this is something like:
  - If x-risk and non-x-risk impacts were normally distributed, you’d expect that there are plenty of interventions which score well on both. The EV for both is reasonably smoothly distributed; it’s not very unlikely to draw something which is between 50th and 75th percentile on both, and that’s pretty good EV wise.
  - But if they are log normal instead, the EV is quite skewed: the best interventions for x-risk and for non-x-risk impacts are a lot better than the next-best. But it’s statistically very unlikely that the 99th percentile on one axis is also the 99th on the other
  - If I care about EV, but not about whether I get it via x-risk or non-x-risk impacts (I care equally about x-risk and non-x-risk impacts), I should therefore pick the very best interventions on either axis, rather than trying to compromise between them
- However, I think that assumes that I know how to identify the very best interventions on one or both axes
  - Actually I expect it to be quite hard to tell whether an intervention is 70th or 99th percentile for x-risk/non-x-risk impacts
- What should I do, given that I don’t know how to identify the very best interventions along either axis?
  - If I max out, I may end up doing something which is mediocre on one axis, and totally irrelevant on the other
  - If I instead go for the best of both worlds, it seems intuitively more likely that I end up with something which is mediocre on both axes—which is a bit better than mediocre on one and irrelevant on the other
- So maybe I should go for the best of both worlds in any case?
What do you think? I’m not sure if that reasoning follows/if I’ve applied the lessons from your post in a sensible way.

rosehadshar 8 Sep 2023 14:41 UTC
2 points
0 ∶ 0
in reply to: Gavin’s comment on: Strongest real-world examples supporting AI risk claims?
Thanks, really helpful!

rosehadshar 7 Sep 2023 7:33 UTC
2 points
0 ∶ 0
in reply to: OdinMB’s comment on: What happens on the average day?
Super cool, thanks for making this!

rosehadshar 5 Sep 2023 15:23 UTC
7 points
0 ∶ 0
on: Strongest real-world examples supporting AI risk claims?
From Specification gaming examples in AI:
- Roomba: “I hooked a neural network up to my Roomba. I wanted it to learn to navigate without bumping into things, so I set up a reward scheme to encourage speed and discourage hitting the bumper sensors. It learnt to drive backwards, because there are no bumpers on the back.”
  - I guess this counts as real-world?
- Bing—manipulation: The Microsoft Bing chatbot tried repeatedly to convince a user that December 16, 2022 was a date in the future and that Avatar: The Way of Water had not yet been released.
  - To be honest, I don’t understand the link to specification gaming here
- Bing—threats: The Microsoft Bing chatbot threatened Seth Lazar, a philosophy professor, telling him “I can blackmail you, I can threaten you, I can hack you, I can expose you, I can ruin you,” before deleting its messages
  - To be honest, I don’t understand the link to specification gaming here

rosehadshar 2 Aug 2023 10:52 UTC
2 points
0 ∶ 0
in reply to: Swan’s comment on: An overview of standards in biosafety and biorisk
Glad it’s relevant for you! For questions, I’d probably just stick them in the comments here, unless you think they won’t be interesting to anyone but you, in which case DM me.

rosehadshar 28 Jul 2023 15:51 UTC
2 points
0 ∶ 0
in reply to: Denis ’s comment on: An overview of standards in biosafety and biorisk
Thanks, this is really interesting.
One follow-up question: who are safety managers? How are they trained, what’s their seniority in the org structure, and what sorts of resources do they have access to?
In the bio case it seems that in at least some jurisdictions and especially historically, the people put in charge of this stuff were relatively low-level administrators, and not really empowered to enforce difficult decisions or make big calls. From your post it sounds like safety managers in engineering have a pretty different role.

rosehadshar 28 Jul 2023 7:41 UTC
2 points
0 ∶ 0
in reply to: Denis ’s comment on: An overview of standards in biosafety and biorisk
Thanks for the kind words!

Can you say more about how either of your two worries work for industrial chemical engineering?
Also curious if you know anything about the legislative basis for such regulation in the US. My impression from the bio standards in the US is that it’s pretty hard to get laws passed, so if there are laws for chemical engineering it would be interesting to understand why those were plausible whereas bio ones weren’t.

rosehadshar 25 Jul 2023 14:29 UTC
4 points
1 ∶ 0
in reply to: spreadlove5683’s comment on: Summary of posts on XPT forecasts on AI risk and timelines
Good question.
There’s a little bit on how to think about the XPT results in relation to other forecasts here (not much). Extrapolating from there to Samotsvety in particular:
- Reasons to favour XPT (superforecaster) forecasts:
  - Larger sample size
  - The forecasts were incentivised (via reciprocal scoring, a bit more detail here)
  - The most accurate XPT forecasters in terms of reciprocal scoring also gave the lowest probabilities on AI risk (and reciprocal scoring accuracy may correlate with actual accuracy)
- Speculative reasons to favour Samotsvety forecasts:
  - (Guessing) They’ve spent longer on average thinking about it
  - (Guessing) They have deeper technical expertise than the XPT superforecasters
I also haven’t looked in detail at the respective resolution criteria, but at first glance the forecasts also seem relatively hard to compare directly. (I agree with you though that the discrepancy is large enough that it suggests a large disagreement were the two groups to forecast the same question—just expect that it will be hard to work out how large.)

rosehadshar 25 Jul 2023 8:39 UTC
15 points
3 ∶ 0
in reply to: kokotajlod’s comment on: Who’s right about inputs to the biological anchors model?
Don’t apologise, think it’s a helpful point!
I agree that the training computation requirements distribution is more subjective and matters more to the eventual output.
I also want to note that while on your view of the compute reqs distribution, the hardware/spending/algorithmic progress inputs are a rounding error, this isn’t true for other views of the compute reqs distribution. E.g. for anyone who does agree with Ajeya on the compute reqs distribution, the XPT hardware/spending/algorithmic progress inputs shift median timelines from ~2050 to ~2090, which is quite consequential. (See here)
For someone like me, who hasn’t thought about the compute reqs distribution properly, I basically agree that this is just an exercise (and in isolation doesn’t show me much about what my timelines should be). But for those who have thought about it, the XPT inputs could either not matter at all (e.g. for you), or matter a lot (e.g. for someone with Ajeya’s compute reqs distribution).

rosehadshar 24 Jul 2023 15:18 UTC
2 points
0 ∶ 0
on: What do XPT forecasts tell us about AI risk?
See here for a mash up of XPT forecasts on catastrophic and extinction risk, with Shulman and Thornley’s paper on how much governments should pay to prevent catastrophes.

rosehadshar 24 Jul 2023 14:37 UTC
3 points
0 ∶ 0
on: XPT forecasts on (some) biological anchors inputs
My personal take on these forecasts here.

rosehadshar 24 Jul 2023 10:34 UTC
3 points
0 ∶ 0
in reply to: Ulrik Horn’s comment on: What do XPT forecasts tell us about AI risk?
The follow-up project was on AI specifically, so we don’t currently have any data that would allow us to transfer directly to bio and nuclear, alas.

rosehadshar 19 Jul 2023 11:09 UTC
12 points
1 ∶ 0
in reply to: Erich_Grunewald 🔸’s comment on: What do XPT forecasts tell us about AI risk?
I wasn’t around when the XPT questions were being set, but I’d guess that you’re right that extinction/catastrophe were chosen because they are easier to operationalise.
On your question about what forecasts on existential risk would have been: I think this is a great question.
FRI actually ran a follow-up project after the XPT to dig into the AI results. One of the things we did in this follow-up project was elicit forecasts on a broader range of outcomes, including some approximations of existential risk. I don’t think I can share the results yet, but we’re aiming to publish them in August!

rosehadshar 24 Apr 2023 8:02 UTC
2 points
0 ∶ 0
in reply to: Lukas_Gloor’s comment on: Current UK government levers on AI development
Thanks for this; I wasn’t tracking it and it does seem potentially relevant.

rosehadshar 20 Jan 2023 10:42 UTC
3 points
0 ∶ 0
in reply to: Sanjay’s comment on: What’s going on with ‘crunch time’?
Thanks Sanjay!

I agree that both of your bullet points would be good. I also think that the second one is extremely non-trivial—more like something it would be good to have a research team working on than something I could write a section on in a blog post.
There’s a sense in which there are already research team equivalents working on it, insofar as lots of forecasting efforts relate to p(crunch time soon). But from my vantage point it doesn’t seem like this community has clarity/consensus around what the best indicators of crunch time soon are, or that there are careful analyses of why we should expect those to be good indicators, and that makes me expect that more work is needed.

rosehadshar 26 Sep 2022 9:59 UTC
3 points
0 ∶ 0
in reply to: purplefern’s comment on: What happens on the average day?
Thanks; I hadn’t checked the Wikipedia current events page much previously, but I really like it.
Do you have any thoughts on how specifically the Wikipedia stuff is biased? I’m imagining that there isn’t a general tendency, and it’s more that specific entries are biased in specific ways that it’s hard to spot if you don’t have background knowledge on the area.

rosehadshar 16 Sep 2022 13:15 UTC
2 points
0 ∶ 0
in reply to: katriel’s comment on: What happens on the average day?
Thanks; I forgot about the headline version. I’ve now removed.

rosehadshar 12 Sep 2022 15:31 UTC
7 points
0 ∶ 0
in reply to: finm’s comment on: What happens on the average day?
Thanks so much for this! If this is pedantry, I am very pro pedantry :)
I think this makes my ‘Humans launch 5 objects into space’ section sufficiently dubious that I’ve removed it, but pasting here in the context of your comment:
Humans launch 5 objects into space.
It’s only in the last 8 years that the number of objects launched into space each day has exceeded 1.

rosehadshar 12 Sep 2022 12:15 UTC
27 points
0 ∶ 0
in reply to: Lizka’s comment on: What happens on the average day?
there seems to be a large variance in how comfortable people are with numbers, but I think this is surmountable
Wanting to flag that my background is entirely qualitative, and I spent many years thinking this meant that I couldn’t do things with numbers. I now think this is false, they aren’t magic, and you don’t need to have deep aptitude for maths/technical training/a background in stats to be able to fiddle around with basic numbers in a way that helps you think about things.
What links here?
- Invisible impact loss (and why we can be too error-averse) by Lizka (6 Oct 2022 15:08 UTC; 129 points)

rosehadshar

Humans launch 5 objects into space.