Iyngkarran Kumar

• Iyngkarran Kumar 28 Mar 2024 15:21 UTC

  1 point

  1 ∶ 0

  on: Long Reflec­tion Read­ing List

  Great resource, thanks for putting this together!

• Iyngkarran Kumar 27 Nov 2023 15:52 UTC

  1 point

  0 ∶ 0

  on: The Grabby Values Selec­tion Th­e­sis: What val­ues do space-far­ing civ­i­liza­tions plau­si­bly have?

  Just stumbled upon this sequence and happy to have found it! There seems to be lots of analysis ripe for picking here.
  

  Some thoughts on the strength of the grabbiness selection effect below. I’ll likely to come back to this to add further thoughts in the future.

  One factor that seems to be relevant here is the number of pathways to technological completion. If we assume that the only civilisations that dominate the universe in the far future are the ones that have reached technological completion (seems pretty true to me), then tautologically, the dominating civilisations must be those who have walked the path to technological completion. Now imagine that in order to reach technological completion, you must tile 50% of the planets under your control with computer chips, but your value system means that you assign huge disvalue to tiling planets with computer chips*. As a result, you’ll refuse to walk the path to technological completion, and be subjugated or wiped out by the civilisations that did go forward with this action.

  The more realistic example here is a future in which suffering subroutines are a necessary step towards technological completion, and so civilisations that disvalue suffering enough to not take this step will be dominated by civilisations that either (1) don’t care for suffering or (2) are willing to bite the bullet of creating suffering sub-routines in order to pre-emptively colonise their available resources.

  So the question here is how many paths are there to technological completion? Technological completion could be like a mountain summit that is accessible from many directions—in that case, if your value system doesn’t allow you to follow one path, you can change course and reach the summit from the other direction. But if there’s just a single path with some steps that are necessary to take, then this will constrain the set of value systems that dominate the far future. Sketching out precedents for technological completion would be a first step to gaining clarity here.
  

  *This value system is just for the thought experiment, I’m not claiming that it’s a likely one.

• Iyngkarran Kumar 4 Oct 2023 10:15 UTC

  1 point

  0 ∶ 0

  in reply to: Max H’s comment on: Model­ling large-scale cy­ber at­tacks from ad­vanced AI sys­tems with Ad­vanced Per­sis­tent Threats

  Yep, the variance of human worker teams should definitely be stressed. It’s plausible that a super team of hackers might have attack workloads on the scale of 100s to 1000s of hours ^[1], whereas for lower quality teams, this may be more like 100,000s of hours.

  Thinking about it, I can probably see significant variance amongst AI systems due to various degrees of finetuning on cyber capabilities ^[2](though as you said, not as much variance as human teams). E.g: A capable foundational model may map to something like a 60th percentile hacker and so have attack workloads on the order of 10,000s of hours (like in this piece). A finetuned model might map to a 95th percentile hacker and so a team of these may have workloads on the scale of 1000s of hours.

  1. ^

     Though 100s of hours seems more on the implausible side—I’m guessing this would require a very large team (100s) of very skilled hackers.

  2. ^

     And other relevant skills, like management

Model­ling large-scale cy­ber at­tacks from ad­vanced AI sys­tems with Ad­vanced Per­sis­tent Threats

• Iyngkarran Kumar 24 Aug 2023 23:49 UTC

  2 points

  1 ∶ 0

  in reply to: WilliamKiely’s comment on: The Hu­man Fu­ture (x-risk and longter­mism-themed video by melodysheep)

  Kurzgesagt script + Melody Sheep music and visuals = great video about the long term future. Someone should get a colab between the two going

• Iyngkarran Kumar 28 Jun 2023 11:00 UTC

  1 point

  0 ∶ 0

  in reply to: Sam Clarke’s comment on: The longter­mist AI gov­er­nance land­scape: a ba­sic overview

  Intuitively I feel that this process does generalise, and I would personally be really keen to read case studies of an idea/​strategy that was moved from left to right in the diagram above. i.e a thinker initially identifies a problem, and over the following years or decades it moves to tactics research, then policy development, then advocacy and finally is implemented. I doubt any idea in AI governance has gone through the full strategy-to-implementation lifecycle, but maybe one in climate change, nuclear risk management, or something else has? Would appreciate if anyone could link case studies of this sort!

• Iyngkarran Kumar 28 Feb 2023 22:15 UTC

  4 points

  0 ∶ 0

  on: Im­por­tant, ac­tion­able re­search ques­tions for the most im­por­tant century

  I’m trying to get a better picture in my head of what good work looks like in this space—i.e, existing work that has given us improved strategic clarity. This could be with regards to TAI itself, or a technology such as nuclear weapons.

  Imo, some examples of valuable strategic work/​insights are:

  • Ajeya Cotra’s work on forecasting TAI via bioanchors

  • Niels Bohr (and other scientists) realising during the development of the nuclear bomb that an oligopoly on nuclear weapons could lead to relative peace between global superpowers. (https://​​www.governance.ai/​​research-paper/​​lessons-atomic-bomb-ord)

  I’m curious as to any other examples of existing work that you think fit into the category of valuable strategic work, of the type that you talk about in this post.

• Iyngkarran Kumar 4 Nov 2022 21:15 UTC

  4 points

  0 ∶ 0

  in reply to: Peter’s comment on: All AGI Safety ques­tions wel­come (es­pe­cially ba­sic ones) [~monthly thread]

  This is a really comprehensive post on pursuing a career in technical AI safety, including how to test fit and skill up

• Iyngkarran Kumar 16 Aug 2022 22:37 UTC

  31 points

  0 ∶ 0

  on: What We Owe The Fu­ture is out today

  Hi Will,

  From listening to your podcast with Ali Abdaal, it seems that you’re relatively optimistic about humanity being able to create aligned AI systems. Could you explain the main reasons behind your thinking here?

  Thanks!

• Iyngkarran Kumar 21 Jul 2022 23:23 UTC

  1 point

  0 ∶ 0

  in reply to: Jack R’s comment on: How to be­come an AI safety researcher

  Strongly second this^