Lizka

Pod­cast on “AI tools for ex­is­ten­tial se­cu­rity” — transcript

• LizkaApr 14, 2025, 4:58 PM

  6 points

  0 ∶ 0

  in reply to: Ben_West🔸’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  I think this is a good question, and it’s something I sort of wanted to look into and then didn’t get to! (If you’re interested, I might be able to connect you with someone/​some folks who might know more, though.)

  Quick general takes on what private companies might be able to do to make their tools more useful on this front (please note that I’m pretty out of my depth here, so take this with a decent amount of salt—and also this isn’t meant to be prioritized or exhaustive):

  • Some of the vetting/​authorization processes (e.g. FedRAMP) are burdensome, and sometimes companies do just give up/​don’t bother (see footnote 12), which narrows the options for agencies; going through this anyway could be very useful

  • Generally lowering costs for tech products can make a real difference for whether agencies will adopt them—also maybe open source products are likelier to be used(?) (and there’s probably stuff like which chips are available, which systems staff are used to, what tradeoffs on speed vs cost or similar make sense...)

  • Security is useful/​important, e.g. the tool/​system can be run locally, can be fine-tuned with sensitive data, etc. (Also I expect things like “we can basically prove that the training /​ behavior of this model satisfies [certain conditions]” will increasingly matter — with conditions like “not trained on X data” or “could not have been corrupted by any small group of people” — but my understanding/​thinking here is very vague!)

  • Relatedly I think various properties of the systems’ scaffolding will matter; e.g. “how well the tool fits with existing and future systems” — so modularity and interoperability^[1] (in general and with common systems) are very useful — and “can this tool be set up ~once but allow for different forms of access/​data” (e.g. handle differences in who can see different kinds of data, what info should be logged, etc.)

  (Note also there’s a pretty huge set of consultancies that focus on helping companies sell to the government, but the frame is quite different.)

  And then in terms of ~market gaps, I’m again very unsure, but expect that (unsurprisingly) lower-budget agencies will be especially undersupplied — in particular the DOD has a lot more funding and capacity for this kind of thing — so building things for e.g. NIST could make sense. (Although it might be hard to figure out what would be particularly useful for agencies like NIST without actually being at NIST. I haven’t really thought about this!)

  1. ^

     I haven’t looked into this at all, but given the prevalence of Microsoft systems (Azure etc.) in the US federal government (which afaik is greater than what we see in the UK), I wonder if Microsoft’s relationship with OpenAI explains why we have ChatGPT Gov in the US, while Anthropic is collaborating with the UK government https://​​www.anthropic.com/​​news/​​mou-uk-government

• LizkaApr 14, 2025, 4:18 PM

  4 points

  0 ∶ 0

  in reply to: OscarD🔸’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  the main question is how high a priority this is, and I am somewhat skeptical it is on the ITN pareto frontier. E.g. I would assume plenty of people care about government efficiency and state capacity generally, and a lot of these interventions are generally about making USG more capable rather than too targeted towards longtermist priorities.

  Agree that “how high-priority should this be” is a key question, and I’m definitely not sure it’s on the ITN pareto frontier! (Nice phrase, btw.)

  Quick notes on some things that raise the importance for me, though:

  1. I agree lots of people care about government efficiency/​ state capacity — but I suspect few of them are seriously considering the possibility of transformative AI in the near future, and I think what you do to ~boost capacity looks pretty different in that world

  2. Also/​relatedly, my worldview means I have extra reasons to care about state capacity, and given my worldview is unusual that means I should expect the world is underinvesting in state capacity (just like most people would love to see a world with fewer respiratory infections, but tracking the possibility of a bioengineered pandemic means I see stuff like far-UVC/​better PPE/​etc. as higher value)

     1. More generally I like the “how much more do I care about X” frame — see this piece from 2014

     2. (It could also be a kind of public good.)

  3. In particular, it seems like a *lot* of the theory of change of AI governance+ relies on competent/​skillful action/​functioning by the US federal government, in periods where AI is starting to radically transform the world (e.g. to mandate testing and be able to tell if that mandate isn’t being followed!), and my sense is that this assumption is fragile/​the govt may very well not actually be sufficiently competent — so we better be working on getting there, or investing more in plans that don’t rely on this assumption

  And I’m pretty worried that a decent amount of work aimed at mitigating the risks of AI could end up net-negative (for its own goals) by not tracking this issue and thus not focusing enough on the interventions that are actually worth pursuing—further harming government AI adoption & competence /​ capacity in the process (e.g. I think some of the OMB/​EO guidance from last year looked positive to me before I dug into this, and now looks negative). So I’d like to nudge some people who work on issues related to existential risk (and government) away from a view like: “all AI is scary/​bad, anything that is ‘pro-AI’ increases existential risk, if this bundle of policies/​barriers inhibits a bunch of different AI things then that’s probably great even if I think only a tiny fraction is truly (existentially) risky”, etc.

  --

  this felt like neither the sort of piece targeted to mainstream US policy folks, nor that convincing for why this should be an EA/​longtermist focus area.

  Totally reasonable reaction IMO. To a large extent I see this as a straightforward flaw of the piece & how I approached it (partly due to lack of time—see my reply to Michael above), although I’ll flag that my main hope was to surface this to people who are in fact kind of in between—e.g. folks at think tanks that do research on existential security and have government experience/​expertise.

  --

  I’m unconvinced that e.g. OP should spin up a grantmaker focused on this (not that you were necessarily recommending this).

  I am in fact not recommending this! (There could be specific interventions in the area that I’d see as worth funding, though, and it’s also related to other clusters where something like the above is reasonable IMO.)

  --

  Also, a few reasons govts may have a better time adopting AI come to mind:

  • Access to large amounts of internal private data

  • Large institutions can better afford one-time upfront costs to train or finetune specialised models, compared to small businesses

  But I agree the opposing reasons you give are probably stronger.

  The data has to be accessible, though, and this is a pretty big problem. See e.g. footnote 17.

  I agree that a major advantage could be that the federal government can in fact move a lot of money when ~it wants to, and could make some (cross-agency/​...) investments into secure models or similar, although my sense is that right now that kind of thing is the exception/​aspiration, not the rule/​standard practice. (Another advantage is that companies do want to maintain good relationships with the government/​admin, and might thus invest more in being useful. Also there are probably a lot of skilled people who are willing to help with this kind of work, for less personal gain.)

  --

  If only this were how USG juggled its priorities!

  🙃 (some decision-makers do, though!)

• LizkaApr 14, 2025, 1:24 PM

  6 points

  0 ∶ 0

  in reply to: Ozzie Gooen’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  I imagine there might be some very clever strategies to get a lot of the benefits of AI without many of the normal costs of integration.
  
  For example:

  1. The federal government makes heavy use of private contractors. These contractors are faster to adopt innovations like AI.

  2. There are clearly some subsets of the government that matter far more than others. And there are some that are much easier to improve than others.

  3. If AI strategy/​intelligence is cheap enough, most of the critical work can be paid for by donors. For example, we have a situation where there’s a think tank that uses AI to figure out the best strategies/​plans for much of the government, and government officials can choose to pay attention to this.

  I’d be excited to see more work in this direction!

  Quick notes: I think (1) is maybe the default way I expect things to go fine (although I have some worries about worlds where almost all US federal govt AI capacity is via private contractors). (2) seems right, and I’d want someone who has (or can develop) a deeper understanding of this area than me to explore this. Stuff like (3) seems quite useful, although I’m worried about things like ensuring access to the right kind of data and decision-makers (but partnerships /​ a mix of (2) and (3) could help).

  (A lot of this probably falls loosely under “build capacity outside the US federal government” in my framework, but I think the lines are very blurry /​ a lot of the same interventions help with appropriate use/​adoption of AI in the government and external capacity. )

  all very similar to previous thinking on how forecasting can be useful to the government

  I hadn’t thought about this — makes sense, and a useful flag, thank you! (I might dig into this a bit more.)

• LizkaApr 14, 2025, 2:03 AM

  26 points

  1 ∶ 0

  in reply to: MichaelDickens’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  Thanks for this comment! I don’t view it as “overly critical.”

  Quickly responding (just my POV, not Forethought’s!) to some of what you brought up ---

  (This ended up very long, sorry! TLDR: I agree with some of what you wrote, disagree with some of the other stuff /​ think maybe we’re talking past each other. No need to respond to everything here!)

  A. Motivation behind writing the piece /​ target audience/​ vibe /​ etc.

  Re:

  …it might help me if you explained more about the motivation [behind writing the article] [...] the article reads like you decided the conclusion and then wrote a series of justifications

  I’m personally glad I posted this piece, but not very satisfied with it for a bunch of reasons, one of which is that I don’t think I ever really figured out what the scope/​target audience should be (who I was writing for/​what the piece was trying to do).

  So I agree it might help to quickly write out the rough ~history of the piece:

  • I’d started looking into stuff related to “differential AI development” (DAID), and generally exploring how the timing of different [AI things] relative to each other could matter.

  • My main focus quickly became exploring ~safety-increasing AI applications/​tools — Owen and I recently posted about this (see the link).

  • But I also kept coming back to a frame of “oh crap, who is using AI how much/​how significantly is gonna matter an increasing amount as time goes on. I expect adoption will be quite uneven — e.g. AI companies will be leading the way — and some groups (whose actions/​ability to make reasonable decisions we care about a lot) will be left behind.”

    • At the time I was thinking about this in terms of “differential AI development and diffusion”

  • IIRC I soon started thinking about governments here; I had the sense that government decision-makers were generally slow on tech use, and I was also using “which types of AI applications will not be properly incentivized by the market” as a way to think about which AI applications might be easier to speed up. (I think we mentioned this here.)

  • This ended up taking me on a mini deep dive on government adoption of AI, which in turn increasingly left me with the impression that (e.g.) the US federal government would either (1) become increasingly overtaken from within by an unusually AI-capable group (or e.g. the DOD), (2) be rendered increasingly irrelevant, leaving (US) AI companies to regulate themselves and likely worsening its ability to deal with other issues, or (3) somehow in fact adopt AI, but likely in a chaotic way that would be especially dangerous (because things would go slowly until a crisis forced a ~war-like undertaking).

  • I ended up poking around in this for a while, mostly as an aside to my main DAID work, feeling like I should probably scope this out and move on. (The ~original DAID memos I’d shared with people discussed government AI adoption.)

  • After a couple of rounds of drafts+feedback I got into a “I should really publish some version of this that I believe and seems useful and then get back to other stuff; I don’t think I’m the right person to work a lot more on this but I’m hoping other people in the space will pick up whatever is correct here and push it forward” mode—and ended up sharing this piece.

  In particular I don’t expect (and wasn’t expecting) that ~policymakers will read this, but hope it’s useful for people at relevant think tanks or similar who have more government experience/​knowledge but might not be paying attention to one “side” of this issue or the other. (For instance, I think a decent fraction of people worried about existential risks from advanced AI don’t really think about how using AI might be important for navigating those risks, partly because all of AI kinda gets lumped together).

  Quick responses to some other things in your comment that seem kinda related to what I’m responding to in this “motivation/​vibe/​…” cluster:

  I also found it odd that the report did not talk about extinction risk. In its list of potential catastrophic outcomes, the final item on the list was “Human disempowerment by advanced AI”, which IMO is an overly euphemistic way of saying “AI will kill everyone”.

  We might have notably different worldviews here (to be clear mine is pretty fuzzy!). For one thing, in my view many of the scary “AI disempowerment” outcomes might not in fact look immediately like “AI kills everyone” (although to be clear that is in fact an outcome I’m very worried about), and unpacking what I mean by “disempowerment” in the piece (or trying to find the ideal way to say it) didn’t seem productive—IIRC I wrote something and moved on. I also want to be clear that rogue AI [disempowering] humans is not the only danger I’m worried about, i.e. it doesn’t dominate everything else for me—the list you’re quoting from wasn’t an attempt to mask AI takeover, but rather a sketch of the kind of thing I’m thinking about. (Note: I do remember moving that item down the list at some point when I was working on a draft, but IIRC this was because I wanted to start with something narrower to communicate the main point, not because I wanted to de-emphasize ~AI takeover.)

  By my reading, this article is meant to be the sort of Very Serious Report That Serious People Take Seriously, which is why it avoids talking about x-risk.

  I might be failing to notice my bias, but I basically disagree here—although I do feel a different version of what you’re maybe pointing to here (see next para). I was expecting that basically anyone who reads the piece will already have engaged at least a bit with “AI might kill all humans”, and likely most of the relevant audience will have thought very deeply about this and in fact has this as a major concern. I also don’t personally feel shy about saying that I think this might happen — although again I definitely don’t want to imply that I think this is overwhelmingly likely to happen or the only thing that matters, because that’s just not what I believe.

  However I did occasionally feel like I was ~LARPing research writing when I was trying to articulate my thoughts, and suspect some of that never got resolved! (And I think I floundered a bit on where to go with the piece when getting contradicting feedback from different people—although ultimately the feedback was very useful.) In my view this mostly shows up in other ways, though. (Related—I really appreciated Joe Carlsmith’s recent post on fake thinking and real thinking when trying to untangle myself here.)

  B. Downside risks of the proposed changes

  1. Making policymakers “more reluctant to place restrictions on AI development...”

     1. I did try to discuss this a bit in the “Government adoption of AI will need to manage important risks” section (and sort of in the “3. Defusing the time bomb of rushed automation” section), and indeed it’s a thing I’m worried about.

     2. I think ultimately my view is that without use of AI in government settings, stuff like AI governance will just be ineffective or fall to private actors anyway, and also that the willingness-to-regulate /​undue influence dynamics will be much worse if the government has no in-house capacity or is working with only one AI company as a provider.

  2. Shortening timelines by increasing AI company revenue

     1. I think this isn’t a major factor here—the govt is a big customer in some areas, but the private sector dominates (as does investment in the form of grants, IIRC)

  3. “The government does not always work in the interests of the people (in fact it frequently works against them!) so making the government more effective/​powerful is not pure upside.”
     

     1. I agree with this, and somewhat worry about it. IIRC I have a footnote on this somewhere -—I decided to scope this out. Ultimately my view right now is that the alternative (~no governance at least in the US, etc.) is worse. Sort of relatedly, I find the “narrow corridor” a useful frame here—see e.g. here.)

  C. Is gov competence actually a bottleneck?

  I don’t think government competence is what’s holding us back from having good AI regulations, it’s government willingness. I don’t see how integrating AI into government workflow will improve AI safety regulations (which is ultimately the point, right?[^1]), and my guess is on balance it would make AI regulations less likely to happen because policy-makers will become more attached to their AI systems and won’t want to restrict them.

  My view is that you need both, we’re not on track for competence, and we should be pretty uncertain about what happens on the willingness side.

  D. Michael’s line item responses

  1.

  > invest in AI and technical talent

  What does that mean exactly? I can’t think of how you could do that without shortening timelines so I don’t know what you have in mind here.

  I’m realizing this can be read as “invest in AI and in technical talent” — I meant “invest in AI talent and (broader) technical talent (in govt).” I’m not sure if this just addresses the comment; my guess is that doing this might have a tiny shortening effect on timelines (but is somewhat unclear, partly because in some cases e.g. raising salaries for AI roles in govt might draw people away from frontier AI companies), but this is unlikely to be the decisive factor. (Maybe related: my view is that generally this kind of thing should be weighed instead of treated as a reason to entirely discard certain kinds of interventions.)

  2.

  > Streamline procurement processes for AI products and related tech

  I also don’t understand this. Procurement by whom, for what purpose? And again, how does this not shorten timelines? (Broadly speaking, more widespread use of AI shortens timelines at least a little bit by increasing demand.)

  I was specifically talking about agencies’ procurement of AI products — e.g. say the DOE wants a system that makes forecasting demand easier or whatever; making it easier for them to actually get such a system faster. I think the effect on timelines will likely be fairly small here (but am not sure), and currently think it would be outweighed by the benefits.

  3.

  > Gradual adoption is significantly safer than a rapid scale-up.

  This sounds plausible but I am not convinced that it’s true, and the article presents no evidence, only speculation. I would like to see more rigorous arguments for and against this position instead of taking it for granted.

  I’d be excited to see more analysis on this, but it’s one of the points I personally am more confident about (and I will probably not dive in right now).

  4.

  > And in a crisis — e.g. after a conspicuous failure, or a jump in the salience of AI adoption for the administration in power — agencies might cut corners and have less time for security measures, testing, in-house development, etc.

  This line seems confused. Why would a conspicuous failure make government agencies want to suddenly start using the AI system that just conspicuously failed? Seems like this line is more talking about regulating AI than adopting AI, whereas the rest of the article is talking about adopting AI.

  Sorry, again my writing here was probably unclear; the scenarios I was picturing were more like:

  • There’s a serious breach—US govt systems get hacked (again) by [foreign nation, maybe using AI] - revealing that they’re even weaker than is currently understood, or publicly embarrassing the admin. The admin pushes for fast modernization on this front.

  • A flashy project isn’t proceeding as desired (especially as things are ramping up), the admin in power is ~upset with the lack of progress, pushes

  • There’s a successful violent attack (e.g. terrorism); turns out [agency] was acting too slowly...

  • Etc.

  Not sure if that answers the question/​confusion?

  5.

  > Frontier AI development will probably concentrate, leaving the government with less bargaining power.

  I don’t think that’s how that works. Government gets to make laws. Frontier AI companies don’t get to make laws. This is only true if you’re talking about an AI company that controls an AI so powerful that it can overthrow the government, and if that’s what you’re talking about then I believe that would require thinking about things in a very different way than how this article presents them.

  This section is trying to argue that AI adoption will be riskier later on, so the “bargaining power” I was talking about here is the bargaining power of the US federal govt (or of federal agencies) as a customer; the companies it’s buying from will have more leverage if they’re effectively monopolies. My understanding is that there are already situations where the US govt has limited negotiation power and maybe even makes policy concessions to specific companies specifically because of its relationship to those companies — e.g. in defense (Lockheed Martin, etc., although this is also kinda complicated) and again maybe Microsoft.

  And: would adopting AI (i.e. paying frontier companies so government employees can use their products) reduce the concentration of power? Wouldn’t it do the opposite?

  Again, the section was specifically trying to argue that later adoption is scarier than earlier adoption (in this case because there are (still) several frontier AI companies). But I do think that building up internal AI capacity, especially talent, would reduce the leverage any specific AI company has over the US federal government.

  6.

  > It’s natural to focus on the broad question of whether we should speed up or slow down government AI adoption. But this framing is both oversimplified and impractical

  Up to this point, the article was primarily talking about how we should speed up government AI adoption. But now it’s saying that’s not a good framing? So why did the article use that framing? I get the sense that you didn’t intend to use that framing, but it comes across as if you’re using it.

  Yeah, I don’t think I navigated this well! (And I think I was partly talking ti myself here.) But maybe my “motivation” notes above give some context?
  In terms of the specific “position” I in practice leaned into: Part of why I led with the benefits of AI adoption was the sense that the ~existential risk community (which is most of my audience) generally focuses on risks of AI adoption/​use/​products, and that’s where my view diverges more. There’s also been more discussion, from an existential risk POV, of the risks of adoption than there has been of the benefits, so I didn’t feel that elaborating too much on the risks would be as useful.

  7.

  > Hire and retain technical talent, including by raising salaries

  I would like to see more justification for why this is a good idea. The obvious upside is that people who better understand AI can write more useful regulations. On the other hand, empirically, it seems that people with more technical expertise (like ML engineers) are on average less in favor of regulations and more in favor of accelerating AI development (shortening timelines, although they usually don’t think “timelines” are a thing). So arguably we should have fewer such people in positions of government power.

  The TLDR of my view here is something like “without more internal AI/​technical talent (most of) the government will be slower on using AI to improve its work & stay relevant, which I think is bad, and also it will be increasingly reliant on external people/​groups/​capacity for technical expertise—e.g. relying on external evals, or trusting external advice on what policy options make sense, etc. and this is bad.”

  8.

  > Explore legal or other ways to avoid extreme concentration in the frontier AI market

  [...]

  The linked article attached to this quote says “It’s very unclear whether centralizing would be good or bad”, but you’re citing it as if it definitively finds centralization to be bad.

  (The linked article is this one: https://​​www.forethought.org/​​research/​​should-there-be-just-one-western-agi-project )

  I was linking to this to point to relevant discussion, not as a justification for a strong claim like “centralization is definitively bad”—sorry for being unclear!

  9.

  > If the US government never ramps up AI adoption, it may be unable to properly respond to existential challenges.

  What does AI adoption have to do with the ability to respond to existential challenges? It seems to me that once AI is powerful enough to pose an existential threat, then it doesn’t really matter whether the US government is using AI internally.

  I suspect we may have fairly different underlying worldviews here, but maybe a core underlying belief on my end is that there are things that it’s helpful for the government to do before we get to ~ASI, and also there will be AI tools pre ~ASI that are very helpful for doing those things. (Or an alt framing: the world will get ~/​fast/​complicated/​weird due to AI before there’s nothing the US gov could in theory do to make things go better.)

  10.

  > Map out scenarios in which AI safety regulation is ineffective and explore potential strategies

  I don’t think any mapping is necessary. Right now AI safety regulation is ineffective in every scenario, because there are no AI safety regulations (by safety I mean notkilleveryoneism). Trivially, regulations that don’t exist are ineffective. Which is one reason why IMO the emphasis of this article is somewhat missing the mark—right now the priority should be to get any sort of safety regulations at all.

  I fairly strongly disagree here (with “the priority should be to get any sort of safety regulations at all”) but don’t have time to get into it, really sorry!

  ---
  

  Finally, thanks a bunch for saying that you enjoyed some of my earlier writing & I changed your thinking on slow vs quick mistakes! That kind of thing is always lovely to hear.

  (Posted on my phone— sorry for typos and similar!)

The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

• LizkaMar 18, 2025, 1:18 AM

  15 points

  1 ∶ 0

  on: Lizka’s Shortform

  Quick (visual) note on something that seems like a confusion in the current conversation:

  

  
  Others have noted similar things (eg, and Will’s earlier take on total vs human extinction). You might disagree with the model (curious if so!), but I’m a bit worried that one way or another people are talking past each other (least from skimming the discussion).
  
  (Commenting via phone, sorry for typos or similar!)

• LizkaMar 17, 2025, 4:59 PM

  8 points

  0 ∶ 0

  on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  What actually changes about what you’d work on if you concluded that improving the future is more important on the current margin than trying to reduce the chance of (total) extinction (or vice versa)?

  Curious for takes from anyone!

• LizkaMar 14, 2025, 7:32 PM

  15 points

  0 ∶ 0

  on: AI Tools for Ex­is­ten­tial Security

  I wrote a Twitter thread that summarizes this piece and has a lot of extra images (I probably went overboard, tbh.)

  I kinda wish I’d included the following image in the piece itself, so I figured I’d share it here:

  

AI Tools for Ex­is­ten­tial Security

• LizkaFeb 17, 2025, 1:29 PM

  9 points

  1 ∶ 0

  on: Fake think­ing and real thinking

  I just want to say that I really appreciated this post — it came at exactly the right time for me and I’ve referenced it several times since you shared it.

Some mod­er­a­tion team principles

• LizkaFeb 4, 2025, 1:42 PM

  4 points

  0 ∶ 0

  in reply to: Lizka’s comment on: Lizka’s Shortform

  Follow-up:

  Quick list of some ways benchmarks might be (accidentally) misleading^[1]

  1. Poor “construct validity”^[2]( & systems that are optimized for the metric)

     1. The connection between what the benchmark is measuring and what it’s trying to measure (or what people think it’s measuring) is broken. In particular:

     2. Missing critical steps

        1. When benchmarks are trying to evaluate progress on some broad capability (like “engineering” or “math ability” or “planning”), they’re often testing specific performance on meaningfully simpler tasks. Performance on those tasks might be missing key aspects of true/​real-world/​relevant progress on that capability.

     3. Besides the inherent difficulty of measuring the right thing, it’s important to keep in mind that systems might have been trained specifically to perform well on a given benchmark.

        1. This is probably a bigger problem for benchmarks that have gotten more coverage.

     4. And some benchmarks have been designed specifically to be challenging for existing leading models, which can make new/​other AI systems appear to have made more progress on these capabilities (relative to the older models) than they actually are.

        1. We’re seeing this with the “Humanity’s Last Exam” benchmark.

     5. ...and sometimes some of the apparent limitations are random or kinda fake, such that a minor improvement appears to lead to radical progress

  2. Discontinuous metrics: (partial) progress on a given benchmark might be misleading.

     1. The difficulty of tasks/​tests in a benchmark often varies significantly (often for good reason), and reporting of results might explain the benchmark by focusing on its most difficult tests instead of the ones that the model actually completed.

        1. I think this was an issue for Frontier Math, although I’m not sure how much strongly to discount some of the results as a result.

     2. -> This (along with issues like 1b above, which can lead to saturation) is part of what makes it harder to extrapolate from e.g. plots of progress on certain benchmarks.

  3. Noise & poisoning of the metric: Even on the metric in question, data might have leaked into the training process, the measurement process itself can be easily affected by things like who’s running it, comparing performance of different models that were tested slightly differently might be messy, etc. Some specific issues here (I might try to add links later):

     1. Brittleness to question phrasing

     2. Data contamination (discussed e.g. here)

     3. Differences in how the measurement/​testing process was actually set up (including how much inference compute was used, how many tries a model was given, access to tools, etc.)

  4. Misc

     1. Selective reporting/​measurement (AI companies want to report their successes)

     2. Tasks that appear difficult (because they’re hard for humans) might be might be especially easier for AI systems (and vice versa) — and this might cause us to think that more (or less) progress is being made than is true

        1. E.g. protein folding is really hard for humans

        2. This looks relevant, but I haven’t read it

     3. Some benchmarks seem mostly irrelevant to what I care about

     4. Systems are tested pre post-training enhancements or other changes

  Additions are welcome! (Also, I couldn’t quickly find a list like this earlier, but I’d be surprised if a better version than what I have above wasn’t available somewhere; I’d love recommendations.)

  Open Phil’s announcement of their now-closed benchmarking RFP has some useful notes on this, particularly the section on “what makes for a strong benchmark.”I also appreciated METR’s list of desiderata here.

  1. ^

     To be clear: I’m not trying to say anything on ways benchmarks might be useful/​harmful here. And I’m really not an expert.

  2. ^

     This paper looks relevant but I haven’t read it.

• LizkaFeb 3, 2025, 10:26 PM

  28 points

  3 ∶ 0

  on: Lizka’s Shortform

  TLDR: Notes on confusions about what we should do about digital minds, even if our assessments of their moral relevance are correct^[1]

  I often feel quite lost when I try to think about how we can “get digital minds right.” It feels like there’s a variety of major pitfalls involved, whether or not we’re right about the moral relevance of some digital minds.

  Digital-minds-related pitfalls in different situations
  Reality ➡️ Our perception ⬇️	These digital minds are (non-trivially) morally relevant[2]	These digital minds are not morally relevant
  We see these digital minds as morally relevant	(1) We’re right But we might still fail to understand how to respond, or collectively fail to implement that response.	(2) We’re wrong We confuse ourselves, waste an enormous amount of resources on this[3] (potentially sacrificing the welfare of other beings that do matter morally), and potentially make it harder to respond to the needs of other digital minds in the future (see also).
  We don’t see these digital minds as morally relevant	(3) We’re wrong The default result here seems to be moral catastrophe through ignorance/​sleepwalking (although maybe we’ll get lucky).	(4) We’re right All is fine (at least on this front).

  Categories (2) and (3) above are ultimately about being confused about which digital minds matter morally — having an inappropriate level of concern, one way or another. A lot of the current research on digital minds seems to be aimed at avoiding this issue. (See e.g. and e.g..) I’m really glad to see this work; the pitfalls in these categories worry me a lot.

  But even if we end up in category (1) — we realize correctly that certain digital minds are morally relevant — how can we actually figure out what we should do? Understanding how to respond probably involves answering questions like:

  • What matters for these digital minds? What improves their experience? (How can we tell?)

  • Is their experience overall negative/​positive? Should we be developing such minds? (How can we tell??) (How can we ethically design digital minds?)

  • Should we respect certain rights for these digital minds? (..?)

  • ??

  Answering these questions seem really difficult.

  In many cases, extrapolating from what we can tell about humans seems inappropriate.^[4] (Do digital minds systems find joy or meaning in some activities? Do they care about survival? What does it mean for a digital mind/​AI system to be in pain? Is it ok to lie to AI systems? Is it ok to design AI systems that have no ~goals besides fulfilling requests from humans?)

  And even concepts we have for thinking about what matters to humans (or animals) often seem ill-suited for helping us with digital minds.^[5] (When does it make sense to talk about freedoms and rights,^[6] or the sets of (relevant) capabilities a digital mind has, or even their preferences? What even is the right “individual” to consider? Self-reports seem somewhat promising, but when can we actually rely on them as signals about what’s important for digital minds?)

  I’m also a bit worried that too much of the work is going towards the question of “which systems/​digital minds are morally relevant” vs to the question of “what do we do if we think that a system is morally relevant (or if we’re unsure)?” (Note however that I’ve read a tiny fraction of this work, and haven’t worked in the space myself!) E.g. this paper focuses on the former and closes by recommending that companies prepare to make thoughtful decisions about how to treat the AI systems identified as potentially morally significant by (a) hiring or appointing a DRI (directly responsible individual) for AI welfare and (b) developing certain kinds of frameworks for AI welfare oversight. These steps do seem quite good to me (at least naively), but — as the paper explicitly acknowledges — they’re definitely not sufficient.

  Some of the work that’s apparently focused at the question of “which digital minds matter morally” involves working towards theories of ~consciousness, and maybe that will also help us with the latter question. But I’m not sure.

  (So my quite uninformed independent impression is that it might be worth investing a bit more in trying to figure out what we should do if we do decide that some digital minds might be morally relevant, or maybe what we should do if we find that we’re making extremely little progress on figuring out whether they are.)

  ---

  These seem like hard problems/​questions, but I want to avoid defeatism.

  I appreciated @rgb‘s closing remark in this post (bold mine):

  To be sure, our neuroscience tools are way less powerful than we would like, and we know far less about the brain than we would like. To be sure, our conceptual frameworks for thinking about sentience seem shaky and open to revision. Even so, trying to actually solve the problem by constructing computational theories which try to explain the full range of phenomena could pay significant dividends. My attitude towards the science of consciousness is similar to Derek Parfit’s attitude towards ethics: since we have only just begun the attempt, we can be optimistic.

  ---

  Some of the other content I’ve read/​skimmed feels like it’s pointing in useful directions on these fronts (and more recommendations are welcome!):

  • Improving the Welfare of AIs: A Nearcasted Proposal—Ryan Greenblatt on LW

  • To understand AI sentience, first understand it in animals—Jonathan Birch in Aeon (I’ve only skimmed this, though)

  • ​​Jeff Sebo on digital minds, and how to avoid sleepwalking into a major moral catastrophe − 80,000 Hours (I’ve only skimmed this, again)

  1. ^

     I’m continuing my quick take spree, with a big caveat that I’m not a philosopher and haven’t read nearly as much research/​writing on digital minds as I want to.

     And I’m not representing Forethought here! I don’t know what Forethought folks think about what I’m writing here.

  2. ^

     The more appropriate term, I think, is “moral patienthood.” And we probably care about the degree of moral consideration that is merited in different cases.

  3. ^

     This paper on “Sharing the world with digital minds” mentions this as one of the many failure modes — note that I’ve only skimmed it.

  4. ^

     I’m often struck by how appropriate the War with the Newts is as an analogy/​illustration/​prediction for a bunch of what I’m seeing today, including on issues related to digital minds. But there’s one major way in which the titular newts differ significantly from potential morally relevant digital minds; the newts are quite similar to humans and can, to some extent, communicate their preferences in ways the book’s humans could understand if they were listening.

     (One potential exception: there’s a tragic-but-brilliant moment in which “The League for the Protection of Salamanders” calls women to action sewing modest skirts and aprons for the Newts in order to appease the Newts’ supposed sense of propriety.)

  5. ^

     This list of propositions touches on related questions/​ideas.

  6. ^

     See e.g. here:

     Whether instrumentally-justified or independently binding, the rights that some AI systems could be entitled to might be different from the rights that humans are entitled to. This could be because, instrumentally, a different set of rights for AI systems promotes welfare. For example, as noted by Shulman and Bostrom (2021), naively granting both “reproductive” rights and voting rights to AI systems would have foreseeably untenable results for existing democratic systems: if AI systems can copy themselves at will, and every copy gets a vote, then elections could be won via tactical copying. This set of rights would not promote welfare and uphold institutions in the same way that they do for humans. Or AI rights could differ because, independently of instrumental considerations, their different properties entitle them to different rights—analogously to how children and animals are plausibly entitled to different rights than adults.

• LizkaFeb 2, 2025, 11:11 PM

  5 points

  0 ∶ 0

  in reply to: NickLaing’s comment on: Lizka’s Shortform

  Thanks for saying this!

  I’m trying to get back into the habit of posting more content, and aiming for a quick take makes it easier for me to get over perfectionism or other hurdles (or avoid spending more time on this kind of thing than I endorse). But I’ll take this as a nudge to consider sharing things as posts more often. :)

• LizkaFeb 2, 2025, 10:59 PM

  82 points

  10 ∶ 0

  on: Lizka’s Shortform

  When I try to think about how much better the world could be, it helps me to sometimes pay attention to the less obvious ways that my life is (much) better than it would have been, had I been born in basically any other time (even if I was born among the elite!).

  So I wanted to make a quick list of some “inconspicuous miracles” of my world. This isn’t meant to be remotely exhaustive, and is just what I thought of as I was writing this up. The order is arbitrary.

  1. Washing machines

  It’s amazing that I can just put dirty clothing (or dishes, etc.) into a machine that handles most of the work for me. (I’ve never had to regularly hand-wash clothing, but from what I can tell, it is physically very hard, and took a lot of time. I have sometimes not had a dishwasher, and really notice the lack whenever I do; my back tends to start hurting pretty quickly when I’m handwashing dishes, and it’s easier to start getting annoyed at people who share the space.)

  

  Source: Illustrations from a Teenage Holiday Journal (1821). See also.

  2. Music (and other media!)

  Just by putting on headphones, I get better music performances than most royalty could ever hope to see in their courts. (The first recording was made in the 19th century. Before that — and also widespread access to radios — few people would get access to performances by “professional” musicians, maybe besides church or other gatherings. (Although I kind of expect many people were somewhat better at singing than most are today.)) (A somewhat silly intuition pump: apparently Franz Liszt inspired a mania. But most 19th century concert attendees wouldn’t be hearing him more than once or maybe a handful of times! So imagine listening to a piece that rocks your world, and then only being able to remember it by others’ recreations or memorabilia you’ve managed to get your hands on.)

  In general, entertainment seems so much better today. Most people in history were illiterate — their entertainment might come from their own experiences or maybe telling each other stories, games, and occasional plays or other events. Of the people who could, at least, read books, most had access to very few. Today, we have a huge number of books and other texts, as well as TV/​movies, and more. I have access to video demonstrations of a functionally endless number of things, (color!) photos of an enormous amount of art (I remember being struck by my mom mentioning that she’d always wished to see some building or other when she was younger, as she’d seen it in a book that had a catalogue of black-and-white architecture pictures at some point), etc.

  

  People listening to a touring phonograph, from the 1900 inaugural issue of Boletín fonográfico. Source: Inventing the recording.

  3. Glasses

  I have pretty mild shortsightedness (with a mild astigmatism) but even then, I’m really glad that I have glasses; wearing them prevents headaches/​ fatigue, makes it easier for me to see people’s faces (or street signs), and makes a huge difference if I’m trying to see a whiteboard (or watch a show or movie). And a number of my friends and family are near-blind without glasses, in ways that make me feel a lot of sympathy for people in the past; eye problems aren’t new. (In some ways, the modern world requires us to engage more with things like written signage, but I think many people in history had duller or more painful lives because they lacked glasses.)

  

  Source: Highlights from Folger Shakespeare Library’s Release of almost 80,000 Images

  4. Not having to marry someone against my will

  By the standards of most of history, I’d be something like an old maid. I’m so grateful that I’ve never felt pressure of any kind to marry someone I didn’t want to.

  

  Source.

  5. Waterproof shoes /​ umbrellas/​ coats

  I live in Oxford, and have shoes in which I can walk through ankle-high puddles without wetting my feet. But whenever I get my feet wet and have to stay out in the cold — if I forget to bring an umbrella or coat and it rains heavily — I get quite miserable! Rain isn’t a modern development; a lot of people had to deal with these problems a lot. Moreover, most people’s houses were leakier and significantly less insulated than mine.

  

  Source: Wikimedia

  6. Photos and videos of loved ones, ability to (video) call them

  We have very few pictures of my brother when he was very young (before my parents moved to the US), and very few childhood photos of my parents or older relatives. There are significantly more pictures of me, and I took tons of photos of my little sister — and love looking at them. And I can call my family and friends when we’re not in the same place! I don’t need to wait weeks to send and receive letters, hoping that they will arrive, or rely on sketches or miniatures.

  7. Sewage systems, garbage disposal, menstrual products, diapers, etc.

  I don’t need to use an outhouse, and walk along streets that are basically free of sewage and trash. I’m not chronically ill; I can be pretty confident that the water I drink isn’t contaminated by excrement. My life is basically normal when I’m on my period. When I’ve changed babies, I could just throw away the diaper.

  

  Source.

  I don’t want to pretend that life in earlier periods was all bad! (People had solutions to some of these problems that meant they could handle them, in some cases, better than I can today when they arose. And people had wonderful and meaningful moments — here’s a really touching collection of photos of friendship I saw as I was finding the pictures above.) But I’m so grateful for some things that have changed at least in some places.

  What do I want to see in the future?

  The world could be so much better. There are daily tragedies or horrors that need to stop, risks we face that I want to reduce, and privileges enjoyed by some that I want to see extended to everyone else. But even the ~daily standards of today’s privileged groups could go up by a lot. For instance, I think our treatment and practices around mental health /​ psychological pain are still far from what they could be — I often feel like we’re in the 18th century equivalent of medicine on this front. Chronic pain management and aging science (e.g handling dementia) is also significantly farther behind what it could be. Pain management (and healthcare more generally) has improved massively over the last decades/​centuries, but people still have horrible migraines, long diseases, etc. I think we still have less quality time to spend with people we love and on things we find meaningful than we could. Technology could augment (or reduce the need for) our willpower in some areas, etc.

  

  Source.

• LizkaFeb 1, 2025, 7:56 PM

  64 points

  4 ∶ 0

  on: Lizka’s Shortform

  Notes on some of my AI-related confusions^[1]

  It’s hard for me to get a sense for stuff like “how quickly are we moving towards the kind of AI that I’m really worried about?” I think this stems partly from (1) a conflation of different types of “crazy powerful AI”, and (2) the way that benchmarks and other measures of “AI progress” de-couple from actual progress towards the relevant things. Trying to represent these things graphically helps me orient/​think.

  First, it seems useful to distinguish the breadth or generality of state-of-the-art AI models and how able they are on some relevant capabilities. Once I separate these out, I can plot roughly where some definitions of “crazy powerful AI” apparently lie on these axes:

  

  (I think there are too many definitions of “AGI” at this point. Many people would make that area much narrower, but possibly in different ways.)

  Visualizing things this way also makes it easier for me^[2] to ask: Where do various threat models kick in? Where do we get “transformative” effects? (Where does “TAI” lie?)

  

  Another question that I keep thinking about is something like: “what are key narrow (sets of) capabilities such that the risks from models grow ~linearly as they improve on those capabilities?” Or maybe “What is the narrowest set of capabilities for which we capture basically all the relevant info by turning the axes above into something like ‘average ability on that set’ and ‘coverage of those abilities’, and then plotting how risk changes as we move the frontier?”

  The most plausible sets of abilities like this might be something like:

  • Everything necessary for AI R&D^[3]

  • Long-horizon planning and technical skills?

  If I try the former, how does risk from different AI systems change?

  

  And we could try drawing some curves that represent our guesses about how the risk changes as we make progress on a narrow set of AI capabilities on the x-axis. This is very hard; I worry that companies focus on benchmarks in ways that make them less meaningful, so I don’t want to put performance on a specific benchmark on the x-axis. But we could try placing some fuzzier “true” milestones along the way, asking what the shape of the curve would be in reference to those, and then trying to approximate how far along we are with respect to them by using a combination of metrics and other measures. (Of course, it’s also really difficult to develop a reasonable/​useful sense for how far apart those milestones are on the most appropriate measure of progress — or how close partial completion of these milestones is to full completion.)

  Here’s a sketch:

  

  Overall I’m really unsure of which milestones I should pay attention to here, and how risk changes as we might move through them.

  It could make sense to pay attention to real-world impacts of (future) AI systems instead of their ~intrinsic qualities, but real-world impacts seem harder to find robust precursors to, rely on many non-AI factors, and interpreting them involves trying to untangle many different cruxes or worldviews. (Paying attention to both intrinsic qualities and real-world impacts seems useful and important, though.)

  All of this also complicates how I relate to questions like “Is AI progress speeding up or slowing down?” (If I ignore all of these confusions and just try to evaluate progress intuitively /​ holistically, it doesn’t seem to be slowing down in relevant ways.^[4])

  ---

  Thoughts/​suggestions/​comments on any of this are very welcome (although I may not respond, at least not quickly).

  ---

  Some content related to at least some of the above (non-exhaustive):

  • METR’s RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts

  • AI Impacts page on HLAI, especially “Human-level” is superhuman

  • List of some definitions of advanced AI systems

  • John Wentworth distinguishing between “early transformative AI” and “superintelligence”

  • Holden Karnofsky’s recent piece for the Carnegie Endowment for International Peace: AI Has Been Surprising for Years

  • Writing on (issues with) benchmarks/​evals: Kelsey Piper in Vox, Anthropic’s “challenges in evaluating AI systems”, Epoch in 2023 on how well compute predicts benchmark performance (pretty well on average, harder individually)

  • Recent post that I appreciated, which outlined timelines via some milestones and then outlined a picture of how the world might change in the background

  1. ^

     Not representing “Forethought views” here! (I don’t know what Forethought folks think of all of this.)

     Written/​drawn very quickly.

  2. ^

     This diagram also makes me wonder how much pushing the bottom right corner further to the right (on some relevant capabilities) would help (especially as an alternative to pushing up or diagonally), given that sub-human general models don’t seem that safety-favoring but some narrow-but-relevant superhuman capabilities could help us deal with the risks of more general, human-level+ systems.

  3. ^

     Could a narrower set work? E.g. to what extent do we just care about ML engineering?

  4. ^

     although I’m somewhat surprised at the lack of apparent real-world effects

• LizkaFeb 1, 2025, 3:40 PM

  11 points

  4 ∶ 0

  on: The Up­com­ing PEPFAR Cut Will Kill Millions, Many of Them Children

  I don’t think the situation is actually properly resolved by the recent waivers (as @Garrison points out below). See e.g. this thread and the linked ProPublica article. From the thread:

  There is mass confusion and fear, both in and outside government. The aid organizations say they either don’t know how to get a waiver exempting them from Trump’s order or have no idea if/​when theirs might get approved.

  And from the article:

  Despite an announcement earlier this week ostensibly allowing lifesaving operations to continue, those earlier orders have not been rescinded.

  Many groups doing such lifesaving work either don’t know the right way to request an exemption to the order, known as a waiver, or have no sense of where their request stands. They’ve received little information from the U.S. government, where, in recent days, humanitarian officials have been summarily ousted or prohibited from communicating with the aid organizations.

  ...

  By Tuesday evening, Trump and Rubio appeared to heed the international pressure and scale back the order by announcing that any “lifesaving” humanitarian efforts would be allowed to continue.

  Aid groups that specialize in saving lives were relieved and thought their stop-work orders would be reversed just as swiftly as they had arrived.

  But that hasn’t happened. Instead, more stop-work orders have been issued. As of Thursday, contractors worldwide were still grounded under the original orders and unable to secure waivers. Top Trump appointees arrested further funding and banned new projects for at least three months.

  “We need to correct the impression that the waiver was self-executing by virtue of the announcement,” said Marcia Wong, the former deputy assistant administrator of USAID’s humanitarian assistance bureau.

  Aid groups that had already received U.S. money were told they could not spend it or do any previously approved work. The contractors quoted in this article spoke on the condition of anonymity because they feared the administration might prolong their suspension or cancel their contracts completely.

  (Thanks for writing the post! I referenced it in a mini thread on Twitter/​X.)

• LizkaDec 12, 2024, 2:43 PM

  11 points

  4 ∶ 0

  in reply to: titotal’s comment on: Prob­a­bil­ities might be off by one per­centage point

  As a datapoint: despite (already) agreeing to a large extent with this post,^[1] IIRC I answered the question assuming that I do trust the premise.

  ---

  Despite my agreement, I do think there are certain kinds of situations in which we can reasonably use small probabilities. (Related post: Most* small probabilities aren’t pascalian, and maybe also related.)

  ---

  More generally: I remember appreciating some discussion on the kinds of thought experiments that are useful, when, etc. I can’t find it quickly, but possible starting points could be this LW post, Least Convenient Possible World, maybe this post from Richard, and stuff about fictional evidence.

  Writing quickly based on a skim, sorry for lack of clarity/​misinterpretations!

  1. ^

     My view is roughly something like:

     at least in the most obviously analogous situations, it’s very rare that we can properly tell the difference between 1.5% and 0.15% (and so the premise is somewhat absurd)

• LizkaNov 12, 2024, 9:56 PM

  4 points

  0 ∶ 0

  in reply to: Lauren Gilbert’s comment on: The Struc­tural Trans­for­ma­tion Case For Peacekeeping

  That makes sense and is roughly how I was interpreting what you wrote (sorry for potentially implying otherwise in my comment) — this is still a lot more positive on peacekeeping than I was expecting it to be :)