Lizka

Notes on dy­namism, power, & virtue

• Lizka 18 May 2025 14:00 UTC

  6 points

  1 ∶ 0

  in reply to: Lizka’s comment on: Three lower-cost op­tions for run­ning the EA Forum

  Yeah actually I think @Habryka [Deactivated] discusses these kinds of dynamics here: https://​​www.lesswrong.com/​​posts/​​4NFDwQRhHBB2Ad4ZY/​​the-filan-cabinet-podcast-with-oliver-habryka-transcript

  Excerpt (bold mine, Habryka speaking):

  One of the core things that I was always thinking about with LessWrong, and that was my kind of primary analysis of what went wrong with previous LessWrong revivals, was [kind of] an iterated, [the term] “prisoner’s dilemma” is overused, but a bit of an iterated prisoner’s dilemma or something where, like, people needed to have the trust on an ongoing basis that the maintainers and the people who run it will actually stick with it. And there’s a large amount of trust that the people need to have that, if they invest in a site and start writing content on it, that the maintainers and the people who run it actually will put the effort into making that content be shepherded well. And the people who want to shepherd it only want to do that if the maintainers actually...

  And so, one of the key things that I was thinking about, was trying to figure out how to guarantee reliability. This meant, to a lot of the core contributors of the site, I made a promise when I started it, that was basically, I’m going to be making sure that LessWrong is healthy and keeps running for five years from the time I started. Which was a huge commitment—five years is a hugely long time. But my sense at the time was that type of commitment is exactly the most important thing. Because the most usual thing that I get when I talk [in] user interviews to authors and commenters is that they don’t want to contribute because they expect the thing to decline in the future. So reliability was a huge part of that.

  And then I also think, signaling that there was real investment here was definitely a good chunk of it. I think UI is important, and readability of the site is important. And I think I made a lot of improvements there to decide that I’m quite happy with. But I think a lot of it was also just a costly signal that somebody cares.

  I don’t know how I feel about that in retrospect. But I think that was a huge effect, where I think people looked on the site, and when [they] looked at LessWrong 2.0, there was just a very concrete sense that I could see in user interviews that they were like, “Oh, this is a site that is being taken care of. This is a thing that people are paying attention to and that is being kept up well.” In a similar [sense] to how, I don’t know, a clean house has the same symbol. I don’t really know. I think a lot of it was, they were like, wow, a lot of stuff is changing. And the fact that a lot of work is being put into this, the work itself is doing a lot of valuable signaling.

• Lizka 18 May 2025 13:52 UTC

  4 points

  1 ∶ 0

  in reply to: Sarah Cheng’s comment on: Three lower-cost op­tions for run­ning the EA Forum

  Re not necessarily “optimizing” for the Forum, I guess my frame is:

  The Online Team is the current custodian of an important shared resource (the Forum). If the team can’t actually commit to fulfilling its “Forum custodian” duties, e.g. because the priorities of CEA might change, then it should probably start trying to (responsibly) hand that role off to another person/​group.

  (TBC this doesn’t mean that Online should be putting all of its efforts into the Forum, just like a parent has no duty to spend all their energy caring for their child. And it’s not necessarily clear what the bar for responsibly fulfilling Forum custodian duties actually is — maybe moderation and bug fixes are core charges, but “no new engineering work” is fine, I’m not sure.)

  I would view this somewhat differently if it was possible for another group to compete with or step in for CEA /​ the Online Team if it seemed that the team is not investing [enough] in the Forum (or investing poorly). But that’s not actually possible — in fact even if the Online Team stopped ~everything, by default no one else would be able to take over. I’d also feel somewhat differently if the the broader community hadn’t invested so much in the Forum, and if I didn’t think that a baseline ~trust in (and therefore clear commitment from) the team was so important for the Forum’s fate (which I believe for reasons loosely outlined in the memo, IIRC).

  ...

  Btw, I very much agree that staring into the abyss (occasionally) is really useful. And I really appreciate you posting this on the Forum, and also engaging deeply/​openly in the replies.

• Lizka 17 May 2025 16:39 UTC

  9 points

  1 ∶ 0

  on: Three lower-cost op­tions for run­ning the EA Forum

  I think going for Option 2 (“A bulletin board”) or 3 (“Shut down”) would be pretty a serious mistake, fwiw. (I have fewer/​weaker opinions on 1, although I suspect I’m more pessimistic about it than ~most others.)

  ...

  An internal memo I wrote in early 2023 (during my time on the Online Team) seems relevant, so I’ve made a public copy: Vision /​ models /​ principles for the Forum and the Forum+ team^[1]

  I probably no longer believe some of what I wrote there, but still endorse the broad points/​models, which lead me think, among other things:

  1. That people would probably ~stop visiting the Forum if it were closer to a bulletin board

  2. That people would just not relate to the Reddit version in a way that gets remotely close to what the Forum currently does (e.g. I probs believe something in the direction of “support for posting longform content in the subreddit is almost irrelevant; people won’t really be doing it”)

  3. That investment from the Online Team matters in part to maintain ~trust that the Forum will continue existing as the kind of space people want to invest in

  4. That major shifts could be irreversible even when they don’t seem inherently so (e.g. “pausing” the Forum to re-evaluate would be closer to killing it, probably; people would very likely not come back /​ the space would be fundamentally different if they did)

  5. Etc.

  Other notes (adding to what the models I’d described in the memo lead me to believe, or articulating nearby/​more specific versions of the claims I make there, etc.):

  • A platform’s vibe/​form actually matters for the kind of content and discussion it gets

    • I expect a lot less high-effort content would get posted even e.g. if the Forum itself suddenly looked like Reddit (and a lot of that kind of content won’t ever get written)

  • I feel like the discussion here should probably more clearly differentiate between:

    • “What should happen to the Forum” /​ “how good or bad are different futures for the Forum” vs

    • “What should the Online Team do/​ how valuable are various kinds of investment”

    • and maybe also: “How can we cut costs /​ fund this work”

      • (I suspect it might make sense for the Online Team to fundraise from the EA community—although I don’t know if that’s an option given the current setup)

  1. ^

     (also called “Equilibrium curves & being stewards of the special thing”)

     The memo outlines some of how I was thinking about how the Forum works, especially an “equilibrium curves” model and a view that trust is an key thing to track/​build towards. It also discusses the value of the Online Team’s work, theories of change, and when (if ever) “closing” the Forum would make sense and how that could work/​play out.

     (I know Sarah’s read this, but figured I’d share it in case others are interested, and because I’m about to reference stuff from there.)

     Note: At least two fairly important parts of my current models seem missing from the doc (and I suspect I’d think of several more if I thought about it for more than the time it took to skim the doc and write this comment): (1) the Forum as a “two-sided marketplace” (“writers /​ content producers” and “readers”), and (2) creation of (object-level) common knowledge as an important thing the Forum does sometimes.

  What links here?

  • Sarah Cheng's comment on Three lower-cost op­tions for run­ning the EA Forum by Sarah Cheng (22 May 2025 20:34 UTC; 4 points)

• Lizka 2 May 2025 1:38 UTC

  45 points

  1 ∶ 0

  on: Reflec­tions on 7 years build­ing the EA Fo­rum — and mov­ing on

  I just want to say that I’m really, really glad that I got the chance to work with JP, and I think JP’s work has contributed a bunch for the Forum.

  Below I’m sharing some quick misc notes on why that is.

  Meta: sharing a public comment like this on a friend’s post is pretty weird for me! But JP’s work has mostly stayed behind the scenes and has thus gotten significantly less acknowledgement than it deserves, IMO, so I did feel the urge to post this kind of thing publicly. As a compromise between those feelings, y’all are getting the comment along with this meta flag.

  ---

  Misc—not exhaustive! --- notes on some subtle but important ways JP has shaped (my experience on) the Forum (team)

  • Deep models of how things work. Conversations with JP have massively shaped the way I think about the Forum^[1] . My models would have been shallower and less accurate without that. More broadly, I think we would have made worse choices as a team if JP had not been not bringing his models in.

    • Quick list of some specific topics in which my models have been especially obviously shaped^[2] by JP:

      • “acculturation” stuff, the role of trust and how it works, what causes some authors to post, [productivity] & how teams (should) work (lots of good blog posts JP pointed me to!), why the rationality & EA communities have developed the way they have & how do they actually work now, [a bunch of classic EA/​rationalist writing/​models that I hadn’t encountered but have from JP], what it means for someone to be aligned in a certain way, etc.

    • In some cases I felt like “JP energy” was a very strong protective barrier between us and potential Goodharting-style mistakes^[3]

      • Over time I think I was increasingly playing a similar role, too, but could do so largely because I had absorbed a bunch of “JP energy” earlier on (and I felt that we were also helping each other improve our models). It wasn’t just me; IMO the whole team was absorbing & refining these models, and without JP this would have gone much worse.

    • (Some fraction of this “deep models” superpower, I think, can be attributed to the fact that JP had been working on the Forum for a very long time.^[4] To me this kind of thing is strong evidence of the value of sticking around, fwiw; you become a really strong pillar and an incredibly rich source of information. But length-of-Forum-tenure is not the only explanation for this superpower, IMO. (Stuff like “deeply caring”, “willingness to mull”, “being a nerd about systems”, … comes to mind.))

  • Integrity /​ being a compass /​ norm/​culture-setting. JP “set the tone” for a good chunk of how I related to my Forum role. He was also one of the first people I turned to (and found really useful to consult) when I felt conflicted about decisions I was trying to make or about internal (e.g. CEA) dynamics.

    • One illustration, from one of the earliest serious incidents I worked on as a moderator (note that my recollection is fuzzy!):

      • What happened: mods had reason to be suspicious, but we weren’t sure if the user in question had actually done the specific norm-violating thing because the relevant information (e.g. vote patterns or something) was private (even if in theory accessible to the team). The Forum privacy expectations^[5] at the time were more confusing/​unwritten, IIRC, and we didn’t have a standard process for handling this kind of issue. (IIRC) JP kicked off and arranged a (multi-person?) process that got us the key info while ensuring that no one actually saw the relevant private info.

      • I remember being really impressed by this. It can seem like a fairly minor thing — realistically this wasn’t that important, it wasn’t the kind of thing where people would be surprised that this info was accessible to us or expect us to be super careful, etc. But being the kind of person/​team that handles these situations with care/​integrity is really valuable, IMO, and it sets an important team norm. The whole thing left me with a stronger/​more visceral sense of responsibility about how we should handle things.

    • Talking to JP helped me untangle some confusing/​stressful internal team/​CEA/​stakeholder dynamics, and deal with them in ways I endorsed more.^[6]

    • And it was just very energizing to work with someone who loves the Forum and the EA community :)

  • Personal side—helping with development & encouragement, being kind

    • JP is great at noticing how certain habits/​patterns can be improved (and explaining), sharing productivity tips, etc. I gained a lot by bringing up various issues and working through them with him, or by following his advice on tech/​systems. (In fact I’m still taking advantage of this opportunity — highly recommend for those with the privilege.)

    • JP helped me “back myself” & fight back some of the imposter syndrome that I started with.

    • Even when I’d just started at CEA, JP was just really warm & kind to me, which makes a big difference to a recent college graduate who’s just moved to a different country.

  (And obviously there was the whole “running the tech” side of things, and then the whole “running the Online Team” thing, lol… Have I mentioned this list is not exhaustive? This is basically an unfiltered babble of thoughts, but I hope it clarifies some of why I’m so grateful for JP’s work on the Forum.)

  I’m really looking forward to seeing what JP does next.

  1. ^

     and probably also my models of EA, and stuff like “how organizations work”

  2. ^

     (To be clear, it’s not like I was constantly in agreement! Our models often diverged, or we felt different points mattered more or were better frames for understanding things, etc. But this was also very useful.)

  3. ^

     I’m not saying that we were close to hardcore Goodharting or similar, to be clear! Although I do feel like there were some potential naivete-driven mistakes that we avoided because of this kind of thing

  4. ^

     By relevant standards

  5. ^

     i.e. what users should expect about when the Online Team might look at some user info that is normally private

  6. ^

     One quite salient example for me: I felt conflicted and stressed about various Forum/​EV-related things in late 2022/​early 2023 [a] (e.g. I was pretty upset about the EV comms policy). Conversations with JP helped me untangle my feelings and thoughts and figure out what I endorsed doing. In general, having JP on the team — even before he took over from Ben — was really important for me during this time.

     [a] the uptick in Forum engagement must have increased the burden, I suppose

Pod­cast on “AI tools for ex­is­ten­tial se­cu­rity” — transcript

• Lizka 14 Apr 2025 16:58 UTC

  6 points

  0 ∶ 0

  in reply to: Ben_West🔸’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  I think this is a good question, and it’s something I sort of wanted to look into and then didn’t get to! (If you’re interested, I might be able to connect you with someone/​some folks who might know more, though.)

  Quick general takes on what private companies might be able to do to make their tools more useful on this front (please note that I’m pretty out of my depth here, so take this with a decent amount of salt—and also this isn’t meant to be prioritized or exhaustive):

  • Some of the vetting/​authorization processes (e.g. FedRAMP) are burdensome, and sometimes companies do just give up/​don’t bother (see footnote 12), which narrows the options for agencies; going through this anyway could be very useful

  • Generally lowering costs for tech products can make a real difference for whether agencies will adopt them—also maybe open source products are likelier to be used(?) (and there’s probably stuff like which chips are available, which systems staff are used to, what tradeoffs on speed vs cost or similar make sense...)

  • Security is useful/​important, e.g. the tool/​system can be run locally, can be fine-tuned with sensitive data, etc. (Also I expect things like “we can basically prove that the training /​ behavior of this model satisfies [certain conditions]” will increasingly matter — with conditions like “not trained on X data” or “could not have been corrupted by any small group of people” — but my understanding/​thinking here is very vague!)

  • Relatedly I think various properties of the systems’ scaffolding will matter; e.g. “how well the tool fits with existing and future systems” — so modularity and interoperability^[1] (in general and with common systems) are very useful — and “can this tool be set up ~once but allow for different forms of access/​data” (e.g. handle differences in who can see different kinds of data, what info should be logged, etc.)

  (Note also there’s a pretty huge set of consultancies that focus on helping companies sell to the government, but the frame is quite different.)

  And then in terms of ~market gaps, I’m again very unsure, but expect that (unsurprisingly) lower-budget agencies will be especially undersupplied — in particular the DOD has a lot more funding and capacity for this kind of thing — so building things for e.g. NIST could make sense. (Although it might be hard to figure out what would be particularly useful for agencies like NIST without actually being at NIST. I haven’t really thought about this!)

  1. ^

     I haven’t looked into this at all, but given the prevalence of Microsoft systems (Azure etc.) in the US federal government (which afaik is greater than what we see in the UK), I wonder if Microsoft’s relationship with OpenAI explains why we have ChatGPT Gov in the US, while Anthropic is collaborating with the UK government https://​​www.anthropic.com/​​news/​​mou-uk-government

• Lizka 14 Apr 2025 16:18 UTC

  4 points

  0 ∶ 0

  in reply to: OscarD🔸’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  the main question is how high a priority this is, and I am somewhat skeptical it is on the ITN pareto frontier. E.g. I would assume plenty of people care about government efficiency and state capacity generally, and a lot of these interventions are generally about making USG more capable rather than too targeted towards longtermist priorities.

  Agree that “how high-priority should this be” is a key question, and I’m definitely not sure it’s on the ITN pareto frontier! (Nice phrase, btw.)

  Quick notes on some things that raise the importance for me, though:

  1. I agree lots of people care about government efficiency/​ state capacity — but I suspect few of them are seriously considering the possibility of transformative AI in the near future, and I think what you do to ~boost capacity looks pretty different in that world

  2. Also/​relatedly, my worldview means I have extra reasons to care about state capacity, and given my worldview is unusual that means I should expect the world is underinvesting in state capacity (just like most people would love to see a world with fewer respiratory infections, but tracking the possibility of a bioengineered pandemic means I see stuff like far-UVC/​better PPE/​etc. as higher value)

     1. More generally I like the “how much more do I care about X” frame — see this piece from 2014

     2. (It could also be a kind of public good.)

  3. In particular, it seems like a *lot* of the theory of change of AI governance+ relies on competent/​skillful action/​functioning by the US federal government, in periods where AI is starting to radically transform the world (e.g. to mandate testing and be able to tell if that mandate isn’t being followed!), and my sense is that this assumption is fragile/​the govt may very well not actually be sufficiently competent — so we better be working on getting there, or investing more in plans that don’t rely on this assumption

  And I’m pretty worried that a decent amount of work aimed at mitigating the risks of AI could end up net-negative (for its own goals) by not tracking this issue and thus not focusing enough on the interventions that are actually worth pursuing—further harming government AI adoption & competence /​ capacity in the process (e.g. I think some of the OMB/​EO guidance from last year looked positive to me before I dug into this, and now looks negative). So I’d like to nudge some people who work on issues related to existential risk (and government) away from a view like: “all AI is scary/​bad, anything that is ‘pro-AI’ increases existential risk, if this bundle of policies/​barriers inhibits a bunch of different AI things then that’s probably great even if I think only a tiny fraction is truly (existentially) risky”, etc.

  --

  this felt like neither the sort of piece targeted to mainstream US policy folks, nor that convincing for why this should be an EA/​longtermist focus area.

  Totally reasonable reaction IMO. To a large extent I see this as a straightforward flaw of the piece & how I approached it (partly due to lack of time—see my reply to Michael above), although I’ll flag that my main hope was to surface this to people who are in fact kind of in between—e.g. folks at think tanks that do research on existential security and have government experience/​expertise.

  --

  I’m unconvinced that e.g. OP should spin up a grantmaker focused on this (not that you were necessarily recommending this).

  I am in fact not recommending this! (There could be specific interventions in the area that I’d see as worth funding, though, and it’s also related to other clusters where something like the above is reasonable IMO.)

  --

  Also, a few reasons govts may have a better time adopting AI come to mind:

  • Access to large amounts of internal private data

  • Large institutions can better afford one-time upfront costs to train or finetune specialised models, compared to small businesses

  But I agree the opposing reasons you give are probably stronger.

  The data has to be accessible, though, and this is a pretty big problem. See e.g. footnote 17.

  I agree that a major advantage could be that the federal government can in fact move a lot of money when ~it wants to, and could make some (cross-agency/​...) investments into secure models or similar, although my sense is that right now that kind of thing is the exception/​aspiration, not the rule/​standard practice. (Another advantage is that companies do want to maintain good relationships with the government/​admin, and might thus invest more in being useful. Also there are probably a lot of skilled people who are willing to help with this kind of work, for less personal gain.)

  --

  If only this were how USG juggled its priorities!

  🙃 (some decision-makers do, though!)

• Lizka 14 Apr 2025 13:24 UTC

  6 points

  0 ∶ 0

  in reply to: Ozzie Gooen’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  I imagine there might be some very clever strategies to get a lot of the benefits of AI without many of the normal costs of integration.
  
  For example:

  1. The federal government makes heavy use of private contractors. These contractors are faster to adopt innovations like AI.

  2. There are clearly some subsets of the government that matter far more than others. And there are some that are much easier to improve than others.

  3. If AI strategy/​intelligence is cheap enough, most of the critical work can be paid for by donors. For example, we have a situation where there’s a think tank that uses AI to figure out the best strategies/​plans for much of the government, and government officials can choose to pay attention to this.

  I’d be excited to see more work in this direction!

  Quick notes: I think (1) is maybe the default way I expect things to go fine (although I have some worries about worlds where almost all US federal govt AI capacity is via private contractors). (2) seems right, and I’d want someone who has (or can develop) a deeper understanding of this area than me to explore this. Stuff like (3) seems quite useful, although I’m worried about things like ensuring access to the right kind of data and decision-makers (but partnerships /​ a mix of (2) and (3) could help).

  (A lot of this probably falls loosely under “build capacity outside the US federal government” in my framework, but I think the lines are very blurry /​ a lot of the same interventions help with appropriate use/​adoption of AI in the government and external capacity. )

  all very similar to previous thinking on how forecasting can be useful to the government

  I hadn’t thought about this — makes sense, and a useful flag, thank you! (I might dig into this a bit more.)

• Lizka 14 Apr 2025 2:03 UTC

  26 points

  1 ∶ 0

  in reply to: MichaelDickens’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  Thanks for this comment! I don’t view it as “overly critical.”

  Quickly responding (just my POV, not Forethought’s!) to some of what you brought up ---

  (This ended up very long, sorry! TLDR: I agree with some of what you wrote, disagree with some of the other stuff /​ think maybe we’re talking past each other. No need to respond to everything here!)

  A. Motivation behind writing the piece /​ target audience/​ vibe /​ etc.

  Re:

  …it might help me if you explained more about the motivation [behind writing the article] [...] the article reads like you decided the conclusion and then wrote a series of justifications

  I’m personally glad I posted this piece, but not very satisfied with it for a bunch of reasons, one of which is that I don’t think I ever really figured out what the scope/​target audience should be (who I was writing for/​what the piece was trying to do).

  So I agree it might help to quickly write out the rough ~history of the piece:

  • I’d started looking into stuff related to “differential AI development” (DAID), and generally exploring how the timing of different [AI things] relative to each other could matter.

  • My main focus quickly became exploring ~safety-increasing AI applications/​tools — Owen and I recently posted about this (see the link).

  • But I also kept coming back to a frame of “oh crap, who is using AI how much/​how significantly is gonna matter an increasing amount as time goes on. I expect adoption will be quite uneven — e.g. AI companies will be leading the way — and some groups (whose actions/​ability to make reasonable decisions we care about a lot) will be left behind.”

    • At the time I was thinking about this in terms of “differential AI development and diffusion”

  • IIRC I soon started thinking about governments here; I had the sense that government decision-makers were generally slow on tech use, and I was also using “which types of AI applications will not be properly incentivized by the market” as a way to think about which AI applications might be easier to speed up. (I think we mentioned this here.)

  • This ended up taking me on a mini deep dive on government adoption of AI, which in turn increasingly left me with the impression that (e.g.) the US federal government would either (1) become increasingly overtaken from within by an unusually AI-capable group (or e.g. the DOD), (2) be rendered increasingly irrelevant, leaving (US) AI companies to regulate themselves and likely worsening its ability to deal with other issues, or (3) somehow in fact adopt AI, but likely in a chaotic way that would be especially dangerous (because things would go slowly until a crisis forced a ~war-like undertaking).

  • I ended up poking around in this for a while, mostly as an aside to my main DAID work, feeling like I should probably scope this out and move on. (The ~original DAID memos I’d shared with people discussed government AI adoption.)

  • After a couple of rounds of drafts+feedback I got into a “I should really publish some version of this that I believe and seems useful and then get back to other stuff; I don’t think I’m the right person to work a lot more on this but I’m hoping other people in the space will pick up whatever is correct here and push it forward” mode—and ended up sharing this piece.

  In particular I don’t expect (and wasn’t expecting) that ~policymakers will read this, but hope it’s useful for people at relevant think tanks or similar who have more government experience/​knowledge but might not be paying attention to one “side” of this issue or the other. (For instance, I think a decent fraction of people worried about existential risks from advanced AI don’t really think about how using AI might be important for navigating those risks, partly because all of AI kinda gets lumped together).

  Quick responses to some other things in your comment that seem kinda related to what I’m responding to in this “motivation/​vibe/​…” cluster:

  I also found it odd that the report did not talk about extinction risk. In its list of potential catastrophic outcomes, the final item on the list was “Human disempowerment by advanced AI”, which IMO is an overly euphemistic way of saying “AI will kill everyone”.

  We might have notably different worldviews here (to be clear mine is pretty fuzzy!). For one thing, in my view many of the scary “AI disempowerment” outcomes might not in fact look immediately like “AI kills everyone” (although to be clear that is in fact an outcome I’m very worried about), and unpacking what I mean by “disempowerment” in the piece (or trying to find the ideal way to say it) didn’t seem productive—IIRC I wrote something and moved on. I also want to be clear that rogue AI [disempowering] humans is not the only danger I’m worried about, i.e. it doesn’t dominate everything else for me—the list you’re quoting from wasn’t an attempt to mask AI takeover, but rather a sketch of the kind of thing I’m thinking about. (Note: I do remember moving that item down the list at some point when I was working on a draft, but IIRC this was because I wanted to start with something narrower to communicate the main point, not because I wanted to de-emphasize ~AI takeover.)

  By my reading, this article is meant to be the sort of Very Serious Report That Serious People Take Seriously, which is why it avoids talking about x-risk.

  I might be failing to notice my bias, but I basically disagree here—although I do feel a different version of what you’re maybe pointing to here (see next para). I was expecting that basically anyone who reads the piece will already have engaged at least a bit with “AI might kill all humans”, and likely most of the relevant audience will have thought very deeply about this and in fact has this as a major concern. I also don’t personally feel shy about saying that I think this might happen — although again I definitely don’t want to imply that I think this is overwhelmingly likely to happen or the only thing that matters, because that’s just not what I believe.

  However I did occasionally feel like I was ~LARPing research writing when I was trying to articulate my thoughts, and suspect some of that never got resolved! (And I think I floundered a bit on where to go with the piece when getting contradicting feedback from different people—although ultimately the feedback was very useful.) In my view this mostly shows up in other ways, though. (Related—I really appreciated Joe Carlsmith’s recent post on fake thinking and real thinking when trying to untangle myself here.)

  B. Downside risks of the proposed changes

  1. Making policymakers “more reluctant to place restrictions on AI development...”

     1. I did try to discuss this a bit in the “Government adoption of AI will need to manage important risks” section (and sort of in the “3. Defusing the time bomb of rushed automation” section), and indeed it’s a thing I’m worried about.

     2. I think ultimately my view is that without use of AI in government settings, stuff like AI governance will just be ineffective or fall to private actors anyway, and also that the willingness-to-regulate /​undue influence dynamics will be much worse if the government has no in-house capacity or is working with only one AI company as a provider.

  2. Shortening timelines by increasing AI company revenue

     1. I think this isn’t a major factor here—the govt is a big customer in some areas, but the private sector dominates (as does investment in the form of grants, IIRC)

  3. “The government does not always work in the interests of the people (in fact it frequently works against them!) so making the government more effective/​powerful is not pure upside.”
     

     1. I agree with this, and somewhat worry about it. IIRC I have a footnote on this somewhere -—I decided to scope this out. Ultimately my view right now is that the alternative (~no governance at least in the US, etc.) is worse. Sort of relatedly, I find the “narrow corridor” a useful frame here—see e.g. here.)

  C. Is gov competence actually a bottleneck?

  I don’t think government competence is what’s holding us back from having good AI regulations, it’s government willingness. I don’t see how integrating AI into government workflow will improve AI safety regulations (which is ultimately the point, right?[^1]), and my guess is on balance it would make AI regulations less likely to happen because policy-makers will become more attached to their AI systems and won’t want to restrict them.

  My view is that you need both, we’re not on track for competence, and we should be pretty uncertain about what happens on the willingness side.

  D. Michael’s line item responses

  1.

  > invest in AI and technical talent

  What does that mean exactly? I can’t think of how you could do that without shortening timelines so I don’t know what you have in mind here.

  I’m realizing this can be read as “invest in AI and in technical talent” — I meant “invest in AI talent and (broader) technical talent (in govt).” I’m not sure if this just addresses the comment; my guess is that doing this might have a tiny shortening effect on timelines (but is somewhat unclear, partly because in some cases e.g. raising salaries for AI roles in govt might draw people away from frontier AI companies), but this is unlikely to be the decisive factor. (Maybe related: my view is that generally this kind of thing should be weighed instead of treated as a reason to entirely discard certain kinds of interventions.)

  2.

  > Streamline procurement processes for AI products and related tech

  I also don’t understand this. Procurement by whom, for what purpose? And again, how does this not shorten timelines? (Broadly speaking, more widespread use of AI shortens timelines at least a little bit by increasing demand.)

  I was specifically talking about agencies’ procurement of AI products — e.g. say the DOE wants a system that makes forecasting demand easier or whatever; making it easier for them to actually get such a system faster. I think the effect on timelines will likely be fairly small here (but am not sure), and currently think it would be outweighed by the benefits.

  3.

  > Gradual adoption is significantly safer than a rapid scale-up.

  This sounds plausible but I am not convinced that it’s true, and the article presents no evidence, only speculation. I would like to see more rigorous arguments for and against this position instead of taking it for granted.

  I’d be excited to see more analysis on this, but it’s one of the points I personally am more confident about (and I will probably not dive in right now).

  4.

  > And in a crisis — e.g. after a conspicuous failure, or a jump in the salience of AI adoption for the administration in power — agencies might cut corners and have less time for security measures, testing, in-house development, etc.

  This line seems confused. Why would a conspicuous failure make government agencies want to suddenly start using the AI system that just conspicuously failed? Seems like this line is more talking about regulating AI than adopting AI, whereas the rest of the article is talking about adopting AI.

  Sorry, again my writing here was probably unclear; the scenarios I was picturing were more like:

  • There’s a serious breach—US govt systems get hacked (again) by [foreign nation, maybe using AI] - revealing that they’re even weaker than is currently understood, or publicly embarrassing the admin. The admin pushes for fast modernization on this front.

  • A flashy project isn’t proceeding as desired (especially as things are ramping up), the admin in power is ~upset with the lack of progress, pushes

  • There’s a successful violent attack (e.g. terrorism); turns out [agency] was acting too slowly...

  • Etc.

  Not sure if that answers the question/​confusion?

  5.

  > Frontier AI development will probably concentrate, leaving the government with less bargaining power.

  I don’t think that’s how that works. Government gets to make laws. Frontier AI companies don’t get to make laws. This is only true if you’re talking about an AI company that controls an AI so powerful that it can overthrow the government, and if that’s what you’re talking about then I believe that would require thinking about things in a very different way than how this article presents them.

  This section is trying to argue that AI adoption will be riskier later on, so the “bargaining power” I was talking about here is the bargaining power of the US federal govt (or of federal agencies) as a customer; the companies it’s buying from will have more leverage if they’re effectively monopolies. My understanding is that there are already situations where the US govt has limited negotiation power and maybe even makes policy concessions to specific companies specifically because of its relationship to those companies — e.g. in defense (Lockheed Martin, etc., although this is also kinda complicated) and again maybe Microsoft.

  And: would adopting AI (i.e. paying frontier companies so government employees can use their products) reduce the concentration of power? Wouldn’t it do the opposite?

  Again, the section was specifically trying to argue that later adoption is scarier than earlier adoption (in this case because there are (still) several frontier AI companies). But I do think that building up internal AI capacity, especially talent, would reduce the leverage any specific AI company has over the US federal government.

  6.

  > It’s natural to focus on the broad question of whether we should speed up or slow down government AI adoption. But this framing is both oversimplified and impractical

  Up to this point, the article was primarily talking about how we should speed up government AI adoption. But now it’s saying that’s not a good framing? So why did the article use that framing? I get the sense that you didn’t intend to use that framing, but it comes across as if you’re using it.

  Yeah, I don’t think I navigated this well! (And I think I was partly talking ti myself here.) But maybe my “motivation” notes above give some context?
  In terms of the specific “position” I in practice leaned into: Part of why I led with the benefits of AI adoption was the sense that the ~existential risk community (which is most of my audience) generally focuses on risks of AI adoption/​use/​products, and that’s where my view diverges more. There’s also been more discussion, from an existential risk POV, of the risks of adoption than there has been of the benefits, so I didn’t feel that elaborating too much on the risks would be as useful.

  7.

  > Hire and retain technical talent, including by raising salaries

  I would like to see more justification for why this is a good idea. The obvious upside is that people who better understand AI can write more useful regulations. On the other hand, empirically, it seems that people with more technical expertise (like ML engineers) are on average less in favor of regulations and more in favor of accelerating AI development (shortening timelines, although they usually don’t think “timelines” are a thing). So arguably we should have fewer such people in positions of government power.

  The TLDR of my view here is something like “without more internal AI/​technical talent (most of) the government will be slower on using AI to improve its work & stay relevant, which I think is bad, and also it will be increasingly reliant on external people/​groups/​capacity for technical expertise—e.g. relying on external evals, or trusting external advice on what policy options make sense, etc. and this is bad.”

  8.

  > Explore legal or other ways to avoid extreme concentration in the frontier AI market

  [...]

  The linked article attached to this quote says “It’s very unclear whether centralizing would be good or bad”, but you’re citing it as if it definitively finds centralization to be bad.

  (The linked article is this one: https://​​www.forethought.org/​​research/​​should-there-be-just-one-western-agi-project )

  I was linking to this to point to relevant discussion, not as a justification for a strong claim like “centralization is definitively bad”—sorry for being unclear!

  9.

  > If the US government never ramps up AI adoption, it may be unable to properly respond to existential challenges.

  What does AI adoption have to do with the ability to respond to existential challenges? It seems to me that once AI is powerful enough to pose an existential threat, then it doesn’t really matter whether the US government is using AI internally.

  I suspect we may have fairly different underlying worldviews here, but maybe a core underlying belief on my end is that there are things that it’s helpful for the government to do before we get to ~ASI, and also there will be AI tools pre ~ASI that are very helpful for doing those things. (Or an alt framing: the world will get ~/​fast/​complicated/​weird due to AI before there’s nothing the US gov could in theory do to make things go better.)

  10.

  > Map out scenarios in which AI safety regulation is ineffective and explore potential strategies

  I don’t think any mapping is necessary. Right now AI safety regulation is ineffective in every scenario, because there are no AI safety regulations (by safety I mean notkilleveryoneism). Trivially, regulations that don’t exist are ineffective. Which is one reason why IMO the emphasis of this article is somewhat missing the mark—right now the priority should be to get any sort of safety regulations at all.

  I fairly strongly disagree here (with “the priority should be to get any sort of safety regulations at all”) but don’t have time to get into it, really sorry!

  ---
  

  Finally, thanks a bunch for saying that you enjoyed some of my earlier writing & I changed your thinking on slow vs quick mistakes! That kind of thing is always lovely to hear.

  (Posted on my phone— sorry for typos and similar!)

The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

• Lizka 18 Mar 2025 1:18 UTC

  15 points

  1 ∶ 0

  on: Lizka’s Shortform

  Quick (visual) note on something that seems like a confusion in the current conversation:

  

  
  Others have noted similar things (eg, and Will’s earlier take on total vs human extinction). You might disagree with the model (curious if so!), but I’m a bit worried that one way or another people are talking past each other (least from skimming the discussion).
  
  (Commenting via phone, sorry for typos or similar!)

• Lizka 17 Mar 2025 16:59 UTC

  8 points

  0 ∶ 0

  on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  What actually changes about what you’d work on if you concluded that improving the future is more important on the current margin than trying to reduce the chance of (total) extinction (or vice versa)?

  Curious for takes from anyone!

• Lizka 14 Mar 2025 19:32 UTC

  15 points

  0 ∶ 0

  on: AI Tools for Ex­is­ten­tial Security

  I wrote a Twitter thread that summarizes this piece and has a lot of extra images (I probably went overboard, tbh.)

  I kinda wish I’d included the following image in the piece itself, so I figured I’d share it here:

  

AI Tools for Ex­is­ten­tial Security

• Lizka 17 Feb 2025 13:29 UTC

  9 points

  1 ∶ 0

  on: Fake think­ing and real thinking

  I just want to say that I really appreciated this post — it came at exactly the right time for me and I’ve referenced it several times since you shared it.

Some mod­er­a­tion team principles

• Lizka 4 Feb 2025 13:42 UTC

  4 points

  0 ∶ 0

  in reply to: Lizka’s comment on: Lizka’s Shortform

  Follow-up:

  Quick list of some ways benchmarks might be (accidentally) misleading^[1]

  1. Poor “construct validity”^[2]( & systems that are optimized for the metric)

     1. The connection between what the benchmark is measuring and what it’s trying to measure (or what people think it’s measuring) is broken. In particular:

     2. Missing critical steps

        1. When benchmarks are trying to evaluate progress on some broad capability (like “engineering” or “math ability” or “planning”), they’re often testing specific performance on meaningfully simpler tasks. Performance on those tasks might be missing key aspects of true/​real-world/​relevant progress on that capability.

     3. Besides the inherent difficulty of measuring the right thing, it’s important to keep in mind that systems might have been trained specifically to perform well on a given benchmark.

        1. This is probably a bigger problem for benchmarks that have gotten more coverage.

     4. And some benchmarks have been designed specifically to be challenging for existing leading models, which can make new/​other AI systems appear to have made more progress on these capabilities (relative to the older models) than they actually are.

        1. We’re seeing this with the “Humanity’s Last Exam” benchmark.

     5. ...and sometimes some of the apparent limitations are random or kinda fake, such that a minor improvement appears to lead to radical progress

  2. Discontinuous metrics: (partial) progress on a given benchmark might be misleading.

     1. The difficulty of tasks/​tests in a benchmark often varies significantly (often for good reason), and reporting of results might explain the benchmark by focusing on its most difficult tests instead of the ones that the model actually completed.

        1. I think this was an issue for Frontier Math, although I’m not sure how much strongly to discount some of the results as a result.

     2. -> This (along with issues like 1b above, which can lead to saturation) is part of what makes it harder to extrapolate from e.g. plots of progress on certain benchmarks.

  3. Noise & poisoning of the metric: Even on the metric in question, data might have leaked into the training process, the measurement process itself can be easily affected by things like who’s running it, comparing performance of different models that were tested slightly differently might be messy, etc. Some specific issues here (I might try to add links later):

     1. Brittleness to question phrasing

     2. Data contamination (discussed e.g. here)

     3. Differences in how the measurement/​testing process was actually set up (including how much inference compute was used, how many tries a model was given, access to tools, etc.)

  4. Misc

     1. Selective reporting/​measurement (AI companies want to report their successes)

     2. Tasks that appear difficult (because they’re hard for humans) might be might be especially easier for AI systems (and vice versa) — and this might cause us to think that more (or less) progress is being made than is true

        1. E.g. protein folding is really hard for humans

        2. This looks relevant, but I haven’t read it

     3. Some benchmarks seem mostly irrelevant to what I care about

     4. Systems are tested pre post-training enhancements or other changes

  Additions are welcome! (Also, I couldn’t quickly find a list like this earlier, but I’d be surprised if a better version than what I have above wasn’t available somewhere; I’d love recommendations.)

  Open Phil’s announcement of their now-closed benchmarking RFP has some useful notes on this, particularly the section on “what makes for a strong benchmark.”I also appreciated METR’s list of desiderata here.

  1. ^

     To be clear: I’m not trying to say anything on ways benchmarks might be useful/​harmful here. And I’m really not an expert.

  2. ^

     This paper looks relevant but I haven’t read it.

• Lizka 3 Feb 2025 22:26 UTC

  28 points

  3 ∶ 0

  on: Lizka’s Shortform

  TLDR: Notes on confusions about what we should do about digital minds, even if our assessments of their moral relevance are correct^[1]

  I often feel quite lost when I try to think about how we can “get digital minds right.” It feels like there’s a variety of major pitfalls involved, whether or not we’re right about the moral relevance of some digital minds.

  Digital-minds-related pitfalls in different situations
  Reality ➡️ Our perception ⬇️	These digital minds are (non-trivially) morally relevant[2]	These digital minds are not morally relevant
  We see these digital minds as morally relevant	(1) We’re right But we might still fail to understand how to respond, or collectively fail to implement that response.	(2) We’re wrong We confuse ourselves, waste an enormous amount of resources on this[3] (potentially sacrificing the welfare of other beings that do matter morally), and potentially make it harder to respond to the needs of other digital minds in the future (see also).
  We don’t see these digital minds as morally relevant	(3) We’re wrong The default result here seems to be moral catastrophe through ignorance/​sleepwalking (although maybe we’ll get lucky).	(4) We’re right All is fine (at least on this front).

  Categories (2) and (3) above are ultimately about being confused about which digital minds matter morally — having an inappropriate level of concern, one way or another. A lot of the current research on digital minds seems to be aimed at avoiding this issue. (See e.g. and e.g..) I’m really glad to see this work; the pitfalls in these categories worry me a lot.

  But even if we end up in category (1) — we realize correctly that certain digital minds are morally relevant — how can we actually figure out what we should do? Understanding how to respond probably involves answering questions like:

  • What matters for these digital minds? What improves their experience? (How can we tell?)

  • Is their experience overall negative/​positive? Should we be developing such minds? (How can we tell??) (How can we ethically design digital minds?)

  • Should we respect certain rights for these digital minds? (..?)

  • ??

  Answering these questions seem really difficult.

  In many cases, extrapolating from what we can tell about humans seems inappropriate.^[4] (Do digital minds systems find joy or meaning in some activities? Do they care about survival? What does it mean for a digital mind/​AI system to be in pain? Is it ok to lie to AI systems? Is it ok to design AI systems that have no ~goals besides fulfilling requests from humans?)

  And even concepts we have for thinking about what matters to humans (or animals) often seem ill-suited for helping us with digital minds.^[5] (When does it make sense to talk about freedoms and rights,^[6] or the sets of (relevant) capabilities a digital mind has, or even their preferences? What even is the right “individual” to consider? Self-reports seem somewhat promising, but when can we actually rely on them as signals about what’s important for digital minds?)

  I’m also a bit worried that too much of the work is going towards the question of “which systems/​digital minds are morally relevant” vs to the question of “what do we do if we think that a system is morally relevant (or if we’re unsure)?” (Note however that I’ve read a tiny fraction of this work, and haven’t worked in the space myself!) E.g. this paper focuses on the former and closes by recommending that companies prepare to make thoughtful decisions about how to treat the AI systems identified as potentially morally significant by (a) hiring or appointing a DRI (directly responsible individual) for AI welfare and (b) developing certain kinds of frameworks for AI welfare oversight. These steps do seem quite good to me (at least naively), but — as the paper explicitly acknowledges — they’re definitely not sufficient.

  Some of the work that’s apparently focused at the question of “which digital minds matter morally” involves working towards theories of ~consciousness, and maybe that will also help us with the latter question. But I’m not sure.

  (So my quite uninformed independent impression is that it might be worth investing a bit more in trying to figure out what we should do if we do decide that some digital minds might be morally relevant, or maybe what we should do if we find that we’re making extremely little progress on figuring out whether they are.)

  ---

  These seem like hard problems/​questions, but I want to avoid defeatism.

  I appreciated @rgb‘s closing remark in this post (bold mine):

  To be sure, our neuroscience tools are way less powerful than we would like, and we know far less about the brain than we would like. To be sure, our conceptual frameworks for thinking about sentience seem shaky and open to revision. Even so, trying to actually solve the problem by constructing computational theories which try to explain the full range of phenomena could pay significant dividends. My attitude towards the science of consciousness is similar to Derek Parfit’s attitude towards ethics: since we have only just begun the attempt, we can be optimistic.

  ---

  Some of the other content I’ve read/​skimmed feels like it’s pointing in useful directions on these fronts (and more recommendations are welcome!):

  • Improving the Welfare of AIs: A Nearcasted Proposal—Ryan Greenblatt on LW

  • To understand AI sentience, first understand it in animals—Jonathan Birch in Aeon (I’ve only skimmed this, though)

  • ​​Jeff Sebo on digital minds, and how to avoid sleepwalking into a major moral catastrophe − 80,000 Hours (I’ve only skimmed this, again)

  1. ^

     I’m continuing my quick take spree, with a big caveat that I’m not a philosopher and haven’t read nearly as much research/​writing on digital minds as I want to.

     And I’m not representing Forethought here! I don’t know what Forethought folks think about what I’m writing here.

  2. ^

     The more appropriate term, I think, is “moral patienthood.” And we probably care about the degree of moral consideration that is merited in different cases.

  3. ^

     This paper on “Sharing the world with digital minds” mentions this as one of the many failure modes — note that I’ve only skimmed it.

  4. ^

     I’m often struck by how appropriate the War with the Newts is as an analogy/​illustration/​prediction for a bunch of what I’m seeing today, including on issues related to digital minds. But there’s one major way in which the titular newts differ significantly from potential morally relevant digital minds; the newts are quite similar to humans and can, to some extent, communicate their preferences in ways the book’s humans could understand if they were listening.

     (One potential exception: there’s a tragic-but-brilliant moment in which “The League for the Protection of Salamanders” calls women to action sewing modest skirts and aprons for the Newts in order to appease the Newts’ supposed sense of propriety.)

  5. ^

     This list of propositions touches on related questions/​ideas.

  6. ^

     See e.g. here:

     Whether instrumentally-justified or independently binding, the rights that some AI systems could be entitled to might be different from the rights that humans are entitled to. This could be because, instrumentally, a different set of rights for AI systems promotes welfare. For example, as noted by Shulman and Bostrom (2021), naively granting both “reproductive” rights and voting rights to AI systems would have foreseeably untenable results for existing democratic systems: if AI systems can copy themselves at will, and every copy gets a vote, then elections could be won via tactical copying. This set of rights would not promote welfare and uphold institutions in the same way that they do for humans. Or AI rights could differ because, independently of instrumental considerations, their different properties entitle them to different rights—analogously to how children and animals are plausibly entitled to different rights than adults.

• Lizka 2 Feb 2025 23:11 UTC

  5 points

  0 ∶ 0

  in reply to: NickLaing’s comment on: Lizka’s Shortform

  Thanks for saying this!

  I’m trying to get back into the habit of posting more content, and aiming for a quick take makes it easier for me to get over perfectionism or other hurdles (or avoid spending more time on this kind of thing than I endorse). But I’ll take this as a nudge to consider sharing things as posts more often. :)