Lizka

• Lizka 8 Dec 2025 13:16 UTC

  34 points

  2 ∶ 0

  on: Lizka’s Shortform

  When thinking about the impacts of AI, I’ve found it useful to distinguish between different reasons for why automation in some area might be slow. In brief:

  1. raw performance issues

  2. trust bottlenecks

  3. intrinsic premiums for “the human factor”

  4. adoption lag

  5. motivated/​active protectionism towards humans

  I’m posting this mainly because I’ve wanted to link to this a few times now when discussing questions like “how should we update on the shape of AI diffusion based on...?”. Not sure how helpful it will be on its own!

  ---

  In a bit more detail:

  (1) Raw performance issues

  There’s a task that I want an AI system to do. An AI system might be able to do it in the future, but the ones we have today just can’t do it.

  For instance:

  • AI systems still struggle to stay coherent over long contexts, so it’s often hard to use an AI system to build out a massive codebase without human help, or write a very consistent detective novel.

  • Or I want an employee who’s more independent; we can get aligned on some goals and they will push them forward, coming up with novel pathways, etc.

  • (Other capability gaps: creativity/​novelty, need for complicated physical labor, …)

  A subclass here might be performance issues that are downstream of “interface mismatch”.^[1] Cases where AI might be good enough at some fundamental task that we’re thinking of (e.g. summarizing content, or similar), but where the systems that surround that task or the interface through which we’re running the thing — which are trivial for humans — is a very poor fit for existing AI, and AI systems struggle to get around that.^[2] (E.g. if the core concept is presented via a diagram, or requires computer use stuff.) In some other cases, we might separately consider whether the AI systems has the right affordances at all.

  This is what we often think about when we think about the AI tech tree /​ AI capabilities. Others are often important, though:

  (2) Verification & trust bottlenecks

  The AI system might be able to do the task, but I can’t easily check its work and prefer to just do it myself or rely on someone I trust. Or I can’t be confident the AI won’t fail spectacularly in some rare but important edge cases (in ways a human almost certainly wouldn’t).^[3]

  For instance, maybe I want someone to pull out the most surprising and important bits from some data, and I can’t trust the judgement of an AI system the way I can trust someone I’ve worked with. Or I don’t want to use a chatbot in customer-facing stuff in case someone finds a way to make it go haywire.

  A subclass here is when one can’t trust AI providers (and open-source/​on-device models aren’t good enough) for some use case. Accountability sinks also play a role on the “trust” front. Using AI for some task might complicate the question of who bears responsibility when things go wrong, which might matter if accountability is load-bearing for that system. In this case we might go for assigning a human overseer.^[4]

  (3) Intrinsic premiums for “the human factor”^[5]

  The task *requires* human involvement, or I intrinsically prefer it for some reason.

  E.g. AI therapists are less effective because knowing that the person on the other end is a person is actually useful to get me to do my exercises. Or: I might pay more to see a performance by Yo Yo Ma than by a robot because that’s my true preference; I get less value from a robot performance.

  A subclass here is cases where the value I’m getting is testimony — e.g. if I want to gather user data or understand someone’s internal experience — that only humans can provide (even if AI can superficially simulate it).

  (4) Adoption lag & institutional inertia

  AI systems can do this, people would generally prefer that, there aren’t legal or similar “hard barriers” to AI use, etc., but in practice this is being done by humans.

  E.g. maybe AI-powered medical research isn’t happening because the people/​institutions who could be setting this kind of automation up just haven’t gotten to it yet.

  Adoption lags might be caused by stuff like: sheer laziness, coordination costs, lack of awareness/​expertise, attention scarcity — the humans currently involved in this process don’t have enough slack — or similar, active (but potentially hidden) incumbent resistance, or maybe bureaucratic dysfunction (no one has the right affordances).

  (My piece on the adoption gap between the US government and ~industry is largely about this.)

  (5) Motivated/​active protectionism towards humans

  AI systems that can do this, there’s no intrinsic need for human involvement, nor real capabilities/​trust bottlenecks getting in the way. But we’ll deliberately continue relying on human labor for ~political reasons — not just because we’re moving slowly.

  E.g. maybe we’ve hard-coded a requirement that a human lawyer or teacher or driver (etc.) is involved. A particularly salient subclass/​set of examples here is when a group of humans has successfully lobbied a government to require human labor (where it might be blatantly obvious that it’s not needed). In other cases, the law (or anatomy of an institution) might incidentally require humans in the loop via some other requirement.

  ---

  This is a low-res breakdown that might be missing stuff. And the lines between these categories can be very fuzzy. For instance, verification difficulties (2) can provide justification for protectionism (5) or further slow down adoption (4).

  But I still think it’s useful to pay attention to what’s really at play, and worry we too often think exclusively in terms of raw performance issues (1) with a bit of diffusion lag (4).

  1. ^

     Note OTOH that sometimes the AI capabilities are already there, but bad UI or lack of some complementary tech is still making those capabilities unusable.

  2. ^

     I’m listing this as a “raw performance issue” because the AI systems/​tools will probably keep improving to better deal with such clashes. But I also expect the surrounding interfaces/​systems to change as people try to get more value from available AI capabilities. (E.g. stuff like robots.txt.)

  3. ^

     Sometimes the edge cases are the whole point. See also: Heuristics That Almost Always Work—by Scott Alexander

  4. ^

     Although I guess then we should be careful about alert fatigue/​false security issues. IIRC this episode discusses related stuff: Machine learning meets malware, with Caleb Fenton (search for “alert fatigue” /​ “hypnosis”)

  5. ^

     (There’s also the opposite; cases where we’d actively prefer humans not be involved. For instance, all else equal, I might want to keep some information private — not want to share it with another person — even if I’d happily share with an AI system.)

• Lizka 15 Oct 2025 3:49 UTC

  6 points

  1 ∶ 0

  in reply to: OscarD🔸’s comment on: A per­sonal take on why you should work at Forethought (maybe)

  Yeah, I guess I don’t want to say that it’d be better if the team had people who are (already) strongly attached to various specific perspectives (like the “AI as a normal technology” worldview—maybe especially that one?^[1]). And I agree that having shared foundations is useful /​ constantly relitigating foundational issues would be frustrating. I also really do think the points I listed under “who I think would be a good fit” — willingness to try on and ditch conceptual models, high openness without losing track of taste, & flexibility — matter, and probably clash somewhat with central examples of “person attached to a specific perspective.”

  = rambly comment, written quickly, sorry! =

  But in my opinion we should not just all (always) be going off of some central AI-safety-style worldviews. And I think that some of the divergence I would like to see more of could go pretty deep—e.g. possibly somewhere in the grey area between what you listed as “basic prerequisites” and “particular topics like AI timelines...”. (As one example, I think accepting terminology or the way people in this space normally talk about stuff like “alignment” or “an AI” might basically bake in a bunch of assumptions that I would like Forethought’s work to not always rely on.)

  One way to get closer to that might be to just defer less or more carefully, maybe. And another is to have a team that includes people who better understand rarer-in-this-space perspectives, which diverge earlier on (or people who are by default inclined to thinking about this stuff in ways that are different from others’ defaults), as this could help us start noticing assumptions we didn’t even realize we were making, translate between frames, etc.

  So maybe my view is that (1) there were more ~independent worldview formation/​ exploration going on, and that (2) the (soft) deferral that is happening (because some deferral feels basically inevitable) were less overlapping.

  (I expect we don’t really disagree, but still hope this helps to clarify things. And also, people at Forethought might still disagree with me.)

  1. ^

     In particular:

     If this perspective involves a strong belief that AI will not change the world much, then IMO that’s just one of the (few?) things that are ~fully out of scope for Forethought. I.e. my guess is that projects with that as a foundational assumption wouldn’t really make much sense to do here. (Although IMO even if, say, I believed that this conclusion was likely right, I might nevertheless be a good fit for Forethought if I were willing to view my work as a bet on the worlds in which AI is transformative.)

     But I don’t really remember what the “AI as normal..” position is, and could imagine that it’s somewhat different — e.g. more in the direction of “automation is the wrong frame for understanding the most likely scenarios” /​ something like this. In that case my take would be that someone exploring this at Forethought could make sense (haven’t thought about this one much), and generally being willing to consider this perspective at least seems good, but I’d still be less excited about people who’d come with the explicit goal of pursuing that worldview & no intention of updating or whatever.

     --

     (Obviously if the “AI will not be a big deal” view is correct, I’d want us to be able to come to that conclusion—and change Forethught’s mission or something. So I wouldn’t e.g. avoid interacting with this view or its proponents, and agree that e.g. inviting people with this POV as visitors could be great.)

• Lizka 14 Oct 2025 23:07 UTC

  9 points

  1 ∶ 0

  in reply to: cb’s comment on: A per­sonal take on why you should work at Forethought (maybe)

  ore Quick sketch of what I mean (and again I think others at Forethought may disagree with me):

  • I think most of the work that gets done at Forethought builds primarily on top of conceptual models that are at least in significant part ~deferring to a fairly narrow cluster of AI worldviews/​paradigms (maybe roughly in the direction of what Joe Carlsmith/​Buck/​Ryan have written about)

    • (To be clear, I think this probably doesn’t cover everyone, and even when it does, there’s also work that does this more/​less, and some explicit poking at these worldviews, etc.)

    • So in general I think I’d prefer the AI worldviews/​deferral to be less ~correlated. On the deferral point — I’m a bit worried that the resulting aggregate models don’t hold together well sometimes, e.g. because it’s hard to realize you’re not actually on board with some further-in-the-background assumptions being made for the conceptual models you want to pull in.

    • And I’d also just like to see more work based on other worldviews, even just ones that complicate the paradigmatic scenarios /​ try to unpack the abstractions to see if some of the stuff that’s been simplified away is blinding us to important complications (or possibilities).^[1]

  • I think people at Forethought often have some underlying ~welfarist frames & intuitions — and maybe more generally a tendency to model a bunch of things via something like utility functions — as opposed to thinking via frames more towards the virtue/​rights/​ethical-relationships/​patterns/​… direction (or in terms of e.g. complex systems)

    • Quick links illustrating some stuff on what I’m trying to gesture at might include this & this

  • We’re not doing e.g. formal empirical experiments, I think (although that could also be fine given this area of work, just listing as it pops into my head)

  • There’s some fuzzy pattern in the core ways in which most(?) people at Forethought seem to naturally think, or in how they prefer to have research discussions, IMO? I notice that the types of conversations (and collaborations) I have with some other people go fairly differently, and this leads me to different places.

    • To roughly gesture at this, in my experience Forethought tends broadly more towards a mode like “read/​think → write a reasonably coherent doc → get comments from various people → write new docs /​ sometimes discuss in Slack or at whiteboards (but often with a particular topic in mind...)”, I think? (Vs things like trying to map more stuff out/​think through really fuzzy thoughts in active collaboration, having some long topical-but-not-too-structured conversations that end up somewhere unexpected, “let’s try to make a bunch of predictions in a row to rapidly iterate on our models,” etc.)

    • (But this probably varies a decent amount between people, and might be context-dependent, etc.)

  • IIRC many people have more expertise on stuff like analytic philosophy, maybe something like econ-style modeling, and the EA flavor of generalist research than e.g. ML/​physics/​cognitive science/​whatever, or maybe hands-on policy/​industry/​tech work, or e.g. history/​culture… And there are various similarities in people’s cultural/​social backgrounds. (I don’t actually remember everyone’s backgrounds, though, and think it’s easy to overindex on this /​ weigh it too heavily. But I’d be surprised if that doesn’t affect things somewhat.)

  I also want to caveat that:

  • (i) I’m not trying to be exhaustive here, just listing what’s salient, and

  • (ii) in general it’ll be harder for me to name things that also strongly describe me (although I’m trying, to some degree), especially as I’m just quickly listing stuff and not thinking too hard.

  (And thanks for the nice meta note!)

  1. ^

     I’ve been struggling to articulate this well, but I’ve recently been feeling like, for instance, proposals on making deals with “early [potential] schemers” implicitly(?) rely on a bunch of assumptions about the anatomy of AI entities we’d get at relevant stages.

     More generally I’ve been feeling pretty iffy about using game-theoretic reasoning about “AIs” (as in “they’ll be incentivized to...” or similar) because I sort of expect it to fail in ways that are somewhat similar to what one gets if one tries to do this with states or large bureaucracies or something—iirc the fourth paper here discussed this kind of thing, although in general there’s a lot of content on this. Similar stuff on e.g. reasoning about the “goals” etc. of AI entities at different points in time without clarifying a bunch of background assumptions (related, iirc).

A per­sonal take on why you should work at Forethought (maybe)

• Lizka 31 Aug 2025 15:52 UTC

  4 points

  0 ∶ 0

  in reply to: Lizka’s comment on: Lizka’s Shortform

  Ah, @Gregory Lewis🔸 says some of the above better. Quoting his comment:

  • [...]

  • So ~everything is ultimately an S-curve. Yet although ‘this trend will start capping out somewhere’ is a very safe bet, ‘calling the inflection point’ before you’ve passed it is known to be extremely hard. Sigmoid curves in their early days are essentially indistinguishable from exponential ones, and the extra parameter which ~guarantees they can better (over?)fit the points on the graph than a simple exponential give very unstable estimates of the putative ceiling the trend will ‘cap out’ at. (cf. 1, 2.)

  • Many important things turn on (e.g.) ‘scaling is hitting the wall ~now’ vs. ‘scaling will hit the wall roughly at the point of the first dyson sphere data center’ As the universe is a small place on a log scale, this range is easily spanned by different analysis choices on how you project forward.

  • Without strong priors on ‘inflecting soon’ vs. ‘inflecting late’, forecasts tend to be volatile: is this small blip above or below trend really a blip, or a sign we’re entering a faster/​slow regime?

  • [...]

• Lizka 31 Aug 2025 15:49 UTC

  2 points

  0 ∶ 0

  in reply to: Eli Rose🔸’s comment on: Lizka’s Shortform

  I tried to clarify things a bit in this reply to titotal: https://​​forum.effectivealtruism.org/​​posts/​​iJSYZJJrLMigJsBeK/​​lizka-s-shortform?commentId=uewYatQz4dxJPXPiv

  In particular, I’m not trying to make a strong claim about exponentials specifically, or that things will line up perfectly, etc.

  (Fwiw, though, it does seem possible that if we zoom out, recent/​near-term population growth slow-downs might be functionally a ~blip if humanity or something like it leaves the Earth. Although at some point you’d still hit physical limits.)

• Lizka 31 Aug 2025 15:44 UTC

  6 points

  0 ∶ 0

  in reply to: titotal’s comment on: Lizka’s Shortform

  Oh, apologies: I’m not actually trying to claim that things will be <<exactly.. exponential>>. We should expect some amount of ~variation in progress/​growth (these are rough models, we shouldn’t be too confident about how things will go, etc.), what’s actually going on is (probably a lot) more complicated than a simple/​neat progression of new s-curves, etc.

  The thing I’m trying to say is more like:

  • When we’ve observed some datapoints about a thing we care about, and they seem to fit some overall curve (e.g. exponential growth) reasonably well,

  • then pointing to specific drivers that we think are responsible for the changes — & focusing on how those drivers might progress or be fundamentally limited, etc. — often makes us (significantly) overestimate bottlenecks/​obstacles standing in the way of progress on the thing that we actually care about.

  • And placing some weight on the prediction that the curve will simply continue^[1] seems like a useful heuristic /​ counterbalance (and has performed well).

  (Apologies if what I’d written earlier was unclear about what I believe — I’m not sure if we still notably disagree given the clarification?)

  ---

  A different way to think about this might be something like:

  • The drivers that we can point to are generally only part of the picture, and they’re often downstream of some fuzzier higher-level/​”meta” force (or a portfolio of forces) like “incentives+...”

    • It’s usually quite hard to draw a boundary around literally everything that’s causing some growth/​progress

    • It’s also often hard to imagine, from a given point in time, very different ways of driving the thing forward

    • (e.g. because we’ve yet to discover other ways of making progress, because proxies we’re looking at locally implicitly bake in some unnecessary assumptions about how progress on the thing we care about will get made, etc.)

  • So our stories about what’s causing some development that we’re observing are often missing important stuff, and sometimes we should trust the extrapolation more than the stories /​ assume the stories are incomplete

  Something like this seems to help explain why views like “the curve we’re observing will (basically) just continue” have seemed surprisingly successful, even when the people holding those “curve go up” views justified their conclusions via apparently incorrect reasoning about the specific drivers of progress. (And so IMO people should place non-trivial weight on stuff like “rough, somewhat naive-seeming extrapolation of the general trends we’re observing^[2].”^[3])

  [See also a classic post on the general topic, and some related discussion here, IIRC: https://​​www.alignmentforum.org/​​posts/​​aNAFrGbzXddQBMDqh/​​moore-s-law-ai-and-the-pace-of-progress ]

  1. ^

     Caveat: I’d add ”...on a big range/​ the scale we care about”; at some point, ~any progress would start hitting ~physical limits. But if that point is after the curve reshapes ~everything we care about, then I’m basically ignoring that consideration for now.

  2. ^

     Obviously there are caveats. E.g.:

     - the metrics we use for such observations can lead us astray in some situations (in particular they might not ~linearly relate to “the true thing we care about”)

     - we often have limited data, we shouldn’t be confident that we’re predicting/​measuring the right thing, things can in fact change over time and we should also not forget that, etc.

     (I think there were nice notes on this here, although I’ve only skimmed and didn’t re-read https://​​arxiv.org/​​pdf/​​2205.15011 )

  3. ^

     Also, sometimes we do know what

  What links here?

  • Lizka's comment on Lizka’s Quick takes by Lizka (31 Aug 2025 15:49 UTC; 2 points)

• Lizka 31 Aug 2025 15:18 UTC

  21 points

  4 ∶ 0

  on: The ‘com­mu­nity’ tag is problematic

  Replying quickly, speaking only for myself^[1]:

  1. I agree that the boundary between Community & non-Community posts is (and has always been) fuzzy

     1. You can see some guidance I wrote in early 2023: Community vs. other^[2] (a “test-yourself” quiz is also linked in the doc)

        1. Note: I am no longer on the Online Team/​an active moderator and don’t actually know what the official guidance is today.^[3]

     2. I also agree that this fuzziness is not a trivial issue, & it continues to bug me

        1. when I go to the Frontpage, I relatively frequently see posts that I think should be “Community” that aren’t, or vice versa

  2. However, the Community tag/​distinction/​section was not “a way to separate out FTX-scandal-related posts”, IMO

     1. Fwiw, I was (also?) really worried about this change when I was working on it, both because I was concerned it would suppress useful criticism/​reflection/​etc., and because I wanted to make sure my decision-making wasn’t somehow secretly tainted by various biases.

        1. I’m writing quickly here — not taking time to dig up my old docs or trying to really remember what was going on in early 2023^[4], but I vaguely remember e.g. calling @Ben_West🔸 at some point when I was feeling particularly stressed, to double check that we endorsed how we were approaching the change

        2. See also footnote 7 here, and footnote 2 here

     2. More:

        1. A lot of the thinking/​earlier attempts had happened before the FTX collapse

           1. “We’ve been hearing about the Forum fixating on certain discussions for over a year [as of Feb 2023]: this has been making a lot of people sad for a long time. [...]”

              1. [from here]

           2. And we’d already down-weighted “community” posts for new & logged-out users (before the FTX collapse); IMO this was always a pretty hacky feature that had a bunch of flaws

           3. The community tag itself had existed for a while before that

        2. And, quoting from myself here (from July 2023):

           1. Just a quick clarification: I don’t think this was a “change to lower community engagement.” Adding the community section was a change, and it did (probably^[1]) lower engagement with community posts, but that wasn’t the actual point (which is a distinction I think is worth making, although maybe some would say it’s the same). In my view, the point was to avoid suffocating other discussions and to make the Forum feel less on-edge/​anxiety-inducing (which we were hearing was at least some people’s experience). In case it helps, this outlines our (or at least my) thinking about it.

     3. (i.e. +1 to @Lorenzo Buonanno🔸 in the comments)

  3. Quick takes on where things are today:

     1. I would like to see more “Community” posts /​ discussions

     2. I think the Community section overall is probably still (quite) positive (but tagging could probably be improved)

        1. Off the top of the head, the two key things I’d probably look at /​think about, if I were considering removing it (or doing something else), are (1) whether it seems like it’s really tanking engagement on Community posts, or preventing them from being written (I’d maybe chat to likely/​potential authors about this, look at the numbers, etc.), and (2) how active users (i.e. readers) feel about the Community section^[5]

  1. ^

     I.e. I’m not speaking for the Online/​Mod teams here, and didn’t run this comment by anyone.

  2. ^

     (I vaguely remember making and linking a public version of this doc somewhere at some point, but couldn’t quickly find that.)

  3. ^

     In fact it looks like I can no longer add or remove the Community tag from posts. I’m still in the Slack; a few people sometimes flag questions about edge cases there.

  4. ^

     (...a lot!)

  5. ^

     I’d guess that most people still prefer it. The change got a lot of support, both before and after (and this is rare).

• Lizka 20 Aug 2025 14:10 UTC

  10 points

  3 ∶ 0

  on: Lizka’s Shortform

  I sometimes see people say stuff like:

  Those forecasts were misguided. If they ended up with good answers, that’s accidental; the trends they extrapolated from have hit limits… (Skeptics get Bayes points.)

  But IMO it’s not a fluke that the “that curve is going up, who knows why” POV has done well.

  A sketch of what I think happens:

  

  There’s a general dynamic here that goes something like:

  1. Some people point to a curve going up (and maybe note some underlying driver)

  2. Others point out that the drivers have inherent constraints (this is an s-curve, not an exponential)

  And then in *some* sense the bottlenecks crowd turn out to be right (the specific drivers/​paradigm peters out, there’s literally no more space for more transistors, companies run low on easily accessible/​high quality training data, etc.)…

  …but then a “surprise new thing” pops up and fills the gap, such that the “true” thing we cared about (whether or not it’s what we were originally measuring) *does* actually continue as people originally predicted, apparently naively

  (and it turned out that the curve consists of a stack of s-curves..)

  We can go too far with this kind of reasoning; some “true things we care about” (e.g. spread of a disease) * are* in fact s-curves, bounded, etc., and only locally look like ~exponentials. (So: no, we shouldn’t expect the baby to weigh trillions of pounds by age 10...)

  But I think the more granular, gears-oriented view — which considers how long specific drivers of the progress we’re seeing could continue, etc. — often underrates the extent to which *other forces* can (and often do) jump in when earlier drivers lose momentum.

  (~Crossposting from Twitter.)

  ---

  “The Bypass Principle: How AI flows around obstacles” from Eric Drexler is a very related (and IMO good) post. Quote (bold mine):

  While shallow assessments focus on visible obstacles — the difficulties of matching human capabilities, of overcoming regulatory barriers, and of restructuring organizations — AI-enabled developments will often find paths that bypass rather than overcome apparent barriers. Existing obstacles are concrete and obvious in a way that alternatives are not. Skewed judgment follows.

  ---

  (This stuff isn’t new; many people have pointed out these kinds of dynamics. But I feel like I’m still seeing them a fair bit — and this came up recently — so I wanted to write this note.)

• Lizka 20 Aug 2025 13:39 UTC

  4 points

  0 ∶ 0

  in reply to: OllieBase’s comment on: ITN 201: pit­falls in ITN BOTECs

  Yeah, this sort of thing is partly why I tend to feel better about BOTECs like (writing very quickly, tbc!):

  What could we actually accomplish if we (e.g.) doubled (the total stock/​ flow of) investment in ~technical AIS work (specifically the stuff focused on catastrophic risks, in this general worldview)? (you could broaden if you wanted to, obviously)

  Well, let’s see:

  • That might look like:

    • adding maybe ~400(??) FTEs similar (in ~aggregate) to the folks working here now, distributed roughly in proportion to current efforts /​ profiles — plus the funding/​AIS-specific infrastructure (e.g. institutional homes) needed to accomodate them

    • E.g. across intent alignment stuff, interpretability, evals, AI control, ~safeguarded AI, AI-for-AIS, etc., across non-profit/​private/​govt (but in fact aimed at loss of control stuff).

  • How good would this be?

    • Maybe (per year of doubling) we’d then get something like a similar-ish value from this as we don from a year of current space (or something like 2x less, if we want to eyeball diminishing returns)

    • Then maybe we can look at what this space has accomplished in the past year and see how much we’d pay for that /​ how valuable that seems...

  • (What other ~costs might we be missing here?)

  You might also decide that you have much better intuitions for how much we’d accomplish (and how valuable that’d be) on a different scale (e.g. adding one project like Redwood/​Goodfire/​Safeguarded AI/​..., i.e. more like 30 FTEs than 400 — although you’d probably want to account for considerations like “for each ‘successful’ project we’d likely need to invest in a bunch of attempts/​ surrounding infrastructure...”), or intuitions about what amount of investment is required to get to some particular desired outcome...

  Or if you took the more ITN-style approach, you could try to approach the BOTEC via something like (1) how much investment has there been so far in this broad ~POV /​ porftolio, (2 (option a)) how much value/​progress has this portfolio made + something like “how much has been made in the second half?” (to get a sense of how much we’re facing diminishing returns at the moment — fwiw without thinking too much about it I think “not super diminishing returns at the mo”), or (2 (option b)) what fraction of the overall “AI safety problem” is “this-sort-of-safety-work-affectable” (i.e. something like “if we scaled up this kind of work — and only this kind of work — to an insane degree, how much of the problem will be fixed?”) + how big/​important the problem is overall… Etc. (Again, for all of this my main question is often “what are the sources of signal or taste /​ heuristics /​ etc. that you’re happier basing your estimates on?)

• Lizka 20 Aug 2025 13:09 UTC

  2 points

  0 ∶ 0

  in reply to: JamesÖz 🔸’s comment on: ITN 201: pit­falls in ITN BOTECs

  Thank you! I used Procreate for these (on an iPad).^[1]

  (I also love Excalidraw for quick diagrams, have used & liked Whimsical before, and now also semi-grudgingly appreciate Canva.)

  ---

  Relatedly, I wrote a quick summary of the post in a Twitter thread a few days ago and added two extra sketches there. Posting here too in case anyone finds them useful:

  

  

  1. ^

     (And a meme generator for the memes.)

ITN 201: pit­falls in ITN BOTECs

A shal­low re­view of what trans­for­ma­tive AI means for an­i­mal welfare

Notes on dy­namism, power, & virtue

• Lizka 18 May 2025 14:00 UTC

  6 points

  1 ∶ 0

  in reply to: Lizka’s comment on: Three lower-cost op­tions for run­ning the EA Forum

  Yeah actually I think @Habryka [Deactivated] discusses these kinds of dynamics here: https://​​www.lesswrong.com/​​posts/​​4NFDwQRhHBB2Ad4ZY/​​the-filan-cabinet-podcast-with-oliver-habryka-transcript

  Excerpt (bold mine, Habryka speaking):

  One of the core things that I was always thinking about with LessWrong, and that was my kind of primary analysis of what went wrong with previous LessWrong revivals, was [kind of] an iterated, [the term] “prisoner’s dilemma” is overused, but a bit of an iterated prisoner’s dilemma or something where, like, people needed to have the trust on an ongoing basis that the maintainers and the people who run it will actually stick with it. And there’s a large amount of trust that the people need to have that, if they invest in a site and start writing content on it, that the maintainers and the people who run it actually will put the effort into making that content be shepherded well. And the people who want to shepherd it only want to do that if the maintainers actually...

  And so, one of the key things that I was thinking about, was trying to figure out how to guarantee reliability. This meant, to a lot of the core contributors of the site, I made a promise when I started it, that was basically, I’m going to be making sure that LessWrong is healthy and keeps running for five years from the time I started. Which was a huge commitment—five years is a hugely long time. But my sense at the time was that type of commitment is exactly the most important thing. Because the most usual thing that I get when I talk [in] user interviews to authors and commenters is that they don’t want to contribute because they expect the thing to decline in the future. So reliability was a huge part of that.

  And then I also think, signaling that there was real investment here was definitely a good chunk of it. I think UI is important, and readability of the site is important. And I think I made a lot of improvements there to decide that I’m quite happy with. But I think a lot of it was also just a costly signal that somebody cares.

  I don’t know how I feel about that in retrospect. But I think that was a huge effect, where I think people looked on the site, and when [they] looked at LessWrong 2.0, there was just a very concrete sense that I could see in user interviews that they were like, “Oh, this is a site that is being taken care of. This is a thing that people are paying attention to and that is being kept up well.” In a similar [sense] to how, I don’t know, a clean house has the same symbol. I don’t really know. I think a lot of it was, they were like, wow, a lot of stuff is changing. And the fact that a lot of work is being put into this, the work itself is doing a lot of valuable signaling.

• Lizka 18 May 2025 13:52 UTC

  4 points

  1 ∶ 0

  in reply to: Sarah Cheng 🔸’s comment on: Three lower-cost op­tions for run­ning the EA Forum

  Re not necessarily “optimizing” for the Forum, I guess my frame is:

  The Online Team is the current custodian of an important shared resource (the Forum). If the team can’t actually commit to fulfilling its “Forum custodian” duties, e.g. because the priorities of CEA might change, then it should probably start trying to (responsibly) hand that role off to another person/​group.

  (TBC this doesn’t mean that Online should be putting all of its efforts into the Forum, just like a parent has no duty to spend all their energy caring for their child. And it’s not necessarily clear what the bar for responsibly fulfilling Forum custodian duties actually is — maybe moderation and bug fixes are core charges, but “no new engineering work” is fine, I’m not sure.)

  I would view this somewhat differently if it was possible for another group to compete with or step in for CEA /​ the Online Team if it seemed that the team is not investing [enough] in the Forum (or investing poorly). But that’s not actually possible — in fact even if the Online Team stopped ~everything, by default no one else would be able to take over. I’d also feel somewhat differently if the the broader community hadn’t invested so much in the Forum, and if I didn’t think that a baseline ~trust in (and therefore clear commitment from) the team was so important for the Forum’s fate (which I believe for reasons loosely outlined in the memo, IIRC).

  ...

  Btw, I very much agree that staring into the abyss (occasionally) is really useful. And I really appreciate you posting this on the Forum, and also engaging deeply/​openly in the replies.

• Lizka 17 May 2025 16:39 UTC

  9 points

  1 ∶ 0

  on: Three lower-cost op­tions for run­ning the EA Forum

  I think going for Option 2 (“A bulletin board”) or 3 (“Shut down”) would be pretty a serious mistake, fwiw. (I have fewer/​weaker opinions on 1, although I suspect I’m more pessimistic about it than ~most others.)

  ...

  An internal memo I wrote in early 2023 (during my time on the Online Team) seems relevant, so I’ve made a public copy: Vision /​ models /​ principles for the Forum and the Forum+ team^[1]

  I probably no longer believe some of what I wrote there, but still endorse the broad points/​models, which lead me think, among other things:

  1. That people would probably ~stop visiting the Forum if it were closer to a bulletin board

  2. That people would just not relate to the Reddit version in a way that gets remotely close to what the Forum currently does (e.g. I probs believe something in the direction of “support for posting longform content in the subreddit is almost irrelevant; people won’t really be doing it”)

  3. That investment from the Online Team matters in part to maintain ~trust that the Forum will continue existing as the kind of space people want to invest in

  4. That major shifts could be irreversible even when they don’t seem inherently so (e.g. “pausing” the Forum to re-evaluate would be closer to killing it, probably; people would very likely not come back /​ the space would be fundamentally different if they did)

  5. Etc.

  Other notes (adding to what the models I’d described in the memo lead me to believe, or articulating nearby/​more specific versions of the claims I make there, etc.):

  • A platform’s vibe/​form actually matters for the kind of content and discussion it gets

    • I expect a lot less high-effort content would get posted even e.g. if the Forum itself suddenly looked like Reddit (and a lot of that kind of content won’t ever get written)

  • I feel like the discussion here should probably more clearly differentiate between:

    • “What should happen to the Forum” /​ “how good or bad are different futures for the Forum” vs

    • “What should the Online Team do/​ how valuable are various kinds of investment”

    • and maybe also: “How can we cut costs /​ fund this work”

      • (I suspect it might make sense for the Online Team to fundraise from the EA community—although I don’t know if that’s an option given the current setup)

  1. ^

     (also called “Equilibrium curves & being stewards of the special thing”)

     The memo outlines some of how I was thinking about how the Forum works, especially an “equilibrium curves” model and a view that trust is an key thing to track/​build towards. It also discusses the value of the Online Team’s work, theories of change, and when (if ever) “closing” the Forum would make sense and how that could work/​play out.

     (I know Sarah’s read this, but figured I’d share it in case others are interested, and because I’m about to reference stuff from there.)

     Note: At least two fairly important parts of my current models seem missing from the doc (and I suspect I’d think of several more if I thought about it for more than the time it took to skim the doc and write this comment): (1) the Forum as a “two-sided marketplace” (“writers /​ content producers” and “readers”), and (2) creation of (object-level) common knowledge as an important thing the Forum does sometimes.

  What links here?

  • Sarah Cheng 🔸's comment on Three lower-cost op­tions for run­ning the EA Forum by Sarah Cheng 🔸 (22 May 2025 20:34 UTC; 4 points)

• Lizka 2 May 2025 1:38 UTC

  45 points

  1 ∶ 0

  on: Reflec­tions on 7 years build­ing the EA Fo­rum — and mov­ing on

  I just want to say that I’m really, really glad that I got the chance to work with JP, and I think JP’s work has contributed a bunch for the Forum.

  Below I’m sharing some quick misc notes on why that is.

  Meta: sharing a public comment like this on a friend’s post is pretty weird for me! But JP’s work has mostly stayed behind the scenes and has thus gotten significantly less acknowledgement than it deserves, IMO, so I did feel the urge to post this kind of thing publicly. As a compromise between those feelings, y’all are getting the comment along with this meta flag.

  ---

  Misc—not exhaustive! --- notes on some subtle but important ways JP has shaped (my experience on) the Forum (team)

  • Deep models of how things work. Conversations with JP have massively shaped the way I think about the Forum^[1] . My models would have been shallower and less accurate without that. More broadly, I think we would have made worse choices as a team if JP had not been not bringing his models in.

    • Quick list of some specific topics in which my models have been especially obviously shaped^[2] by JP:

      • “acculturation” stuff, the role of trust and how it works, what causes some authors to post, [productivity] & how teams (should) work (lots of good blog posts JP pointed me to!), why the rationality & EA communities have developed the way they have & how do they actually work now, [a bunch of classic EA/​rationalist writing/​models that I hadn’t encountered but have from JP], what it means for someone to be aligned in a certain way, etc.

    • In some cases I felt like “JP energy” was a very strong protective barrier between us and potential Goodharting-style mistakes^[3]

      • Over time I think I was increasingly playing a similar role, too, but could do so largely because I had absorbed a bunch of “JP energy” earlier on (and I felt that we were also helping each other improve our models). It wasn’t just me; IMO the whole team was absorbing & refining these models, and without JP this would have gone much worse.

    • (Some fraction of this “deep models” superpower, I think, can be attributed to the fact that JP had been working on the Forum for a very long time.^[4] To me this kind of thing is strong evidence of the value of sticking around, fwiw; you become a really strong pillar and an incredibly rich source of information. But length-of-Forum-tenure is not the only explanation for this superpower, IMO. (Stuff like “deeply caring”, “willingness to mull”, “being a nerd about systems”, … comes to mind.))

  • Integrity /​ being a compass /​ norm/​culture-setting. JP “set the tone” for a good chunk of how I related to my Forum role. He was also one of the first people I turned to (and found really useful to consult) when I felt conflicted about decisions I was trying to make or about internal (e.g. CEA) dynamics.

    • One illustration, from one of the earliest serious incidents I worked on as a moderator (note that my recollection is fuzzy!):

      • What happened: mods had reason to be suspicious, but we weren’t sure if the user in question had actually done the specific norm-violating thing because the relevant information (e.g. vote patterns or something) was private (even if in theory accessible to the team). The Forum privacy expectations^[5] at the time were more confusing/​unwritten, IIRC, and we didn’t have a standard process for handling this kind of issue. (IIRC) JP kicked off and arranged a (multi-person?) process that got us the key info while ensuring that no one actually saw the relevant private info.

      • I remember being really impressed by this. It can seem like a fairly minor thing — realistically this wasn’t that important, it wasn’t the kind of thing where people would be surprised that this info was accessible to us or expect us to be super careful, etc. But being the kind of person/​team that handles these situations with care/​integrity is really valuable, IMO, and it sets an important team norm. The whole thing left me with a stronger/​more visceral sense of responsibility about how we should handle things.

    • Talking to JP helped me untangle some confusing/​stressful internal team/​CEA/​stakeholder dynamics, and deal with them in ways I endorsed more.^[6]

    • And it was just very energizing to work with someone who loves the Forum and the EA community :)

  • Personal side—helping with development & encouragement, being kind

    • JP is great at noticing how certain habits/​patterns can be improved (and explaining), sharing productivity tips, etc. I gained a lot by bringing up various issues and working through them with him, or by following his advice on tech/​systems. (In fact I’m still taking advantage of this opportunity — highly recommend for those with the privilege.)

    • JP helped me “back myself” & fight back some of the imposter syndrome that I started with.

    • Even when I’d just started at CEA, JP was just really warm & kind to me, which makes a big difference to a recent college graduate who’s just moved to a different country.

  (And obviously there was the whole “running the tech” side of things, and then the whole “running the Online Team” thing, lol… Have I mentioned this list is not exhaustive? This is basically an unfiltered babble of thoughts, but I hope it clarifies some of why I’m so grateful for JP’s work on the Forum.)

  I’m really looking forward to seeing what JP does next.

  1. ^

     and probably also my models of EA, and stuff like “how organizations work”

  2. ^

     (To be clear, it’s not like I was constantly in agreement! Our models often diverged, or we felt different points mattered more or were better frames for understanding things, etc. But this was also very useful.)

  3. ^

     I’m not saying that we were close to hardcore Goodharting or similar, to be clear! Although I do feel like there were some potential naivete-driven mistakes that we avoided because of this kind of thing

  4. ^

     By relevant standards

  5. ^

     i.e. what users should expect about when the Online Team might look at some user info that is normally private

  6. ^

     One quite salient example for me: I felt conflicted and stressed about various Forum/​EV-related things in late 2022/​early 2023 [a] (e.g. I was pretty upset about the EV comms policy). Conversations with JP helped me untangle my feelings and thoughts and figure out what I endorsed doing. In general, having JP on the team — even before he took over from Ben — was really important for me during this time.

     [a] the uptick in Forum engagement must have increased the burden, I suppose

Pod­cast on “AI tools for ex­is­ten­tial se­cu­rity” — transcript

• Lizka 14 Apr 2025 16:58 UTC

  6 points

  0 ∶ 0

  in reply to: Ben_West🔸’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  I think this is a good question, and it’s something I sort of wanted to look into and then didn’t get to! (If you’re interested, I might be able to connect you with someone/​some folks who might know more, though.)

  Quick general takes on what private companies might be able to do to make their tools more useful on this front (please note that I’m pretty out of my depth here, so take this with a decent amount of salt—and also this isn’t meant to be prioritized or exhaustive):

  • Some of the vetting/​authorization processes (e.g. FedRAMP) are burdensome, and sometimes companies do just give up/​don’t bother (see footnote 12), which narrows the options for agencies; going through this anyway could be very useful

  • Generally lowering costs for tech products can make a real difference for whether agencies will adopt them—also maybe open source products are likelier to be used(?) (and there’s probably stuff like which chips are available, which systems staff are used to, what tradeoffs on speed vs cost or similar make sense...)

  • Security is useful/​important, e.g. the tool/​system can be run locally, can be fine-tuned with sensitive data, etc. (Also I expect things like “we can basically prove that the training /​ behavior of this model satisfies [certain conditions]” will increasingly matter — with conditions like “not trained on X data” or “could not have been corrupted by any small group of people” — but my understanding/​thinking here is very vague!)

  • Relatedly I think various properties of the systems’ scaffolding will matter; e.g. “how well the tool fits with existing and future systems” — so modularity and interoperability^[1] (in general and with common systems) are very useful — and “can this tool be set up ~once but allow for different forms of access/​data” (e.g. handle differences in who can see different kinds of data, what info should be logged, etc.)

  (Note also there’s a pretty huge set of consultancies that focus on helping companies sell to the government, but the frame is quite different.)

  And then in terms of ~market gaps, I’m again very unsure, but expect that (unsurprisingly) lower-budget agencies will be especially undersupplied — in particular the DOD has a lot more funding and capacity for this kind of thing — so building things for e.g. NIST could make sense. (Although it might be hard to figure out what would be particularly useful for agencies like NIST without actually being at NIST. I haven’t really thought about this!)

  1. ^

     I haven’t looked into this at all, but given the prevalence of Microsoft systems (Azure etc.) in the US federal government (which afaik is greater than what we see in the UK), I wonder if Microsoft’s relationship with OpenAI explains why we have ChatGPT Gov in the US, while Anthropic is collaborating with the UK government https://​​www.anthropic.com/​​news/​​mou-uk-government