Lizka

• Lizka 15 Oct 2025 3:49 UTC

  6 points

  1 ∶ 0

  in reply to: OscarD🔸’s comment on: A per­sonal take on why you should work at Forethought (maybe)

  Yeah, I guess I don’t want to say that it’d be better if the team had people who are (already) strongly attached to various specific perspectives (like the “AI as a normal technology” worldview—maybe especially that one?^[1]). And I agree that having shared foundations is useful /​ constantly relitigating foundational issues would be frustrating. I also really do think the points I listed under “who I think would be a good fit” — willingness to try on and ditch conceptual models, high openness without losing track of taste, & flexibility — matter, and probably clash somewhat with central examples of “person attached to a specific perspective.”

  = rambly comment, written quickly, sorry! =

  But in my opinion we should not just all (always) be going off of some central AI-safety-style worldviews. And I think that some of the divergence I would like to see more of could go pretty deep—e.g. possibly somewhere in the grey area between what you listed as “basic prerequisites” and “particular topics like AI timelines...”. (As one example, I think accepting terminology or the way people in this space normally talk about stuff like “alignment” or “an AI” might basically bake in a bunch of assumptions that I would like Forethought’s work to not always rely on.)

  One way to get closer to that might be to just defer less or more carefully, maybe. And another is to have a team that includes people who better understand rarer-in-this-space perspectives, which diverge earlier on (or people who are by default inclined to thinking about this stuff in ways that are different from others’ defaults), as this could help us start noticing assumptions we didn’t even realize we were making, translate between frames, etc.

  So maybe my view is that (1) there were more ~independent worldview formation/​ exploration going on, and that (2) the (soft) deferral that is happening (because some deferral feels basically inevitable) were less overlapping.

  (I expect we don’t really disagree, but still hope this helps to clarify things. And also, people at Forethought might still disagree with me.)

  1. ^

     In particular:

     If this perspective involves a strong belief that AI will not change the world much, then IMO that’s just one of the (few?) things that are ~fully out of scope for Forethought. I.e. my guess is that projects with that as a foundational assumption wouldn’t really make much sense to do here. (Although IMO even if, say, I believed that this conclusion was likely right, I might nevertheless be a good fit for Forethought if I were willing to view my work as a bet on the worlds in which AI is transformative.)

     But I don’t really remember what the “AI as normal..” position is, and could imagine that it’s somewhat different — e.g. more in the direction of “automation is the wrong frame for understanding the most likely scenarios” /​ something like this. In that case my take would be that someone exploring this at Forethought could make sense (haven’t thought about this one much), and generally being willing to consider this perspective at least seems good, but I’d still be less excited about people who’d come with the explicit goal of pursuing that worldview & no intention of updating or whatever.

     --

     (Obviously if the “AI will not be a big deal” view is correct, I’d want us to be able to come to that conclusion—and change Forethught’s mission or something. So I wouldn’t e.g. avoid interacting with this view or its proponents, and agree that e.g. inviting people with this POV as visitors could be great.)

• Lizka 14 Oct 2025 23:07 UTC

  9 points

  1 ∶ 0

  in reply to: cb’s comment on: A per­sonal take on why you should work at Forethought (maybe)

  ore Quick sketch of what I mean (and again I think others at Forethought may disagree with me):

  • I think most of the work that gets done at Forethought builds primarily on top of conceptual models that are at least in significant part ~deferring to a fairly narrow cluster of AI worldviews/​paradigms (maybe roughly in the direction of what Joe Carlsmith/​Buck/​Ryan have written about)

    • (To be clear, I think this probably doesn’t cover everyone, and even when it does, there’s also work that does this more/​less, and some explicit poking at these worldviews, etc.)

    • So in general I think I’d prefer the AI worldviews/​deferral to be less ~correlated. On the deferral point — I’m a bit worried that the resulting aggregate models don’t hold together well sometimes, e.g. because it’s hard to realize you’re not actually on board with some further-in-the-background assumptions being made for the conceptual models you want to pull in.

    • And I’d also just like to see more work based on other worldviews, even just ones that complicate the paradigmatic scenarios /​ try to unpack the abstractions to see if some of the stuff that’s been simplified away is blinding us to important complications (or possibilities).^[1]

  • I think people at Forethought often have some underlying ~welfarist frames & intuitions — and maybe more generally a tendency to model a bunch of things via something like utility functions — as opposed to thinking via frames more towards the virtue/​rights/​ethical-relationships/​patterns/​… direction (or in terms of e.g. complex systems)

    • Quick links illustrating some stuff on what I’m trying to gesture at might include this & this

  • We’re not doing e.g. formal empirical experiments, I think (although that could also be fine given this area of work, just listing as it pops into my head)

  • There’s some fuzzy pattern in the core ways in which most(?) people at Forethought seem to naturally think, or in how they prefer to have research discussions, IMO? I notice that the types of conversations (and collaborations) I have with some other people go fairly differently, and this leads me to different places.

    • To roughly gesture at this, in my experience Forethought tends broadly more towards a mode like “read/​think → write a reasonably coherent doc → get comments from various people → write new docs /​ sometimes discuss in Slack or at whiteboards (but often with a particular topic in mind...)”, I think? (Vs things like trying to map more stuff out/​think through really fuzzy thoughts in active collaboration, having some long topical-but-not-too-structured conversations that end up somewhere unexpected, “let’s try to make a bunch of predictions in a row to rapidly iterate on our models,” etc.)

    • (But this probably varies a decent amount between people, and might be context-dependent, etc.)

  • IIRC many people have more expertise on stuff like analytic philosophy, maybe something like econ-style modeling, and the EA flavor of generalist research than e.g. ML/​physics/​cognitive science/​whatever, or maybe hands-on policy/​industry/​tech work, or e.g. history/​culture… And there are various similarities in people’s cultural/​social backgrounds. (I don’t actually remember everyone’s backgrounds, though, and think it’s easy to overindex on this /​ weigh it too heavily. But I’d be surprised if that doesn’t affect things somewhat.)

  I also want to caveat that:

  • (i) I’m not trying to be exhaustive here, just listing what’s salient, and

  • (ii) in general it’ll be harder for me to name things that also strongly describe me (although I’m trying, to some degree), especially as I’m just quickly listing stuff and not thinking too hard.

  (And thanks for the nice meta note!)

  1. ^

     I’ve been struggling to articulate this well, but I’ve recently been feeling like, for instance, proposals on making deals with “early [potential] schemers” implicitly(?) rely on a bunch of assumptions about the anatomy of AI entities we’d get at relevant stages.

     More generally I’ve been feeling pretty iffy about using game-theoretic reasoning about “AIs” (as in “they’ll be incentivized to...” or similar) because I sort of expect it to fail in ways that are somewhat similar to what one gets if one tries to do this with states or large bureaucracies or something—iirc the fourth paper here discussed this kind of thing, although in general there’s a lot of content on this. Similar stuff on e.g. reasoning about the “goals” etc. of AI entities at different points in time without clarifying a bunch of background assumptions (related, iirc).

A per­sonal take on why you should work at Forethought (maybe)

• Lizka 31 Aug 2025 15:52 UTC

  4 points

  0 ∶ 0

  in reply to: Lizka’s comment on: Lizka’s Shortform

  Ah, @Gregory Lewis🔸 says some of the above better. Quoting his comment:

  • [...]

  • So ~everything is ultimately an S-curve. Yet although ‘this trend will start capping out somewhere’ is a very safe bet, ‘calling the inflection point’ before you’ve passed it is known to be extremely hard. Sigmoid curves in their early days are essentially indistinguishable from exponential ones, and the extra parameter which ~guarantees they can better (over?)fit the points on the graph than a simple exponential give very unstable estimates of the putative ceiling the trend will ‘cap out’ at. (cf. 1, 2.)

  • Many important things turn on (e.g.) ‘scaling is hitting the wall ~now’ vs. ‘scaling will hit the wall roughly at the point of the first dyson sphere data center’ As the universe is a small place on a log scale, this range is easily spanned by different analysis choices on how you project forward.

  • Without strong priors on ‘inflecting soon’ vs. ‘inflecting late’, forecasts tend to be volatile: is this small blip above or below trend really a blip, or a sign we’re entering a faster/​slow regime?

  • [...]

• Lizka 31 Aug 2025 15:49 UTC

  2 points

  0 ∶ 0

  in reply to: Eli Rose🔸’s comment on: Lizka’s Shortform

  I tried to clarify things a bit in this reply to titotal: https://​​forum.effectivealtruism.org/​​posts/​​iJSYZJJrLMigJsBeK/​​lizka-s-shortform?commentId=uewYatQz4dxJPXPiv

  In particular, I’m not trying to make a strong claim about exponentials specifically, or that things will line up perfectly, etc.

  (Fwiw, though, it does seem possible that if we zoom out, recent/​near-term population growth slow-downs might be functionally a ~blip if humanity or something like it leaves the Earth. Although at some point you’d still hit physical limits.)

• Lizka 31 Aug 2025 15:44 UTC

  6 points

  0 ∶ 0

  in reply to: titotal’s comment on: Lizka’s Shortform

  Oh, apologies: I’m not actually trying to claim that things will be <<exactly.. exponential>>. We should expect some amount of ~variation in progress/​growth (these are rough models, we shouldn’t be too confident about how things will go, etc.), what’s actually going on is (probably a lot) more complicated than a simple/​neat progression of new s-curves, etc.

  The thing I’m trying to say is more like:

  • When we’ve observed some datapoints about a thing we care about, and they seem to fit some overall curve (e.g. exponential growth) reasonably well,

  • then pointing to specific drivers that we think are responsible for the changes — & focusing on how those drivers might progress or be fundamentally limited, etc. — often makes us (significantly) overestimate bottlenecks/​obstacles standing in the way of progress on the thing that we actually care about.

  • And placing some weight on the prediction that the curve will simply continue^[1] seems like a useful heuristic /​ counterbalance (and has performed well).

  (Apologies if what I’d written earlier was unclear about what I believe — I’m not sure if we still notably disagree given the clarification?)

  ---

  A different way to think about this might be something like:

  • The drivers that we can point to are generally only part of the picture, and they’re often downstream of some fuzzier higher-level/​”meta” force (or a portfolio of forces) like “incentives+...”

    • It’s usually quite hard to draw a boundary around literally everything that’s causing some growth/​progress

    • It’s also often hard to imagine, from a given point in time, very different ways of driving the thing forward

    • (e.g. because we’ve yet to discover other ways of making progress, because proxies we’re looking at locally implicitly bake in some unnecessary assumptions about how progress on the thing we care about will get made, etc.)

  • So our stories about what’s causing some development that we’re observing are often missing important stuff, and sometimes we should trust the extrapolation more than the stories /​ assume the stories are incomplete

  Something like this seems to help explain why views like “the curve we’re observing will (basically) just continue” have seemed surprisingly successful, even when the people holding those “curve go up” views justified their conclusions via apparently incorrect reasoning about the specific drivers of progress. (And so IMO people should place non-trivial weight on stuff like “rough, somewhat naive-seeming extrapolation of the general trends we’re observing^[2].”^[3])

  [See also a classic post on the general topic, and some related discussion here, IIRC: https://​​www.alignmentforum.org/​​posts/​​aNAFrGbzXddQBMDqh/​​moore-s-law-ai-and-the-pace-of-progress ]

  1. ^

     Caveat: I’d add ”...on a big range/​ the scale we care about”; at some point, ~any progress would start hitting ~physical limits. But if that point is after the curve reshapes ~everything we care about, then I’m basically ignoring that consideration for now.

  2. ^

     Obviously there are caveats. E.g.:

     - the metrics we use for such observations can lead us astray in some situations (in particular they might not ~linearly relate to “the true thing we care about”)

     - we often have limited data, we shouldn’t be confident that we’re predicting/​measuring the right thing, things can in fact change over time and we should also not forget that, etc.

     (I think there were nice notes on this here, although I’ve only skimmed and didn’t re-read https://​​arxiv.org/​​pdf/​​2205.15011 )

  3. ^

     Also, sometimes we do know what

  What links here?

  • Lizka's comment on Lizka’s Quick takes by Lizka (31 Aug 2025 15:49 UTC; 2 points)

• Lizka 31 Aug 2025 15:18 UTC

  21 points

  4 ∶ 0

  on: The ‘com­mu­nity’ tag is problematic

  Replying quickly, speaking only for myself^[1]:

  1. I agree that the boundary between Community & non-Community posts is (and has always been) fuzzy

     1. You can see some guidance I wrote in early 2023: Community vs. other^[2] (a “test-yourself” quiz is also linked in the doc)

        1. Note: I am no longer on the Online Team/​an active moderator and don’t actually know what the official guidance is today.^[3]

     2. I also agree that this fuzziness is not a trivial issue, & it continues to bug me

        1. when I go to the Frontpage, I relatively frequently see posts that I think should be “Community” that aren’t, or vice versa

  2. However, the Community tag/​distinction/​section was not “a way to separate out FTX-scandal-related posts”, IMO

     1. Fwiw, I was (also?) really worried about this change when I was working on it, both because I was concerned it would suppress useful criticism/​reflection/​etc., and because I wanted to make sure my decision-making wasn’t somehow secretly tainted by various biases.

        1. I’m writing quickly here — not taking time to dig up my old docs or trying to really remember what was going on in early 2023^[4], but I vaguely remember e.g. calling @Ben_West🔸 at some point when I was feeling particularly stressed, to double check that we endorsed how we were approaching the change

        2. See also footnote 7 here, and footnote 2 here

     2. More:

        1. A lot of the thinking/​earlier attempts had happened before the FTX collapse

           1. “We’ve been hearing about the Forum fixating on certain discussions for over a year [as of Feb 2023]: this has been making a lot of people sad for a long time. [...]”

              1. [from here]

           2. And we’d already down-weighted “community” posts for new & logged-out users (before the FTX collapse); IMO this was always a pretty hacky feature that had a bunch of flaws

           3. The community tag itself had existed for a while before that

        2. And, quoting from myself here (from July 2023):

           1. Just a quick clarification: I don’t think this was a “change to lower community engagement.” Adding the community section was a change, and it did (probably^[1]) lower engagement with community posts, but that wasn’t the actual point (which is a distinction I think is worth making, although maybe some would say it’s the same). In my view, the point was to avoid suffocating other discussions and to make the Forum feel less on-edge/​anxiety-inducing (which we were hearing was at least some people’s experience). In case it helps, this outlines our (or at least my) thinking about it.

     3. (i.e. +1 to @Lorenzo Buonanno🔸 in the comments)

  3. Quick takes on where things are today:

     1. I would like to see more “Community” posts /​ discussions

     2. I think the Community section overall is probably still (quite) positive (but tagging could probably be improved)

        1. Off the top of the head, the two key things I’d probably look at /​think about, if I were considering removing it (or doing something else), are (1) whether it seems like it’s really tanking engagement on Community posts, or preventing them from being written (I’d maybe chat to likely/​potential authors about this, look at the numbers, etc.), and (2) how active users (i.e. readers) feel about the Community section^[5]

  1. ^

     I.e. I’m not speaking for the Online/​Mod teams here, and didn’t run this comment by anyone.

  2. ^

     (I vaguely remember making and linking a public version of this doc somewhere at some point, but couldn’t quickly find that.)

  3. ^

     In fact it looks like I can no longer add or remove the Community tag from posts. I’m still in the Slack; a few people sometimes flag questions about edge cases there.

  4. ^

     (...a lot!)

  5. ^

     I’d guess that most people still prefer it. The change got a lot of support, both before and after (and this is rare).

• Lizka 20 Aug 2025 14:10 UTC

  10 points

  3 ∶ 0

  on: Lizka’s Shortform

  I sometimes see people say stuff like:

  Those forecasts were misguided. If they ended up with good answers, that’s accidental; the trends they extrapolated from have hit limits… (Skeptics get Bayes points.)

  But IMO it’s not a fluke that the “that curve is going up, who knows why” POV has done well.

  A sketch of what I think happens:

  

  There’s a general dynamic here that goes something like:

  1. Some people point to a curve going up (and maybe note some underlying driver)

  2. Others point out that the drivers have inherent constraints (this is an s-curve, not an exponential)

  And then in *some* sense the bottlenecks crowd turn out to be right (the specific drivers/​paradigm peters out, there’s literally no more space for more transistors, companies run low on easily accessible/​high quality training data, etc.)…

  …but then a “surprise new thing” pops up and fills the gap, such that the “true” thing we cared about (whether or not it’s what we were originally measuring) *does* actually continue as people originally predicted, apparently naively

  (and it turned out that the curve consists of a stack of s-curves..)

  We can go too far with this kind of reasoning; some “true things we care about” (e.g. spread of a disease) * are* in fact s-curves, bounded, etc., and only locally look like ~exponentials. (So: no, we shouldn’t expect the baby to weigh trillions of pounds by age 10...)

  But I think the more granular, gears-oriented view — which considers how long specific drivers of the progress we’re seeing could continue, etc. — often underrates the extent to which *other forces* can (and often do) jump in when earlier drivers lose momentum.

  (~Crossposting from Twitter.)

  ---

  “The Bypass Principle: How AI flows around obstacles” from Eric Drexler is a very related (and IMO good) post. Quote (bold mine):

  While shallow assessments focus on visible obstacles — the difficulties of matching human capabilities, of overcoming regulatory barriers, and of restructuring organizations — AI-enabled developments will often find paths that bypass rather than overcome apparent barriers. Existing obstacles are concrete and obvious in a way that alternatives are not. Skewed judgment follows.

  ---

  (This stuff isn’t new; many people have pointed out these kinds of dynamics. But I feel like I’m still seeing them a fair bit — and this came up recently — so I wanted to write this note.)

• Lizka 20 Aug 2025 13:39 UTC

  4 points

  0 ∶ 0

  in reply to: OllieBase’s comment on: ITN 201: pit­falls in ITN BOTECs

  Yeah, this sort of thing is partly why I tend to feel better about BOTECs like (writing very quickly, tbc!):

  What could we actually accomplish if we (e.g.) doubled (the total stock/​ flow of) investment in ~technical AIS work (specifically the stuff focused on catastrophic risks, in this general worldview)? (you could broaden if you wanted to, obviously)

  Well, let’s see:

  • That might look like:

    • adding maybe ~400(??) FTEs similar (in ~aggregate) to the folks working here now, distributed roughly in proportion to current efforts /​ profiles — plus the funding/​AIS-specific infrastructure (e.g. institutional homes) needed to accomodate them

    • E.g. across intent alignment stuff, interpretability, evals, AI control, ~safeguarded AI, AI-for-AIS, etc., across non-profit/​private/​govt (but in fact aimed at loss of control stuff).

  • How good would this be?

    • Maybe (per year of doubling) we’d then get something like a similar-ish value from this as we don from a year of current space (or something like 2x less, if we want to eyeball diminishing returns)

    • Then maybe we can look at what this space has accomplished in the past year and see how much we’d pay for that /​ how valuable that seems...

  • (What other ~costs might we be missing here?)

  You might also decide that you have much better intuitions for how much we’d accomplish (and how valuable that’d be) on a different scale (e.g. adding one project like Redwood/​Goodfire/​Safeguarded AI/​..., i.e. more like 30 FTEs than 400 — although you’d probably want to account for considerations like “for each ‘successful’ project we’d likely need to invest in a bunch of attempts/​ surrounding infrastructure...”), or intuitions about what amount of investment is required to get to some particular desired outcome...

  Or if you took the more ITN-style approach, you could try to approach the BOTEC via something like (1) how much investment has there been so far in this broad ~POV /​ porftolio, (2 (option a)) how much value/​progress has this portfolio made + something like “how much has been made in the second half?” (to get a sense of how much we’re facing diminishing returns at the moment — fwiw without thinking too much about it I think “not super diminishing returns at the mo”), or (2 (option b)) what fraction of the overall “AI safety problem” is “this-sort-of-safety-work-affectable” (i.e. something like “if we scaled up this kind of work — and only this kind of work — to an insane degree, how much of the problem will be fixed?”) + how big/​important the problem is overall… Etc. (Again, for all of this my main question is often “what are the sources of signal or taste /​ heuristics /​ etc. that you’re happier basing your estimates on?)

• Lizka 20 Aug 2025 13:09 UTC

  2 points

  0 ∶ 0

  in reply to: JamesÖz 🔸’s comment on: ITN 201: pit­falls in ITN BOTECs

  Thank you! I used Procreate for these (on an iPad).^[1]

  (I also love Excalidraw for quick diagrams, have used & liked Whimsical before, and now also semi-grudgingly appreciate Canva.)

  ---

  Relatedly, I wrote a quick summary of the post in a Twitter thread a few days ago and added two extra sketches there. Posting here too in case anyone finds them useful:

  

  

  1. ^

     (And a meme generator for the memes.)

ITN 201: pit­falls in ITN BOTECs

A shal­low re­view of what trans­for­ma­tive AI means for an­i­mal welfare

Notes on dy­namism, power, & virtue

• Lizka 18 May 2025 14:00 UTC

  6 points

  1 ∶ 0

  in reply to: Lizka’s comment on: Three lower-cost op­tions for run­ning the EA Forum

  Yeah actually I think @Habryka [Deactivated] discusses these kinds of dynamics here: https://​​www.lesswrong.com/​​posts/​​4NFDwQRhHBB2Ad4ZY/​​the-filan-cabinet-podcast-with-oliver-habryka-transcript

  Excerpt (bold mine, Habryka speaking):

  One of the core things that I was always thinking about with LessWrong, and that was my kind of primary analysis of what went wrong with previous LessWrong revivals, was [kind of] an iterated, [the term] “prisoner’s dilemma” is overused, but a bit of an iterated prisoner’s dilemma or something where, like, people needed to have the trust on an ongoing basis that the maintainers and the people who run it will actually stick with it. And there’s a large amount of trust that the people need to have that, if they invest in a site and start writing content on it, that the maintainers and the people who run it actually will put the effort into making that content be shepherded well. And the people who want to shepherd it only want to do that if the maintainers actually...

  And so, one of the key things that I was thinking about, was trying to figure out how to guarantee reliability. This meant, to a lot of the core contributors of the site, I made a promise when I started it, that was basically, I’m going to be making sure that LessWrong is healthy and keeps running for five years from the time I started. Which was a huge commitment—five years is a hugely long time. But my sense at the time was that type of commitment is exactly the most important thing. Because the most usual thing that I get when I talk [in] user interviews to authors and commenters is that they don’t want to contribute because they expect the thing to decline in the future. So reliability was a huge part of that.

  And then I also think, signaling that there was real investment here was definitely a good chunk of it. I think UI is important, and readability of the site is important. And I think I made a lot of improvements there to decide that I’m quite happy with. But I think a lot of it was also just a costly signal that somebody cares.

  I don’t know how I feel about that in retrospect. But I think that was a huge effect, where I think people looked on the site, and when [they] looked at LessWrong 2.0, there was just a very concrete sense that I could see in user interviews that they were like, “Oh, this is a site that is being taken care of. This is a thing that people are paying attention to and that is being kept up well.” In a similar [sense] to how, I don’t know, a clean house has the same symbol. I don’t really know. I think a lot of it was, they were like, wow, a lot of stuff is changing. And the fact that a lot of work is being put into this, the work itself is doing a lot of valuable signaling.

• Lizka 18 May 2025 13:52 UTC

  4 points

  1 ∶ 0

  in reply to: Sarah Cheng 🔸’s comment on: Three lower-cost op­tions for run­ning the EA Forum

  Re not necessarily “optimizing” for the Forum, I guess my frame is:

  The Online Team is the current custodian of an important shared resource (the Forum). If the team can’t actually commit to fulfilling its “Forum custodian” duties, e.g. because the priorities of CEA might change, then it should probably start trying to (responsibly) hand that role off to another person/​group.

  (TBC this doesn’t mean that Online should be putting all of its efforts into the Forum, just like a parent has no duty to spend all their energy caring for their child. And it’s not necessarily clear what the bar for responsibly fulfilling Forum custodian duties actually is — maybe moderation and bug fixes are core charges, but “no new engineering work” is fine, I’m not sure.)

  I would view this somewhat differently if it was possible for another group to compete with or step in for CEA /​ the Online Team if it seemed that the team is not investing [enough] in the Forum (or investing poorly). But that’s not actually possible — in fact even if the Online Team stopped ~everything, by default no one else would be able to take over. I’d also feel somewhat differently if the the broader community hadn’t invested so much in the Forum, and if I didn’t think that a baseline ~trust in (and therefore clear commitment from) the team was so important for the Forum’s fate (which I believe for reasons loosely outlined in the memo, IIRC).

  ...

  Btw, I very much agree that staring into the abyss (occasionally) is really useful. And I really appreciate you posting this on the Forum, and also engaging deeply/​openly in the replies.

• Lizka 17 May 2025 16:39 UTC

  9 points

  1 ∶ 0

  on: Three lower-cost op­tions for run­ning the EA Forum

  I think going for Option 2 (“A bulletin board”) or 3 (“Shut down”) would be pretty a serious mistake, fwiw. (I have fewer/​weaker opinions on 1, although I suspect I’m more pessimistic about it than ~most others.)

  ...

  An internal memo I wrote in early 2023 (during my time on the Online Team) seems relevant, so I’ve made a public copy: Vision /​ models /​ principles for the Forum and the Forum+ team^[1]

  I probably no longer believe some of what I wrote there, but still endorse the broad points/​models, which lead me think, among other things:

  1. That people would probably ~stop visiting the Forum if it were closer to a bulletin board

  2. That people would just not relate to the Reddit version in a way that gets remotely close to what the Forum currently does (e.g. I probs believe something in the direction of “support for posting longform content in the subreddit is almost irrelevant; people won’t really be doing it”)

  3. That investment from the Online Team matters in part to maintain ~trust that the Forum will continue existing as the kind of space people want to invest in

  4. That major shifts could be irreversible even when they don’t seem inherently so (e.g. “pausing” the Forum to re-evaluate would be closer to killing it, probably; people would very likely not come back /​ the space would be fundamentally different if they did)

  5. Etc.

  Other notes (adding to what the models I’d described in the memo lead me to believe, or articulating nearby/​more specific versions of the claims I make there, etc.):

  • A platform’s vibe/​form actually matters for the kind of content and discussion it gets

    • I expect a lot less high-effort content would get posted even e.g. if the Forum itself suddenly looked like Reddit (and a lot of that kind of content won’t ever get written)

  • I feel like the discussion here should probably more clearly differentiate between:

    • “What should happen to the Forum” /​ “how good or bad are different futures for the Forum” vs

    • “What should the Online Team do/​ how valuable are various kinds of investment”

    • and maybe also: “How can we cut costs /​ fund this work”

      • (I suspect it might make sense for the Online Team to fundraise from the EA community—although I don’t know if that’s an option given the current setup)

  1. ^

     (also called “Equilibrium curves & being stewards of the special thing”)

     The memo outlines some of how I was thinking about how the Forum works, especially an “equilibrium curves” model and a view that trust is an key thing to track/​build towards. It also discusses the value of the Online Team’s work, theories of change, and when (if ever) “closing” the Forum would make sense and how that could work/​play out.

     (I know Sarah’s read this, but figured I’d share it in case others are interested, and because I’m about to reference stuff from there.)

     Note: At least two fairly important parts of my current models seem missing from the doc (and I suspect I’d think of several more if I thought about it for more than the time it took to skim the doc and write this comment): (1) the Forum as a “two-sided marketplace” (“writers /​ content producers” and “readers”), and (2) creation of (object-level) common knowledge as an important thing the Forum does sometimes.

  What links here?

  • Sarah Cheng 🔸's comment on Three lower-cost op­tions for run­ning the EA Forum by Sarah Cheng 🔸 (22 May 2025 20:34 UTC; 4 points)

• Lizka 2 May 2025 1:38 UTC

  45 points

  1 ∶ 0

  on: Reflec­tions on 7 years build­ing the EA Fo­rum — and mov­ing on

  I just want to say that I’m really, really glad that I got the chance to work with JP, and I think JP’s work has contributed a bunch for the Forum.

  Below I’m sharing some quick misc notes on why that is.

  Meta: sharing a public comment like this on a friend’s post is pretty weird for me! But JP’s work has mostly stayed behind the scenes and has thus gotten significantly less acknowledgement than it deserves, IMO, so I did feel the urge to post this kind of thing publicly. As a compromise between those feelings, y’all are getting the comment along with this meta flag.

  ---

  Misc—not exhaustive! --- notes on some subtle but important ways JP has shaped (my experience on) the Forum (team)

  • Deep models of how things work. Conversations with JP have massively shaped the way I think about the Forum^[1] . My models would have been shallower and less accurate without that. More broadly, I think we would have made worse choices as a team if JP had not been not bringing his models in.

    • Quick list of some specific topics in which my models have been especially obviously shaped^[2] by JP:

      • “acculturation” stuff, the role of trust and how it works, what causes some authors to post, [productivity] & how teams (should) work (lots of good blog posts JP pointed me to!), why the rationality & EA communities have developed the way they have & how do they actually work now, [a bunch of classic EA/​rationalist writing/​models that I hadn’t encountered but have from JP], what it means for someone to be aligned in a certain way, etc.

    • In some cases I felt like “JP energy” was a very strong protective barrier between us and potential Goodharting-style mistakes^[3]

      • Over time I think I was increasingly playing a similar role, too, but could do so largely because I had absorbed a bunch of “JP energy” earlier on (and I felt that we were also helping each other improve our models). It wasn’t just me; IMO the whole team was absorbing & refining these models, and without JP this would have gone much worse.

    • (Some fraction of this “deep models” superpower, I think, can be attributed to the fact that JP had been working on the Forum for a very long time.^[4] To me this kind of thing is strong evidence of the value of sticking around, fwiw; you become a really strong pillar and an incredibly rich source of information. But length-of-Forum-tenure is not the only explanation for this superpower, IMO. (Stuff like “deeply caring”, “willingness to mull”, “being a nerd about systems”, … comes to mind.))

  • Integrity /​ being a compass /​ norm/​culture-setting. JP “set the tone” for a good chunk of how I related to my Forum role. He was also one of the first people I turned to (and found really useful to consult) when I felt conflicted about decisions I was trying to make or about internal (e.g. CEA) dynamics.

    • One illustration, from one of the earliest serious incidents I worked on as a moderator (note that my recollection is fuzzy!):

      • What happened: mods had reason to be suspicious, but we weren’t sure if the user in question had actually done the specific norm-violating thing because the relevant information (e.g. vote patterns or something) was private (even if in theory accessible to the team). The Forum privacy expectations^[5] at the time were more confusing/​unwritten, IIRC, and we didn’t have a standard process for handling this kind of issue. (IIRC) JP kicked off and arranged a (multi-person?) process that got us the key info while ensuring that no one actually saw the relevant private info.

      • I remember being really impressed by this. It can seem like a fairly minor thing — realistically this wasn’t that important, it wasn’t the kind of thing where people would be surprised that this info was accessible to us or expect us to be super careful, etc. But being the kind of person/​team that handles these situations with care/​integrity is really valuable, IMO, and it sets an important team norm. The whole thing left me with a stronger/​more visceral sense of responsibility about how we should handle things.

    • Talking to JP helped me untangle some confusing/​stressful internal team/​CEA/​stakeholder dynamics, and deal with them in ways I endorsed more.^[6]

    • And it was just very energizing to work with someone who loves the Forum and the EA community :)

  • Personal side—helping with development & encouragement, being kind

    • JP is great at noticing how certain habits/​patterns can be improved (and explaining), sharing productivity tips, etc. I gained a lot by bringing up various issues and working through them with him, or by following his advice on tech/​systems. (In fact I’m still taking advantage of this opportunity — highly recommend for those with the privilege.)

    • JP helped me “back myself” & fight back some of the imposter syndrome that I started with.

    • Even when I’d just started at CEA, JP was just really warm & kind to me, which makes a big difference to a recent college graduate who’s just moved to a different country.

  (And obviously there was the whole “running the tech” side of things, and then the whole “running the Online Team” thing, lol… Have I mentioned this list is not exhaustive? This is basically an unfiltered babble of thoughts, but I hope it clarifies some of why I’m so grateful for JP’s work on the Forum.)

  I’m really looking forward to seeing what JP does next.

  1. ^

     and probably also my models of EA, and stuff like “how organizations work”

  2. ^

     (To be clear, it’s not like I was constantly in agreement! Our models often diverged, or we felt different points mattered more or were better frames for understanding things, etc. But this was also very useful.)

  3. ^

     I’m not saying that we were close to hardcore Goodharting or similar, to be clear! Although I do feel like there were some potential naivete-driven mistakes that we avoided because of this kind of thing

  4. ^

     By relevant standards

  5. ^

     i.e. what users should expect about when the Online Team might look at some user info that is normally private

  6. ^

     One quite salient example for me: I felt conflicted and stressed about various Forum/​EV-related things in late 2022/​early 2023 [a] (e.g. I was pretty upset about the EV comms policy). Conversations with JP helped me untangle my feelings and thoughts and figure out what I endorsed doing. In general, having JP on the team — even before he took over from Ben — was really important for me during this time.

     [a] the uptick in Forum engagement must have increased the burden, I suppose

Pod­cast on “AI tools for ex­is­ten­tial se­cu­rity” — transcript

• Lizka 14 Apr 2025 16:58 UTC

  6 points

  0 ∶ 0

  in reply to: Ben_West🔸’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  I think this is a good question, and it’s something I sort of wanted to look into and then didn’t get to! (If you’re interested, I might be able to connect you with someone/​some folks who might know more, though.)

  Quick general takes on what private companies might be able to do to make their tools more useful on this front (please note that I’m pretty out of my depth here, so take this with a decent amount of salt—and also this isn’t meant to be prioritized or exhaustive):

  • Some of the vetting/​authorization processes (e.g. FedRAMP) are burdensome, and sometimes companies do just give up/​don’t bother (see footnote 12), which narrows the options for agencies; going through this anyway could be very useful

  • Generally lowering costs for tech products can make a real difference for whether agencies will adopt them—also maybe open source products are likelier to be used(?) (and there’s probably stuff like which chips are available, which systems staff are used to, what tradeoffs on speed vs cost or similar make sense...)

  • Security is useful/​important, e.g. the tool/​system can be run locally, can be fine-tuned with sensitive data, etc. (Also I expect things like “we can basically prove that the training /​ behavior of this model satisfies [certain conditions]” will increasingly matter — with conditions like “not trained on X data” or “could not have been corrupted by any small group of people” — but my understanding/​thinking here is very vague!)

  • Relatedly I think various properties of the systems’ scaffolding will matter; e.g. “how well the tool fits with existing and future systems” — so modularity and interoperability^[1] (in general and with common systems) are very useful — and “can this tool be set up ~once but allow for different forms of access/​data” (e.g. handle differences in who can see different kinds of data, what info should be logged, etc.)

  (Note also there’s a pretty huge set of consultancies that focus on helping companies sell to the government, but the frame is quite different.)

  And then in terms of ~market gaps, I’m again very unsure, but expect that (unsurprisingly) lower-budget agencies will be especially undersupplied — in particular the DOD has a lot more funding and capacity for this kind of thing — so building things for e.g. NIST could make sense. (Although it might be hard to figure out what would be particularly useful for agencies like NIST without actually being at NIST. I haven’t really thought about this!)

  1. ^

     I haven’t looked into this at all, but given the prevalence of Microsoft systems (Azure etc.) in the US federal government (which afaik is greater than what we see in the UK), I wonder if Microsoft’s relationship with OpenAI explains why we have ChatGPT Gov in the US, while Anthropic is collaborating with the UK government https://​​www.anthropic.com/​​news/​​mou-uk-government

• Lizka 14 Apr 2025 16:18 UTC

  6 points

  0 ∶ 0

  in reply to: OscarD🔸’s comment on: The AI Adop­tion Gap: Prepar­ing the US Govern­ment for Ad­vanced AI

  the main question is how high a priority this is, and I am somewhat skeptical it is on the ITN pareto frontier. E.g. I would assume plenty of people care about government efficiency and state capacity generally, and a lot of these interventions are generally about making USG more capable rather than too targeted towards longtermist priorities.

  Agree that “how high-priority should this be” is a key question, and I’m definitely not sure it’s on the ITN pareto frontier! (Nice phrase, btw.)

  Quick notes on some things that raise the importance for me, though:

  1. I agree lots of people care about government efficiency/​ state capacity — but I suspect few of them are seriously considering the possibility of transformative AI in the near future, and I think what you do to ~boost capacity looks pretty different in that world

  2. Also/​relatedly, my worldview means I have extra reasons to care about state capacity, and given my worldview is unusual that means I should expect the world is underinvesting in state capacity (just like most people would love to see a world with fewer respiratory infections, but tracking the possibility of a bioengineered pandemic means I see stuff like far-UVC/​better PPE/​etc. as higher value)

     1. More generally I like the “how much more do I care about X” frame — see this piece from 2014

     2. (It could also be a kind of public good.)

  3. In particular, it seems like a *lot* of the theory of change of AI governance+ relies on competent/​skillful action/​functioning by the US federal government, in periods where AI is starting to radically transform the world (e.g. to mandate testing and be able to tell if that mandate isn’t being followed!), and my sense is that this assumption is fragile/​the govt may very well not actually be sufficiently competent — so we better be working on getting there, or investing more in plans that don’t rely on this assumption

  And I’m pretty worried that a decent amount of work aimed at mitigating the risks of AI could end up net-negative (for its own goals) by not tracking this issue and thus not focusing enough on the interventions that are actually worth pursuing—further harming government AI adoption & competence /​ capacity in the process (e.g. I think some of the OMB/​EO guidance from last year looked positive to me before I dug into this, and now looks negative). So I’d like to nudge some people who work on issues related to existential risk (and government) away from a view like: “all AI is scary/​bad, anything that is ‘pro-AI’ increases existential risk, if this bundle of policies/​barriers inhibits a bunch of different AI things then that’s probably great even if I think only a tiny fraction is truly (existentially) risky”, etc.

  --

  this felt like neither the sort of piece targeted to mainstream US policy folks, nor that convincing for why this should be an EA/​longtermist focus area.

  Totally reasonable reaction IMO. To a large extent I see this as a straightforward flaw of the piece & how I approached it (partly due to lack of time—see my reply to Michael above), although I’ll flag that my main hope was to surface this to people who are in fact kind of in between—e.g. folks at think tanks that do research on existential security and have government experience/​expertise.

  --

  I’m unconvinced that e.g. OP should spin up a grantmaker focused on this (not that you were necessarily recommending this).

  I am in fact not recommending this! (There could be specific interventions in the area that I’d see as worth funding, though, and it’s also related to other clusters where something like the above is reasonable IMO.)

  --

  Also, a few reasons govts may have a better time adopting AI come to mind:

  • Access to large amounts of internal private data

  • Large institutions can better afford one-time upfront costs to train or finetune specialised models, compared to small businesses

  But I agree the opposing reasons you give are probably stronger.

  The data has to be accessible, though, and this is a pretty big problem. See e.g. footnote 17.

  I agree that a major advantage could be that the federal government can in fact move a lot of money when ~it wants to, and could make some (cross-agency/​...) investments into secure models or similar, although my sense is that right now that kind of thing is the exception/​aspiration, not the rule/​standard practice. (Another advantage is that companies do want to maintain good relationships with the government/​admin, and might thus invest more in being useful. Also there are probably a lot of skilled people who are willing to help with this kind of work, for less personal gain.)

  --

  If only this were how USG juggled its priorities!

  🙃 (some decision-makers do, though!)