tylermjohn

• tylermjohnApr 14, 2025, 8:27 AM

  5 points

  1 ∶ 0

  in reply to: calebp’s comment on: tyler­mjohn’s Quick takes

  I think these are fair points, I agree the info hazard stuff has smothered a lot of talent development and field building, and I agree the case for x-risk from misaligned advanced AI is more compelling. At the same time, I don’t talk to a lot of EAs and people in the broader ecosystem these days who are laser focused on extinction over GCR, that seems like a small subset of the community. So I expect various social effects, making a bunch more money, and AI being really cool and interesting and fast-moving are probably a bigger deal than x-risk compellingness simpliciter. Or at least they have had a bigger effect on my choices!

  But insufficiently successful talent development /​ salience /​ comms is probably the biggest thing, I agree.

• tylermjohnApr 13, 2025, 9:53 AM

  6 points

  0 ∶ 0

  in reply to: Marcus Abramovitch 🔸’s comment on: tyler­mjohn’s Quick takes

  Yup! The highest level plan is in Kevin Esvelt’s “Delay, Detect, Defend”: use access controls and regulation to delay worst-case pandemics, build a nucleic acid observatory and other tools to detect amino acid sequences for superpandemics, and defend by hardening the world against biological attacks.

  The basic defense, as per DDD, is:

  • Develop and distribute adequate PPE to all essential workers

  • Make sure the supply chain is robust to ensure that essential workers can distribute food and essential supplies in the event of a worst-case pandemic

  • Environmental defenses like far-UVC that massively reduce the spread and replication rate of pandemic pathogens

  IMO “delay” has so far basically failed but “detect” has been fairly successful (though incompletely). Most of the important work now needs to rapidly be done on the “defend” side of things.

  There’s a lot more details on this and the biosecurity community has really good ideas now about how to develop and distribute effective PPE and rapidly scale environmental defenses. There’s also now interest in developing small molecule countermeasures that can stop pandemics early but are general enough to stop a lot of different kinds of biological attacks. A lot of this is bottlenecked by things like developing industrial-scale capacity for defense production or solving logistics around supply chain robustness and PPE distribution. Happy to chat more details or put you in touch with people better suited than me if it’s relevant to your planning.

• tylermjohnApr 12, 2025, 12:51 PM

  16 points

  1 ∶ 1

  on: tyler­mjohn’s Quick takes

  AIxBio looks pretty bad and it would be great to see more people work on it

  • We’re pretty close to having a country of virologists in a data center with AI models that can give detailed and accurate instructions for all steps of a biological attack — with recent reasoning models, we might have this already

  • These models have safeguards but they’re trivial to overcome — Pliny the Liberator manages to jailbreak every new model within 24 hours and open sources the jailbreaks

  • Open source will continue to be just a few months behind the frontier given distillation and amplification, and these can be fine-tuned to remove safeguards in minutes for less than $50

  • People say it’s hard to actually execute the biology work, but I don’t see any bottlenecks to bioweapon production that can’t be done by a bio undergrad with limitless scientific knowledge; on my current understanding, the bottlenecks are not manual dexterity bottlenecks like playing a violin which require years of practice, they are knowledge bottlenecks

  • Bio supply chain controls that make it harder to get ingredients aren’t working and aren’t on track to work

  • So it seems like we’re very close to democratizing (even bespoke) bioweapons. When I talk to bio experts about this they often reassure me that few people want to conduct a biological attack, but I haven’t seen much analysis on this and it seems hard to be highly confident.

  While we gear up for a bioweapon democracy it seems that there are very few people working on worst-case bio, and most of the people working on it are working on access controls and evaluations. But I don’t expect access controls to succeed, and I expect evaluations to mostly be useful for scaring politicians, due in part to the open source issue meaning we just can’t give frontier models robust safeguards. The most likely thing to actually work is biodefense.

  I suspect that too many people working on GCR have moved into working on AI alignment and reliability issues and too few are working on bio. I suspect there are bad incentives, given that AI is the new technology frontier and working with AI is good career capital, and given that AI work is higher status.

  When I talk to people at the frontier of biosecurity, I learn that there’s a clear plan and funding available, but the work is bottlenecked by entrepreneurial people who can pick up a big project and execute on it autonomously — these people don’t even need a bio background. On my current guess, the next 3-5 such people who are ambivalent about what to do should go into bio rather than AI, in part because AI seems to be more bottlenecked by less generalist skills, like machine learning, communications, and diplomacy.

• tylermjohnApr 8, 2025, 5:31 PM

  2 points

  0 ∶ 1

  on: An­nounc­ing our book, After the Spike, and an Op­por­tu­nity to Help

  Very glad to see this coming out. Your team’s research has convinced me that if exponential AI progress doesn’t lead to a kind of above replacement fertility, where we can supplement biological humans with digital ones in all the relevant senses, then turning the spike into a steady climb will be one of the most important global priorities in the years ahead.

• tylermjohnMar 27, 2025, 6:32 AM

  1 point

  0 ∶ 0

  in reply to: Linch’s comment on: Power Laws of Value

  For what it’s worth, I do specifically have power-law shaped intuitions about the value of pleasure as you arrange matter to optimize for it more and more. But I agree with you both, I didn’t argue for this and it’s not important to my core point.

• tylermjohnMar 26, 2025, 10:25 AM

  1 point

  0 ∶ 0

  in reply to: Davidmanheim’s comment on: ...but is in­creas­ing the value of fu­tures tractable?

  Sure, there are various ways to do this. Scale up ems, for example, or build superintelligence from symbolic systems with strong verifiability guarantees, for starters.

• tylermjohnMar 26, 2025, 9:44 AM

  3 points

  1 ∶ 0

  in reply to: Linch’s comment on: Power Laws of Value

  Appreciate this comment, and very much agree. I generally think that humanity’s descendents are going to saturate the stars with Dyson swarms making stuff (there’s good incentives to achieve explosive growth) but I think we’re (1) too quick to assume that, (2) too quick to assume we will stop being attached to inefficient earth stuff, and (3) too quick to assume the Dyson swarms will be implementing great stuff rather than, say, insentient digital slaves used to amass power or solve scientific problems.

  Let’s say there are three threat models here: (a) Weird Stuff Matters A Lot, (b) Attachment to Biological Organisms, (c) Disneyland With No Children (the machines aren’t conscious).

  I focused mainly on Weird Stuff Matters A Lot. The main reason I focused on this rather than Attachment to Biological Organisms is that I still think that computers are going to be so much more economically efficient than biology that in expectation ~75% of everything is computer. Computers are just much more useful than animals for most purposes, and it would be super crazy from most perspectives not to turn most of the stars into computers. (I wouldn’t totally rule out us failing to do that, but incentives push towards it strongly.) If, in expectation, ~75% of everything is computer, then maximizing computer only makes the world better by ¹⁄₃.

  I think the Disneyland With No Children threat model is much scarier. I focused on it less here because I wanted to shore up broadly appealing theoretical reasons for trajectory change, and this argument feels much more partisan. But on my partisan worldview:

  • Consciousness is an indeterminate folk notion that we will reduce to computational properties. (Or something very close to this.)

  • These computational properties are going to be much, much more precise and gradable than our folk notion and they won’t wear folk psychological properties on their sleeves.

  • As a result we’re just going to have to make some choices about what stuff we think is conscious and what isn’t. There’s going to be a sharp borderline we’re going to have to pick arbitrarily, probably based on nothing more than whimsical values.

  • People will disagree about where the borderline is.

  • Even if people don’t disagree about the borderline, they’ll disagree substantially about cardinality, i.e. how much to value different computational properties relative to others.

  • Given the Power Laws of Value point, some people’s choices will be a mere shadow of value from the perspective of other people’s choices.

  If this “irrealist” view is right, it’s extremely easy to lose out on almost all value.

  Separately, I just don’t think our descendents are going to care very much about whether the computers are actually conscious, and so AI design choices are going to be orthogonal to moral value. On this different sort of orthogonality thesis, we’ll lose out on most value just because our descendents will use AI for practical reasons other than moral reasons, and so their intrinsic value will be unoptimized.

  So Disneyland With No Children type threat models look very credible to me.

  (I do think humans will make a lot of copies of themselves, which is decently valuable, but not if you’re comparing it to the most valuable world or if you value diversity.)

  You could have a more realist view where we just make a big breakthrough in cognitive science and realize that a very glowy, distinctive set of computational properties was what we were talking about all along when we talked about consciousness, and everyone would agree to that. I don’t really think that’s how science works, but even if you did have that view it’s hard to see how the computational properties would just wear their cardinality on their sleeves. Whatever computational properties you find you can always value them differently. If you find some really natural measure of hedons in computational space you can always map hedons to moral value with different functions. (E.g. map 1 hedon to 1 value, 2 hedons to 10 value, 3 hedons to 100 value...)

  So I didn’t focus on it here, but I think it’s definitely good to think about the Disneyland concern and it’s closely related to what I was thinking about when writing the OP.

  I really liked @Joe_Carlsmith articulation of your 23-word summary: what if all people are paperclippers relative to one another? Though it does make stronger assumptions than we are here.

• tylermjohnMar 21, 2025, 4:31 PM

  1 point

  0 ∶ 0

  in reply to: Parker_Whitfill’s comment on: tyler­mjohn’s Quick takes

  I don’t know! It’s possible that you can just solve a bargain and then align AI to that, like you can align AI to citizens assemblies. I want to be pitched.

• tylermjohnMar 21, 2025, 2:21 PM

  1 point

  0 ∶ 0

  in reply to: Davidmanheim’s comment on: ...but is in­creas­ing the value of fu­tures tractable?

  I don’t understand why that matters. Whatever discount rate you have, if you’re prioritizing between extinction risk and trajectory change you will have some parameters that tell you something about what is going to happen over N years. It doesn’t matter how long this time horizon is. I think you’re not thinking about whether your claims have bearing on the actual matter at hand.

  It would probably be most useful for you to try to articulate a view that avoids the dilemma I mentioned in the first comment of this thread.

• tylermjohnMar 21, 2025, 9:11 AM

  1 point

  0 ∶ 0

  in reply to: Davidmanheim’s comment on: ...but is in­creas­ing the value of fu­tures tractable?

  You’re not going to be prioritizing between extinction risk and long term trajectory changes based on tractability if you don’t care about the far future. And for any moral theory you can ask “why do you think this will be a good outcome?” and as long as you don’t value life intrinsically you’ll have to state some empirical hypotheses about the far future

• tylermjohnMar 20, 2025, 3:22 PM

  1 point

  0 ∶ 0

  in reply to: Davidmanheim’s comment on: ...but is in­creas­ing the value of fu­tures tractable?

  Any disagreement about longtermist prioritization should presuppose longtermism

tyler­mjohn’s Quick takes

• tylermjohnMar 20, 2025, 11:29 AM

  4 points

  1 ∶ 0

  on: tyler­mjohn’s Quick takes

  I want to see a bargain solver for AI alignment to groups: a technical solution that would allow AI systems to solve the pie cutting problem for groups and get them the most of what they want, for AI alignment. The best solutions I’ve seen for maximizing long run value involve using a bargain solver to decide what ASI does, which preserves the richness and cardinality of people’s value functions and gives everyone as much of what they want as possible, weighted by importance. (See WWOTF Afterwards, the small literature on bargaining-theoretic approaches to moral uncertainty.) But existing democratic approaches to AI alignment seem to not be fully leveraging AI tools, and instead aligning AI systems to democratic processes that aren’t empowered with AI tools (e.g. CIPs and CAIS’S alignment to the written output of citizens’ assemblies.) Moreover, in my experience the best way to make something happen is just to build the solution. If you might be interested in building this tool and have the background, I would love to try to connect you to funding for it.
  
  For deeper motivation see here.

• tylermjohnMar 19, 2025, 12:00 PM

  4 points

  0 ∶ 0

  in reply to: tylermjohn’s comment on: ...but is in­creas­ing the value of fu­tures tractable?

  Here’s a shower thought:

  • If you think extinction risk reduction is highly valuable, then you need some kind of a model of what Earth-originating life will do with its cosmic endowment

  • Some of the parameters in you model must be related to things other than mere survival, like what this life is motivated by or will attempt to do

  • Plausibly, there are things you can do to change the values of those parameters and not just the extinction parameter

  It won’t work for every model (maybe the other parameters just won’t budge), but for some of them it should.

• tylermjohnMar 19, 2025, 11:53 AM

  10 points

  3 ∶ 4

  on: ...but is in­creas­ing the value of fu­tures tractable?

  (Low effort comment as I run out the door, but hope it adds value) To me the most compelling argument in favour of tractability is:

  • We could make powerful AI agents whose goals are well understood and do not change or update in ex ante predictable ways.

  • These agents are effectively immortal and the most powerful thing in the affectable universe, with no natural competition. They would be able to overcome potentially all natural obstacles, so they would determine what happens in the lightcone.

  • So, we can make powerful AI agents that determine what happens in the lightcone, whose goals are well understood and update in ex ante predictable ways.

  • So, we can take actions that determine what happens in the lightcone in an ex ante predictable way.

• tylermjohnMar 19, 2025, 8:44 AM

  8 points

  2 ∶ 0

  in reply to: William_MacAskill’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  A cynical and oversimplified — but hopefully illuminating — view (and roughly my view) is that trajectory changes are just longterm power grabs by people with a certain set of values (moral, epistemic, or otherwise). One argument in the other direction is that lots of people are trying to grab power — it’s all powerful people do! And conflict with powerful people over resources is a significant kind of non-neglectedness. But very few people are trying to control the longterm future, due to (e.g.) hyperbolic discounting. So on this view, neglectedness provisionally favours trajectory changes that don’t reallocate power until the future, so that they are not in competition with people seeking power today. A similar argument would apply to other domains where power can be accrued but where competitors are not seeking power.

• tylermjohnMar 18, 2025, 3:46 PM

  5 points

  0 ∶ 0

  on: Power Laws of Value

  Some nice and insightful comments from Anders Sandberg on X:

  This is very interesting, I liked reading it. I am not sure I entirely agree with the analysis, but I think MPL may well be true, and if this is true leads to very different long term strategies (e.g. a need for more moral hedging).

  I am less convinced that we cannot find high-value states, or that they have to be human-parochial.

  A key assumption seems to be that not getting maximal value is a disaster, but I think one can equally have a glass-half-full positive view that the search will find greater and greater values.

  This essay also fits with my thinking that there might be new values out there, as yet unrealized. Once there was no life, and hence none of the values linked to living beings. Then consciousness, thinking and culture emerged, adding new kinds of value.

  I suspect this might keep on going. Not obvious that new levels add fundamentally greater values, but potentially fundamentally different values. And it is not implausible that some are lexically better than others.

  … if we sample new values at a constant rate and the values turn out to have a power law distribution, then the highest value found will in expectation grow (trying to work out the formula, but roughly linearly).

• tylermjohnMar 17, 2025, 5:15 PM

  3 points

  0 ∶ 0

  in reply to: Davidmanheim’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  That is an excellent question. I think ethical theory matters a lot — see Power Laws of Value. But I also just think our superintelligent descendants are going to be pretty derpy and act on enlightened self-interest as they turn the stars into computers, not pursue very good things. And that might be somewhere where, e.g., @William_MacAskill and I disagree.

• tylermjohnMar 17, 2025, 5:12 PM

  3 points

  0 ∶ 1

  in reply to: Davidmanheim’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  I think there is a large minority chance that we will successfully align ASI this century, so I definitely think it is possible.

• tylermjohnMar 17, 2025, 5:10 PM

  3 points

  0 ∶ 0

  in reply to: William_MacAskill’s comment on: Ex­is­ten­tial Choices Sym­po­sium with Will MacAskill and other spe­cial guests (3-5pm GMT Mon­day)

  Thank you! IMO the best argument for subjectivists not having these views would be thinking that (1) humans generally value reasoning processes, (2) there are not that many different reasoning processes you could adopt or as a matter of biological or social fact we all value roughly the same reasoning processes, and (3) these processes have clear and determinate implications. Or, in short, Kant was right: if we reason from the standpoint of “reason”, which is some well-defined and unified thing that we all care about, we all end up in the same place. But I reject all of these premises.
  
  The other argument is that our values are only determinate over Earthly things we are familiar with in our ancestral environment, and among Earthly things we empirically all kinda care about the same things. (I discuss this a bit here.)