Tom_Davidson

• Tom_Davidson 16 Dec 2025 16:34 UTC

  2 points

  0 ∶ 0

  on: Rerun­ning the Time of Perils

  I think my biggest uncertainty about this is:

  If there were a catastrophic setback of this kind, and civilisation tried hard to save and maintain the weights of superintelligent AI (which they presumably would), how likely are they to succeed?

  My hunch is that they very likely could succeed. E.g. in the first cpl of decades they’d have continued access to superintelligent AI advice (and maybe robotics) from pre-existing hardware. They could use that to bootstrap to longer periods of time. E.g. saving the weights on hard drives rather than SSDs, and then later transferring them to a more secure, long lasting format. Then figure out the minimal-effort-version of compute maintenance and/​or production needed to keep running some superintelligences indefinitely

• Tom_Davidson 16 Dec 2025 16:28 UTC

  7 points

  1 ∶ 0

  on: Rerun­ning the Time of Perils

  Really like this post!

  I’m wondering whether human-level AI and robotics will significantly decrease civilisation’s susceptibility to catastrophic setbacks?

  AI systems and robots can’t be destroyed by pandemics. They don’t depend on agriculture—just mining and some form of energy production. And a very small number of systems could hold tacit expertise for ~all domains.

  Seems like this this might reduce the risk by a lot, such that the 10% numbers you’re quoting are too high. E.g. you’re assigning 10% to a bio-driven set-back. But i’d have thought that would have to happen before we get human-level robotics?

• Tom_Davidson 16 Oct 2025 17:45 UTC

  4 points

  0 ∶ 0

  on: A per­sonal take on why you should work at Forethought (maybe)

  I also work at Forethought!

  I agree with a lot of this, but wanted to flag that I would be very excited for ppl doing blue skies research to apply and want Forethought to be a place that’s good for that. We want to work on high impact research and understand that sometimes mean doing things where it’s unclear up front if it will bear fruit.

• Tom_Davidson 11 Aug 2025 16:44 UTC

  4 points

  0 ∶ 0

  in reply to: OscarD🔸’s comment on: How quick and big would a soft­ware in­tel­li­gence ex­plo­sion be?

  Thanks, good Q.

  I’m saying that if there is such a new paradigm then we could have >10 years worth of AI progress at rates of 2020-5, and >10 OOMs of effective compute growth according to the old paradigm. But that, perhaps, within the new paradigm these gains are achieved while the efficiency of AI algs only increases slightly. E.g. a new paradigm where each doubling of compute increases capabilities as much as 1000X does today. Then, measured within the new paradigm, the algorithmic progress might seem like just a couple of OOMs so ‘effective compute’ isn’t rising fast, but relative to the old paradigm progress (and effective compute growth) is massive.

  Fwiw, this X thread discusses the units I’m using for “year of Ai progress”, and Eli Lifland gives a reasonable alternative. Either work as a way to understand the framework.

• Tom_Davidson 7 Jun 2025 15:15 UTC

  5 points

  0 ∶ 0

  on: The Soul of EA is in Trouble

  Can you give more examples of where ppl are getting this wrong?

  I support the 80k pivot, and the blue dot page seems ok (but yes, I’d maybe prefer smg more opinionated).

  While these concerns make sense in theory I’m not sure whether it’s a problem in practice

• Tom_Davidson 3 Jun 2025 7:51 UTC

  5 points

  1 ∶ 0

  in reply to: Parker_Whitfill’s comment on: Es­ti­mat­ing the Sub­sti­tutabil­ity be­tween Com­pute and Cog­ni­tive La­bor in AI Research

  Nice!

  I think that condition is equivalent to saying that A_cog explodes iff either

  • phi_cog + lambda > 1 and phi_exp + lambda > 1, or

  • phi_cog > 1

  Where the second possibility is the unrealistic one where it could explode with just human input

• Tom_Davidson 2 Jun 2025 17:24 UTC

  6 points

  0 ∶ 0

  in reply to: Owen Cotton-Barratt’s comment on: Es­ti­mat­ing the Sub­sti­tutabil­ity be­tween Com­pute and Cog­ni­tive La­bor in AI Research

  Agree that i wouldn’t particularly expect the efficiency curves to be the same.

  But if the phi>0 for both types of efficiency, then I think this argument will still go through.

  To put it in math, there would be two types of AI software technology, one for experimental efficiency and one for cognitive labour efficiency: A_exp and A_cog. The equations are then:

  dA_exp = A_exp^phi_exp F(A_exp K_res, A_cog K_inf)

  dA_cog = A_cog^phi_cog F(A_exp K_res, A_cog K_inf)

  And then I think you’ll find that, even with sigma < 1, it explodes when phi_exp>0 and phi_cog>0.

• Tom_Davidson 2 Jun 2025 17:19 UTC

  2 points

  0 ∶ 0

  in reply to: Parker_Whitfill’s comment on: Es­ti­mat­ing the Sub­sti­tutabil­ity be­tween Com­pute and Cog­ni­tive La­bor in AI Research

  Although note that this argument works only with the CES in compute formulation. For the CES in frontier experiments, you would have the $\frac{A{K}_{res}}{A{K}_{train}}$ so the A cancels out.

  Yep, as you say in your footnote, you can choose to freeze the frontier, so you train models of a fixed capability using less and less compute (at least for a while).

• Tom_Davidson 2 Jun 2025 11:42 UTC

  8 points

  2 ∶ 0

  on: Es­ti­mat­ing the Sub­sti­tutabil­ity be­tween Com­pute and Cog­ni­tive La­bor in AI Research

  However, if $\sigma <1$, then a software-only intelligence explosion occurs only if $\varphi >1$. But if this condition held, we could get an intelligence explosion with constant, human-only research input. While not impossible, we find this condition fairly implausible.

  Hmm, I think a software-only intelligence explosion is plausible even if $\sigma <1$, but without the implication that you can do it with human-only research input.

  The basic idea is that when you double the efficiency of software, you can now:

  • Run twice as many experiments

  • Have twice as much cognitive labour

  So both the inputs to software R&D double.

  I think this corresponds to:

  dA = A^phi F(A K_res, A K_inf)

  And then you only need phi > 0 to get an intelligence explosion. Not phi > 1.

  This is really an explosion in the efficiency at which you can run AI algorithms, but you could do that for a while and then quickly use your massive workforce to develop superintelligence, or start training your ultra-efficient algorithms using way more compute.

• Tom_Davidson 2 Jun 2025 11:36 UTC

  17 points

  4 ∶ 0

  on: Es­ti­mat­ing the Sub­sti­tutabil­ity be­tween Com­pute and Cog­ni­tive La­bor in AI Research

  Thanks for this!

  Let me try and summarise what I think is the high-level dynamic driving the result, and you can correct me if I’m confused.

  CES in compute.

  Compute has become cheaper while wages have stayed ~constant. The economic model then implies that:

  • If compute and labour were complements, then labs would spend a greater fraction of their research budgets on labour. (This prevents labour from becoming a bottleneck as compute becomes cheaper.)

  Labs aren’t doing this, suggesting that compute and labour are substitutes.

  CES in frontier experiments.

  Frontier experiments have become more expensive while wages have stayed ~constant. The economic model then implies that:

  • If compute and labour were complements, then labs would spend a greater fraction of their research budgets on compute. (This relieves the key bottleneck of expensive frontier experiments.)

  Labs are indeed doing this, suggesting that compute and labour are indeed complements.

  (Though your ‘Research compute per employee’ data shows they’re not doing that much since 2018, so the argument against the intelligence explosion is weaker here than I’d have expected.)

• Tom_Davidson 2 Jun 2025 11:06 UTC

  5 points

  0 ∶ 0

  on: Es­ti­mat­ing the Sub­sti­tutabil­ity be­tween Com­pute and Cog­ni­tive La­bor in AI Research

  The $\sigma >1$ condition is exactly what Epoch and Forethought consider when they analyze whether the returns to research are high enough for a singularity.^[5]

  Though we initially consider this, we then adjust for compute as an input to R&D and so end up considering the sigma=1 condition. It’s under that condition that I think it’s more likely than not that the condition for a software-only intelligence explosion holds

• Tom_Davidson 18 Mar 2025 22:18 UTC

  6 points

  1 ∶ 0

  on: Donat­ing more and bet­ter is the best strat­egy to max­imise im­pact for the vast ma­jor­ity of peo­ple work­ing in im­pact-fo­cussed or­gani­sa­tions?

  So you think ppl doing direct work should quit and earn to give if they could thereby double their salary? Can’t be the right recommendation for everyone!

• Tom_Davidson 26 Sep 2024 12:46 UTC

  1 point

  0 ∶ 0

  on: How much is 1.8 mil­lion years of work?

  I like the vividness of the comparisons!

  A few points against this being nearly as crazy as the comparisons suggest:

  • GPT-2030 may learn much less sample efficiently, and much less compute efficiently, than humans. In fact, this is pretty likely. Ball-parking, humans do 1e24 FLOP before they’re 30, which is ~20X less than GPT-4. And we learn languages/​maths from way fewer data points. So the actual rate at which GPT-2030 itself gets smarter will be lower than the rates implied.

    • This is a sense of “learn” as in “improves its own understanding”. There’s another sense which is “produces knowledge for the rest of the world to use, eg science papers” where I think your comparisons are right.

  • Learning may be bottlenecked by serial thinking time past a certain point, after which adding more parallel copies won’t help. This could make the conclusion much less extreme.

  • Learning may also be bottlenecked by experiments in the real world, which may not immediately get much faster.

• Tom_Davidson 26 Sep 2024 12:38 UTC

  2 points

  1 ∶ 0

  in reply to: Morgan_Rivers’s comment on: Take­off speeds pre­sen­ta­tion at Anthropic

  Thanks, this is a great comment.

  The first and second examples seems pretty good, and useful reference points.
  
  The third example don’t seem like they are nearly as useful though. What’s particularly unusual about this case is that there are two useful inputs to AI R&D—cognitive labour and compute for experiments—and the former will rise very rapidly but the other will not. In particular, I imagine CS departments also saw compute inputs growing in that time. And I imagine some of the developments discussed (eg proofs about algorithms) only have cognitive labour as an input.

  The second example (quant finance), I suppose the ‘data’ input to doing this work stayed constant while the cognitive effort rose. So it works as an example. Though it may be a field with an unusual superabundance of data, unlike ML.
  
  The first example involves a kind of ‘data overhang’ that the cognitive labour quickly eats up. Perhaps in a similar way AGI will “eat up” all the insights that are implicit in existing data from ML experiments.

  What i think all the examples currently lack is a measure of how the pace of overall progress changed that isn’t completely made up. Could be interested to list out the achievements in each time period and ask some experts what they think. There an interesting empirical project here I think.

  All the examples also lack anything like the scale to which cognitive labour will increase with AGI. This makes comparison even harder. (Though if we can get 3X speed-ups from mild influxes of cognitive labour, that makes 10X speed ups more plausible.)

  I tried to edit the paragraph (though LW won’t let me) to:
  
  I think we don’t know what perspective is right, we haven’t had many examples where a huge amount of cognitive labour has been dumped on a scientific field and other inputs to progress have remained constant and we’ve accurately measured how much overall progress in that field accelerates. (Edit: though this comment suggests some interesting examples.)

• Tom_Davidson 17 Aug 2024 16:58 UTC

  1 point

  0 ∶ 0

  in reply to: Ryan Greenblatt’s comment on: Cos­mic NIMBYs and the Repug­nant Conclusion

  • I think utilitarianism is often a natural generalization of “I care about the experience of XYZ, it seems arbitrary/​dumb/​bad to draw the boundary narrowly, so I should extend this further” (This is how I get to utilitarianism.) I think the AI optimization looks considerably worse than this by default.

  Why is this different between AIs and humans? Do you expect AIs to care less about experience than humans, maybe bc humans get reward during life-time learning about AIs don’t get reward during in context learning?

• Tom_Davidson 17 Aug 2024 16:55 UTC

  1 point

  0 ∶ 0

  in reply to: Ryan Greenblatt’s comment on: Cos­mic NIMBYs and the Repug­nant Conclusion

  • I can directly observe AIs and make predictions of future training methods and their values seem to result from a much more heavily optimized and precise thing with less “slack” in some sense. (Perhaps this is related to genetic bottleneck, I’m unsure.)

  Can you say more about how slack (or genetic bottleneck) would affect whether AIs have values that are good by human lights?

• Tom_Davidson 17 Aug 2024 16:54 UTC

  1 point

  0 ∶ 0

  in reply to: Ryan Greenblatt’s comment on: Cos­mic NIMBYs and the Repug­nant Conclusion

  • AIs will be primarily trained in things which look extremely different from “cooperatively achieving high genetic fitness”.

  They might well be trained to cooperate with other copies on tasks, if this is they way they’ll be deployed in practice?

• Tom_Davidson 17 Aug 2024 16:52 UTC

  1 point

  0 ∶ 0

  in reply to: Ryan Greenblatt’s comment on: Cos­mic NIMBYs and the Repug­nant Conclusion

  • Current AIs seem to use the vast, vast majority of their reasoning power for purposes which aren’t directly related to their final applications. I predict this will also apply for internal high level reasoning of AIs. This doesn’t seem true for humans.

  In what sense do AIs use their reasoning power in this way? How that that affect whether they will have values that humans like?

• Tom_Davidson 2 May 2023 18:59 UTC

  27 points

  7 ∶ 1

  in reply to: titotal’s comment on: The costs of caution

  I agree that bottlenecks like the ones you mention will slow things down. I think that’s compatible with this being a “jump in forward a century” thing though.

  Let’s consider the case of a cure for cancer. First of all, even if it takes “years to get it out due to the need for human trials and to actually build and distribute the thing” AGI could still bring the cure forward from 2200 to 2040 (assuming we get AGI in 2035).

  Second, the excess top-quality labour from AGI could help us route-around the bottlenecks you mentioned:

  • Human trials: AGI might develop ultra-high-reliability ways to verify that drugs work without human trials. That could either lead to a change in regulatory requirements or to people buying the AGI-designed drugs sooner in countries where that’s legal.

  • Manufacturing and distributing the drug: Imagine if we’d had 100 million of the most competent humans working (remotely) full time on optimising every step of the manufacturing+distribution process for COVID? They could have:

    • Planned out how to use all the US’ available manufacturing and transportation infrastructure maximally efficiently

    • Give real-time instructions to all the humans working in those industries so that they were more productive and better coordinated.

    • Recruit and train of new human workers (again instructing them in real-time) to increase the available labour.

    • More speculatively, it might not take long for AGI to design robots that could do the physical labour needed to manufacture and distribute the vaccines.

• Tom_Davidson 25 Apr 2023 1:11 UTC

  12 points

  6 ∶ 0

  in reply to: Jakob’s comment on: No, the EMH does not im­ply that mar­kets have long AGI timelines

  It seems to me like you disagree with Carl because you write:

  • The reason for an investor to make a bet, is that they believe they will profit later

  • However, if they believe in near-term TAI, savvy investors won’t value future profits (since they’ll be dead or super rich anyways)

  • Therefore, there is no way for them to win by betting on near-term TAI

  So you’re saying that investors can’t win from betting on near-term TAI. But Carl thinks they can win.