Anything I write here is written purely on my own behalf, and does not represent my employerâs views (unless otherwise noted).
Erich_Grunewald đ¸
Karma: 2,862
I agree that animal welfare is underfunded, but who is this post trying to convince? If funders should fund the most cost-effective interventions or cause areas, then the argument needs to be made on the merits, which are not covered here. Animals donât just automatically win because there is lots of animal suffering today (unless maybe you are a fairly near-termist negative utilitarian).
Did you have any system prompt or other instructions active when you asked these things? As someone else mentioned in another comment, incognito mode just means that Anthropic doesnât save the chat, but your general instructions for Claude are still accessible to it in incognito mode.
I agree that the Forum is (fairly) dead; I donât read it much anymore, and feel somewhat sad about this. I also think a lot of energy has moved to Substack, and would guess that this is a stronger effect than LessWrong cross-posting.
Itâs true that technological progress so far has been largely good for humans and bad for animals (due to factory farms, but the effects on wild animals complicate this a lot).
But I also think human values towards animals have improved compared to how they were historically, and e.g., house and work animals are likely treated better on the whole now than in the past. So I think thereâs been some moral progress, but this improvement has been dominated by technology simultaneously making animal food production much more cost-effective (and, as a side-effect, more suffering-producing).
I think eventually technological progress will make it cheaper to act on animal-friendly values, because Iâm guessing the taste/âprice/âconvenience/âfriction of animal meat is hitting diminishing returns, while thereâs much more room for improvement with non-animal-based foods. So I think there will be a crossover point at some point, sort of the way it was with the Enlightenment and the industrial revolution, where the economic effects of technology first made many people probably worse off, but eventually the better values won out and people were left better off on the whole.
I separately also think AGI and especially ASI, if aligned with human values, can wisely advise us on good courses of actions and help improve our values and promote good values, which would also help. ASI could also help a lot with wild animal welfare, where we are currently quite at a loss.
I think this is beautiful and touching and a little scary.
Yet nobody is building the I-O equivalent for AI compute governance.
Iâm kind of skeptical that the data exists to create a good input-output model for AI chip export controls. For example, we know very little about which companies (especially in China) rent which AI chips from which cloud providers, and in which volumes; about any of the substation elasticities; or about smuggling volumes. We know more about AI chip sales now, but still little about which companies are buying them. But Iâd be very interested in knowing if Iâm wrong about this lack of data being a problem for input-output models?
Also, I feel quite confident, based on style, that this post was at least partially AI-written, in a way that made me like it less and take it less seriously. Make of that what you will!
A much better order of operations would be to 1) try to negotiate with China to establish an international regulatory framework (plan A), with export control and other stuff being imposed as something that is explicitly linked to China not agreeing to that framework, in the same way sanctions on Russia are imposed explicitly because its aggression against Ukraine, and 2) only if they refuse, try to crush them (plan B).
Maybe if you are President of the United States you can first try the one thing, and then the other. But from the perspective of an individual, you have to assume thereâs some probability of each of these plans (and other strategies) being executed, and that everything will be really messy (e.g., different actors having different strategies in mind, even within the US). Softening export controls seems like something you could do as part of executing Plan A, but as I mentioned above, itâs very unclear to me whether unilaterally doing so makes Plan A more likely to be the chosen strategy, and it does likely make Plan B and Plan C go worse.
When political will in the US to try for plan A is lacking, I think waiting until circumstances make that plan realistic while preparing the groundwork for it is a better strategy than going straight ahead for plan B.
I think youâre thinking people have more control over which strategy is adopted than I think they do? Or, what circumstances do you have in mind? Because waiting seems pretty costly.
But I think maybe the cruxiest bits are (a) I think export controls seem great in Plan B/âC worlds, which seem much likelier than Plan A worlds, and (b) I think unilaterally easing export controls is unlikely to substantially affect the likelihood of Plan A happening (all else equal). It seems like you disagree with both, or at least with (b)?
I think if your sole objective is to enact a bilateral pause, then easing export controls may be the best option, or maybe not. Itâs pretty unclear to me how that shakes out, I could definitely also see unilateral concessions as being quite detrimental (for reasons similar to those Peter mention in the other comment).
But I would guess most of the people you are responding to think enacting a cooperative pause is some combination of very unlikely and/âor undesirable, and also that export controls help a lot in the absence of such an agreement. The main way export controls help for other plans are by giving the US more slack to (one can hope) spend on safety, and/âor because superintelligence developed in the US would imo likely be safer (cf. this comment), and/âor because imo US values are better, and this would likely be reflected in the AIs (cf. Claude versus Grok).
If you thought Yudkowsky and Soares used overly confident language and would have taken the âQEDâ as further evidence of that, but this particular example turns out not to have been written by Yudkowsky and Soares, thatâs some evidence against your hypothesis. But instead of updating away a little, you seemed to dismiss that evidence and double down. (I think you originally replied to the original comment approvingly or at least non-critically, but then deleted that comment after I replied to it, but I could be misremembering that.)
For what itâs worth, I think youâre right that Yudkowsky at least uses overly confident language sometimesâor I should say, is overly confident sometimes, because I think his language generally reflects his beliefsâbut I wouldâve been surprised to see him use âQEDâ in that way, which is why I reacted to the original comment here with skepticism and checked whether âQEDâ actually appeared in the book (it didnât). I take that to imply I was better calibrated than anyone who did not so react.
Interesting!
Given that these failures were predictable, it should be possible to systematically predict many analogous failures that might result from training AI systems on specific data sets or (simulated) environments.
Your framework seems to work for simple cases like âice cream, sucralose, or sex with contraceptionâ, but I donât think it works for more complex cases like âpeacocks would like giant colorful tailsâ?
There is so much human behaviour also that would have been essentially impossible to predict just from first principles and natural selection under constraints: poetry, chess playing, comedy, monasticism, sports, philosophy, effective altruism. These behaviours seem further removed from your detectors for instrumentally important subgoals, and/âor to have a more complex relationship to those detectors, but theyâre still widespread and important parts of human life. This seems to support the argument that the relationship between how a mind was evolved (e.g., by natural selection) and what it ends up wanting is unpredictable, possibly in dangerous ways.
Your model might still tell us that generalisation failures are very likely to occur, even if, as I am suggesting, it canât predict many of the specific ways things will misgeneralise. But Iâm not sure this offers much practical guidance when trying to develop safer AI systems. But maybe Iâm wrong about that?
Who are âtheyâ? If you mean Yudkowsky and Soares, âQEDâ is something that Hanson (the author of this critique) includes in his paraphrase of Yudkowsky and Soares, but I donât think itâs anything Yudkowsky and Soares wrote in their book. The quoted argument is not actually a quote, but a paraphrase.
For what itâs worth, I would guess that though the âfunnessâ of AI safety research, or maybe especially technical AI safety research, is probably a factor in determining how many people are interested in working on that, I would be surprised if itâs a factor in determining how much money is allocated towards that as a field.
(To be clear, I do think many of these charities do some good and are run with the best of intentions, etc. But I still also stand by the statement in the parent comment.)
That is the most PR-optimized list of donations I have ever seen in my life.
And also to include the link? Or maybe Iâm too dumb to see it.
Thanks for sharing this. I did an Erasmus exchange year in Italy in 2010-11 that was very important for my personal growth, although it was not particularly beneficial professionally or academically.
Nice work!
On AI chip smuggling, rather than the report you listed, which is rather outdated now, I recommend reading Countering AI Chip Smuggling Has Become a National Security Priority, which is essentially a Pareto improvement over the older one.
I also think Chris Millerâs How US Export Controls Have (and Havenât) Curbed Chinese AI provides a good overview of the AI chip export controls, and it is still quite up-to-date.
On timelines, I think itâs worth separating out export controls on different items:
Controls on AI chips themselves start having effects on AI systems within a year or so probably (say 6-12 months to procure and install the chips, and 6-18 months to develop/âtrain/âpost-train a model with them), or even sooner for deployment/âinference, i.e. 1-2 years or so.
Controls on semiconductor manufacturing equipment (SME) take longer to have an impact as you say, but I think not that long. SMIC (and therefore future Ascend GPUs) is clearly limited by the 2019 ban on EUV photolithography, and I would say this was apparent as early as 2023. So I think SME controls instituted now would start having an effect on chip production in the late 2020s already, and on AI systems 1-2 years after that.
Most other relevant products (e.g., HBM and EDA software) probably fall between those two in terms of how quickly controls affect downstream AI systems.
So that means policy changes in 2025 could start affecting Chinese AI models in 2027 (for chips) and around 2030 (for SME) already, which seems relevant to even short-timeline worlds. For example, Daniel Kokotajloâs median for superhuman coders is now 2029, and IIUC Eli Liflandâs median is in the (early?) 2030s.
But I would go further to say that export controls now can substantially affect compute access well into the 2030s or even the 2040s. You write that
the technical barriers [to Chinese indigenization of leading-edge chip fabrication] are higher today, but not so high that intense Chinese investment canât dent it over the course of a decade. SMEE is investing in laser-induced discharge plasma tech, with rumored trial production as soon as the end of this year. SMIC is using DUV more efficiently for (lower-yield, but still effective) chip production. Thereâs also work on Nanoimprint lithography, immersion lithography, packaging, etc. And that wonât affect market shares, until it does.
I wonât have time to go into great detail here, but I have researched this a fair amount and I think you are too bullish on Chinese leading-edge chip fabrication. To be clear, China can and will certainly produce AI chips, and these are decent AI chips. But they will likely produce those chips less cost-efficiently and at lower volumes due to having worse equipment, and they will have worse performance than TSMC-fabbed chips due to using older-generation processes. The lack of EUV machines, which will likely last at least another five years and plausibly well into the 2030s, alone is a very significant constraint.
On SMEE and SMIC in particularâyou write:
SMEE is investing in laser-induced discharge plasma tech, with rumored trial production as soon as the end of this year.
SMEE was established 23 years ago to produce indigenous lithography, and 23 years later it still has essentially no market share, and it still has not produced an immersion DUV machine, let alone an EUV machine, which is far more difficult. I would not be surprised if, when the indigenous Chinese immersion DUV machine does finally arrive, it is a SiCarrier (or subsidiary) product and not an SMEE product.
SMIC is using DUV more efficiently for (lower-yield, but still effective) chip production.
In what sense do you mean SMIC is using DUV more efficiently? It is using immersion DUV multi-patterning (with ASML machines) to compensate for its lack of EUV machines. But as you note this means worse yield and lower throughput. I donât see any sense in which SMIC is using DUV more efficiently; itâs just using it more, in order to get around a constraint that TSMC doesnât have. In any case, multi-patterning with immersion DUV can only take you so far; thereâs likely a hard stop around whatâs vaguely called 2 nm or 1.4 nm process nodes, even if you do multi-patterning perfectly. (For reference, TSMC is starting mass production on its â2 nmâ process this year.)
(I have already read these posts and much prefer them over the post weâre commenting on here.)