Programme Director at ML4Good.
I also have done work on China policy, AI governance, and animal advocacy in Asia.
Also interested in effective giving (mainly animal charities), economic development (and how AI will affect it), AI x Animals, wild animal welfare, cause prioritisation, and various meta-EA topics.
Good question, but I implore you not to take this post too seriously. Itâs a real phenomenon, but itâs a real stretch to claim that this applies in the implied way to cause areas like AI safety.
The model in the article is just a toy model of a world where existential threats are randomly distributed according to a power law, where genuinely high-probability threats are, by assumption, basically absent from the space of possible threats, and where thereâs no process of updating based on evidence.
A more narrow claim like: âsingle narrowly defined x-risk estimates of genuinely speculative, unresearched causes are likely to be inflatedâ might be valid, but that doesnât seem to be what titotal is implying. Heâs making a claim way beyond anything implied by the model, even if the model were a valid representation of the phenomenon. He seems to believe that almost all of todayâs concern for AI risk is all downstream of a belief cultivated within a narrow subcommunity subject to the optimiserâs curseâan extraordinary claim that requires a lot more evidence than that supplied in the article.
The claim being made is something like:
Some time in the 2000s, Eliezer Yudkowsky and friends made up some numbers for AI risk
Community dynamics (rather than the merits of the arguments) spread these numbers from Bostrom to Tegmark to Sam Harris to Elon Musk to the EA community etc.
These social dynamics had such an effect that multiple seemingly independent and unrelated people/âexperts from Geoffrey Hinton to Yoshua Bengio to Chinese academics and tech people from completely different intellectual lineages have absorbed this false belief from the cultural milieu (again, not at all based on the merits of the arguments), leading many of these people, as well as specialists across unrelated fields, to make estimates of AI x-risk that remain orders of magnitude too high
Note that this requires some pretty wild, difficult-to-justify assumptions on how this belief has spread.
The opposing narrative (which I would advocate for) is that:
AI x-risk is something that has independently been identified as a risk by multiple peopleâoften far before quantifying or ranking risks.
The spread of these beliefs was obviously affected by community dynamics, but people largely adopted somewhat independent beliefs based on the merits of rational arguments
Quantitative predictions were inherently imprecise because theyâre so dependent on messy world models, but they were grounded enough in reason and evidence, and composed of enough independent estimates, that any optimiserâs curse is massively weakened
As certain predictions came to pass, and current LLMs approach AGI in non-ideal geopolitical and competitive circumstances, this is increasingly being seen by a wider range of thinkers as a >1% x-risk
Even if there were âoptimiser curseâ risks in initial prioritisation of AI, itâs now increasingly recognised that AI will be a massive deal. AI-assisted engineered pandemic uplift work, observed cyber-capabilities, and signs of misalignment/âscheming etc. are building on the strong theoretical evidence base that AI-generated catastrophe is possible.
And on your particular question of how to act, even given the optimiserâs curse as stated in the toy model, working on the speculative thing could still be optimal. If a highly uncertain intervention seems exceptionally promising or x-risky, the value of information becomes incredibly high, because accurate or well-reasoned research will lead to this intervention being prioritised or not by far more people. If you have a research focus, itâs therefore probably more recommended to focus on a more uncertain, âhigh-risk-high-rewardâ area. You could also draw up a toy model for explore/âexploit based on the optimiserâs curse.
Finally, you donât necessarily have to âpickâ from a narrow set of pre-defined cause areas. You can also divide or merge risks, cause areas, skill-sets etc. to be more robust, precise, or coherent with your own world model (e.g. focusing on engineered pandemics because you realise this could interact with AI-related x-risk, GCBRs, and global health).
Your piece went kinda viral on Substack, and most of the comments on the comments thread, and elsewhere on Substack, were constructive and positive. This seems a moderate win for alternative/âsocial media.
I would barely update at all on this paper. Effect sizes are very small and donât show a logical causal mechanism.
Looking at the full paper, in the fully adjusted matched model, smaller tattoos (under one hand palm) give you a higher risk (1.27) than larger ones (1.14). In the unmatched model, large tattoos had no effect. This seems contradictory with their suggested mechanism (ink affecting lymph nodes).
And their results imply that itâs most dangerous getting tattoos before thereâs a plausible chemical pathway to harm (risk is highest for those who got tattoos under 2 years ago). They say this could be a result of antagonising existing tumours, but seems likely to be a statistical artifact or a non causal correlation.
If you have other evidence, though, Iâd be happy to hear it.
I wrote a (draft amnesty) post on a very small subset of thisâwhat global health charities you should donate to given different worldviews.
I agree with the âyou donât have to debate on their termsâ point here, but I think for 99% of your readers/âlisteners, it cuts far more strongly in a different way than that youâre implying.
The debate has generally been set in terms of âAnthropic vs. DoWâ, and, while I know zero people in our community who have taken the governmentâs side on this, Iâve seen many EAs and adjacent people become increasingly uncritical supporters of Anthropic, just because theyâre standing against the obviously bad actor in this situation.
I think itâs important to remember:
If you thought Anthropic was at least somewhat untrustworthy before this, you shouldnât update too much the other wayâespecially when they backtracked on their RSP over the same period.
If you thought that Anthropicâs decision to join the race towards AGI was perilous, you shouldnât really update your view on this based on the Pentagon being absurd and unpredictable.
Regardless of the intention and character of the government actors, itâs potentially still a worrying sign that the most powerful state in the world has tried to shut down a frontier AI lab and failed spectacularly.
The growth and popularity of Anthropic and Claude Code have since caused the AI 2027 team to shorten their AGI timelines.
Excellent post, very clever humour!
Iâm relieved at the results.
Being forced by the data and our epistemic norms to say âwell, actually...â would not have been a good look for public-facing EAs.
Thanks for this post, itâs super valuable to get a better sense of this ecosystem.
On the apparent lack of Chinese companies, I think this is a methodological thing; a few possible blind spots:
Most obviously, English-language, web-based search is probably going to miss some Chinese AI-aquaculture innovators that would otherwise meet the paperâs inclusion criteria. Using Chinese-language platforms might be necessary.
AI for aquaculture in China is often embedded within broader, more integrated âsmart aquacultureâ systems rather than marketed as standalone AI products. e.g. Iâm not sure if Limap ĺąĺžéŤç§ came up on your search, but itâs a huge aquaculture and fisheries innovation company, deploying integrated aquaculture platforms across China, covering over 50 species etc. Some of their products and systems are explicitly AI-enabled & welfare relevant (e.g. computer-vision-based fish health and disease detection, visible in Chinese-language demos and videos), but they might have been excluded/âmissed because theyâre so broad.
Chinaâs innovation system is more state-led than elsewhere, and a lot of innovation happens through Universities, Agricultural Science and Technology Parks, local government programmes, âdemonstration basesâ 示čĺşĺ° etc. For example, Chinaâs Ministry of Agriculture and Rural Affairs recently released a 丝ć¨ććŻ (âmain promoted technologyâ) notice, announcing that AI-enabled âsmart aquaculture factoryâ technologies (including behaviour recognition, automated feeding, inspection robots, and large-model-based decision systems) are being supported through national programmes, implying a state-led deployment process. So you might be missing Chinese AI innovation in aquaculture thatâs not strictly commercial.
Iâd lean towards the World Happiness Report results here. IPSOS uses a fully online sample, which means you end up losing the âbottom halfâ of the population. World Happiness Report is phone and in-person.
Hi Klara, thanks for the response.
I donât think I am entering the abortion debate by assigning moral value to unborn lives any more than Iâm entering any other debate that considers unborn or potential lives (e.g. the ethics of moderate drinking while pregnant, the ethics of having children in space, or the repugnant conclusion).
I think Iâm comfortable with having mostly sidestepped the maternal health issues, given that I was focusing on interventions that are robustly good for the mother. If I were to do a stronger and more robust cost-effectiveness analysis, or tackle more controversial interventions where the interests of the mother and child clearly diverged, I would consider maternal health outcomes separately. I hope my piece makes it clear that we should prioritise uncontroversial and neglected interventions that treat or prevent painful conditions that women suffer from.
Although I do recognise that the ethics of pregnancy, lived experience of the mother, and autonomy trade-offs are important considerations, Iâm afraid that attempting to tackle these here would have made this an impossibly long post!
When I say âthe economics are looking good,â I mean that the conditions for capital allocation towards AGI-relevant work are strong. Enormous investment inflows, a bunch of well-capitalised competitors, and mass adoption of AI products means that, if someone has a good idea to build AGI within or around these labs, the money is there. It seems this is a trivial pointâif there were significantly less capital, then labs couldnât afford extensive R&D, hardware or large-scale training runs.
WRT Scaling vs. fundamental research, obviously âfundamental researchâ is a bit fuzzy, but itâs pretty clear that labs are doing a bit of everything. DeepMind is the most transparent about this, theyâre doing Gemini-related model research, Fundamental science, AI theory and safety etc. and have published thousands of papers. But Iâm sure a significant proportion of OpenAI & Anthropicâs work can also be classed as fundamental research.
I think there are two categories of answer here: 1) Finance as an input towards AGI, and 2) Finance as an indicator of AGI.
1) is the idea that the bubble popping would make it more difficult to develop AGI, because of financial constraints. Regardless of whether you think current LLM-based AI has fundamental flaws or not, the fact that insane amounts of capital are going into 5+ competing companies providing commonly-used AI products should be strong evidence that economics are looking good (edit: this avenue of R&D is perceived by multiple independent actors as providing promising returns) and that if AGI is technically possible using something like current tech, then all the incentives and resources are in place to find the appropriate architectures. If suddenly the bubble were to completely burst, even if we believed strongly that LLM-based AGI is imminent, there might be no more free money, so weâd now have an economic bottleneck to training new models.
In this scenario, weâd have to update our timelines/âestimates significantly (especially if you think straightforward scaling is a our likely pathway to AGI).
2) is the idea that the level of investment in AI provides signals about the near-term possibility of AGI, therefore the bubble bursting would be evidence that itâs less possible.
Whether or not this would change my mind depends on the situation. Financial markets are fickle enough that the bubble could pop for a bunch of reasons unrelated to current model trendsârare-earth export controls having an impact, something TSMC-related in Taiwan, slightly lower uptake figures, the decision of one struggling player (e.g. Meta) to leave the LLM space, or one highly-hyped but ultimately disappointing application, for example. If I was unsure of the reason, would I assume that the market knows something I donât? I might update slightly, but Iâm not sure to what extent Iâd trust the market to provide valuable information about AGI more than direct information about model capabilities and diffusion.
But of course, if we do update on market shifts, it has to be at least somewhat symmetrical. If a market collapse would slow down your timelines, insane market growth should accelerate your timelines for the same reason.
âLongtermism isnât necessary to think that x-risk (of at least some varieties) is a top priority problem.â
I donât think itâs a niche viewpoint in EA to think that, mainly because of farmed and wild animal suffering, the short term future is net-negative in expectation, but the long-term future could be incredibly good. This means that some variety of longtermism is essential in order to not embrace x-risk in our lifetimes as desirable.
I definitely identify with where youâre coming from here, but these insights might also imply a potential partner post on âHow to avoid EA senescence (if you want to)â.
Based on your examples, this might look like:
Specialise, even if itâs not your jobâdive very deep into at least one relevant EA area. If you can find something you find interesting and neglected, can you become top 1% knowledgeable (within EA) in an obscure sub-field?
Develop (and share) a niche perspective on where to donate based on your specific worldview. If youâre very convinced about insect sentience, or you lean negative utilitarian, you will very quickly realise that EA Funds are not the highest EV option for you!
Prioritise boosting/âmaintaining your âEA energyâ
Host more parties
Thanks for the post! This is a very valuable topic, and the development econ mainstream is totally lost on this question!
I agree with some of your points, but I think we need to distinguish very carefully between âdeveloping countriesâ. All the factors you mention with regards to labour displacement (structure of the economy, data availability, telecommunication infra) are wildly different between, say, Togo, Brazil, and Indonesia. Same with private- vs. public sector diffusion; within âdeveloping countriesâ, youâve got countries with massive tech hubs and their own tech billionaires, and those where most people still donât have electricity.
For me, the most important development question with regards to TAI (and the reason itâs important to distinguish) is the feasibility of the export-led development model. Generally, if countries manage to develop a high-value added export sector, they attract FDI, get foreign currency, climb up the value chain, and become richer. If they donât, they stay poor. Except for the occasional country finding insane levels of natural resources, this is the only real way that countries have become rich over the last 100 years.
If we get safe, transformative AI, we can imagine that demand for imports massively rises in the West, and middle-income countries like China, Vietnam, and Indonesia with strong export sectors (and the infrastructure to build on their existing exports) are able to take advantage of this. As these countries already have good infrastructure (e.g. electrification, internet access, land and shipping transport) they can probably also benefit from AI & Robotics to develop âIndustry 4.0â and make their export sector even more dominant.
Iâd therefore estimate that a few of these âdevelopingâ countries with existing strong export sectors will catch-up and become rich relatively soon.
But what of the poorest countries?!
Most African countries with a GDP below, say $3000 are very low down the value chain in all sectors, with little but raw materials (e.g. coffee, cocoa, oil if they have it) as exports. Theyâre struggling to compete with Asian developing powerhouses, and they havenât got the transport infrastructure, governance, or capital etc. to develop a quality export-led economy. In a world without TAI, as middle-income countries get rich, poor countries would develop the export industries and climb the ladder themselves, but this seems very unlikely with displacement of manufacturing labour by robotics.
My overall take is that (in an optimistic AI scenario) well-governed middle-income countries would probably end up more similar to rich countries. But weâd have really âkicked away the ladderâ from the very poor countries.
My ideas for posts (Iâll try to write at least one):
I recently learned that malaria causes about as many miscarriages and stillbirths as it causes live infant deaths, but we only count neonatal deaths in most cost-effectiveness estimates. Intermittent preventive treatment with Sulphadoxine-Pyrimethamine (IPTp-SP) for pregnant women seems to be more cost-effective than bed-nets for preventing malaria-related stillbirths and miscarriage. Unsure whether to write a narrow post on that, or a deeper post on âWhat are the most effective charities, given worldviews where unborn children have similar value to new-borns?â
Some Europeans have been asking me a lot about what people in smaller countries can do to make AI go better (or slow down) - especially with regards to China. I think weâve got a lot of lessons from (especially Cold War) history about third countries using their relations with superpowers to increase existential safety, but I donât think anyoneâs written an EA forum post about it.
I wrote a blog post on what I call âThe Great Happiness Stagnationââlooking at the flattening of happiness in many rich countries since they became rich. Iâve been thinking about converting it to a forum post, but it currently seems insufficiently rigorous to be worthy of the forum!
Agree with your point about the Chinese study reference, about healthy aging for elderly Chinese people. The OP uses it to make three separate points, about cognitive impairment, dose-response effects and lower overall odds of healthy aging, but itâs pretty clear that the study is basically showing the effects of poverty on health in old age.
Elderly Chinese people are mostly vegetarian or vegan because a) they canât afford meat, or b) have stopped eating meat because they struggle with other health issues, both of which would massively bias the outcomes! So their poor outcomes might be partly through diet-related effects, like nutrient/âprotein deficiency, but could also be sanitation, malnutrition in earlier life (these are people brought up in extreme famines), education (particularly for the cognitive impairment test), and the health issues that cause them to reduce meat.
The study fails to control for extreme poverty by grouping together everyone who earned <8000 Yuan a year (80% of the survey sample!), which is pretty ridiculous, because the original dataset should have continuous data...
The paper also makes it very clear that diet quality is the real driver, and that healthy plant-based diets score similarly to omnivorous diets âwith vegetarians of higher diet quality not significantly differing in terms of overall healthy aging and individual outcomes when compared to omnivoresâ.
Probably less importantly, it conditions on survival to 80, which creates a case of survivorship bias/âcollider bias. So there could be a story where less healthy omnivores tend to die earlier (you get effects like this with older smokers, sometimes), and the survivors appear healthier.
At ML4Good (where I work) our bootcamps function as a kind of on-ramp. Weâve had a bunch of our mid-career alumni move into AI safety after attending our gov & strategy bootcamps. A few takes:
Except for slight adjustments in marketing, we havenât had to try too hard getting MCPs! LinkedIn, 80k job board, BlueDot and alumni referrals are some of our main sources of participants, and we get lots of promising candidates with limited targeted outreach. This does mean weâre selecting for people with a few toes in the space already.
Selection can be difficult with mid-career people. It can be difficult to determine how much to weight experience and expertise vs alignment and context. Sometimes itâs obvious, thoughâwe get some ideal applicants, who have genuine expertise in a neglected field (e.g. chip hardware, tech policy, law), are super aligned with EA/âAIS values, and need precisely what we can provide to make a transition
These bootcamps are 8 days, so they require both some existing commitment and a not-crazily-busy life
Weâre also testing a new programme targeting legal professionals (https://ââfrontier-ai-law.org/ââ) and recruiting participants has been a little bit more like the âheadhuntingâ approach you mentionâtargeted outreach to senior lawyers. Weâve found that there are loads of senior people who would attend a programme to help them upskill in AI Safety (broadly defined) so that they can do their job better, without explicitly nudging them to make a transition, so weâre excited about this approach as an on-ramp.
Happy to chat in more detail!