Programme Director at ML4Good. I also do some work on China policy, AI governance, and animal advocacy in Asia.
Also interested in effective giving (mainly animal charities), economic development (and how AI will affect it), AI x Animals, wild animal welfare, cause prioritisation, and various meta-EA topics.
Excellent post!
Iâm relieved at the results.
Being forced by the data and our epistemic norms to say âwell, actually...â would not have been a good look for public-facing EAs.
Thanks for this post, itâs super valuable to get a better sense of this ecosystem.
On the apparent lack of Chinese companies, I think this is a methodological thing; a few possible blind spots:
Most obviously, English-language, web-based search is probably going to miss some Chinese AI-aquaculture innovators that would otherwise meet the paperâs inclusion criteria. Using Chinese-language platforms might be necessary.
AI for aquaculture in China is often embedded within broader, more integrated âsmart aquacultureâ systems rather than marketed as standalone AI products. e.g. Iâm not sure if Limap ĺąĺžéŤç§ came up on your search, but itâs a huge aquaculture and fisheries innovation company, deploying integrated aquaculture platforms across China, covering over 50 species etc. Some of their products and systems are explicitly AI-enabled & welfare relevant (e.g. computer-vision-based fish health and disease detection, visible in Chinese-language demos and videos), but they might have been excluded/âmissed because theyâre so broad.
Chinaâs innovation system is more state-led than elsewhere, and a lot of innovation happens through Universities, Agricultural Science and Technology Parks, local government programmes, âdemonstration basesâ 示čĺşĺ° etc. For example, Chinaâs Ministry of Agriculture and Rural Affairs recently released a 丝ć¨ććŻ (âmain promoted technologyâ) notice, announcing that AI-enabled âsmart aquaculture factoryâ technologies (including behaviour recognition, automated feeding, inspection robots, and large-model-based decision systems) are being supported through national programmes, implying a state-led deployment process. So you might be missing Chinese AI innovation in aquaculture thatâs not strictly commercial.
Iâd lean towards the World Happiness Report results here. IPSOS uses a fully online sample, which means you end up losing the âbottom halfâ of the population. World Happiness Report is phone and in-person.
Hi Klara, thanks for the response.
I donât think I am entering the abortion debate by assigning moral value to unborn lives any more than Iâm entering any other debate that considers unborn or potential lives (e.g. the ethics of moderate drinking while pregnant, the ethics of having children in space, or the repugnant conclusion).
I think Iâm comfortable with having mostly sidestepped the maternal health issues, given that I was focusing on interventions that are robustly good for the mother. If I were to do a stronger and more robust cost-effectiveness analysis, or tackle more controversial interventions where the interests of the mother and child clearly diverged, I would consider maternal health outcomes separately. I hope my piece makes it clear that we should prioritise uncontroversial and neglected interventions that treat or prevent painful conditions that women suffer from.
Although I do recognise that the ethics of pregnancy, lived experience of the mother, and autonomy trade-offs are important considerations, Iâm afraid that attempting to tackle these here would have made this an impossibly long post!
When I say âthe economics are looking good,â I mean that the conditions for capital allocation towards AGI-relevant work are strong. Enormous investment inflows, a bunch of well-capitalised competitors, and mass adoption of AI products means that, if someone has a good idea to build AGI within or around these labs, the money is there. It seems this is a trivial pointâif there were significantly less capital, then labs couldnât afford extensive R&D, hardware or large-scale training runs.
WRT Scaling vs. fundamental research, obviously âfundamental researchâ is a bit fuzzy, but itâs pretty clear that labs are doing a bit of everything. DeepMind is the most transparent about this, theyâre doing Gemini-related model research, Fundamental science, AI theory and safety etc. and have published thousands of papers. But Iâm sure a significant proportion of OpenAI & Anthropicâs work can also be classed as fundamental research.
I think there are two categories of answer here: 1) Finance as an input towards AGI, and 2) Finance as an indicator of AGI.
For 1) regardless of whether you think current LLM-based AI has fundamental flaws or not, the fact that insane amounts of capital are going into 5+ competing companies providing commonly-used AI products should be strong evidence that the economics are looking good, and that if AGI is technically possible using something like current tech, then all the incentives and resources are in place to find the appropriate architectures. If suddenly the bubble were to completely burst, even if we believed strongly that LLM-based AGI is imminent, there might be no more free money, so weâd now have an economic bottleneck to training new models. In this scenario, weâd have to update our timelines/âestimates significantly (especially if you think straightforward scaling is a our likely pathway to AGI).
For 2), probably notâdepends on the situation. Financial markets are fickle enough that the bubble could pop for a bunch of reasons unrelated to current model trendsârare-earth export controls having an impact, slightly lower uptake figures, the decision of one struggling player (e.g. Meta) to leave the LLM space, or one highly-hyped but ultimately disappointing application, for example. If I was unsure of the reason, would I assume that the market knows something I donât? Probably not. I might update slightly, but Iâm not sure to what extent Iâd trust the market to provide valuable information about AGI more than direct information about model capabilities and diffusion.
But of course, if we do update on market shifts, it has to be at least somewhat symmetrical. If a market collapse would slow down your timelines, insane market growth should accelerate your timelines for the same reason.
âLongtermism isnât necessary to think that x-risk (of at least some varieties) is a top priority problem.â
I donât think itâs a niche viewpoint in EA to think that, mainly because of farmed and wild animal suffering, the short term future is net-negative in expectation, but the long-term future could be incredibly good. This means that some variety of longtermism is essential in order to not embrace x-risk in our lifetimes as desirable.
I definitely identify with where youâre coming from here, but these insights might also imply a potential partner post on âHow to avoid EA senescence (if you want to)â.
Based on your examples, this might look like:
Specialise, even if itâs not your jobâdive very deep into at least one relevant EA area. If you can find something you find interesting and neglected, can you become top 1% knowledgeable (within EA) in an obscure sub-field?
Develop (and share) a niche perspective on where to donate based on your specific worldview. If youâre very convinced about insect sentience, or you lean negative utilitarian, you will very quickly realise that EA Funds are not the highest EV option for you!
Prioritise boosting/âmaintaining your âEA energyâ
Host more parties
Thanks for the post! This is a very valuable topic, and the development econ mainstream is totally lost on this question!
I agree with some of your points, but I think we need to distinguish very carefully between âdeveloping countriesâ. All the factors you mention with regards to labour displacement (structure of the economy, data availability, telecommunication infra) are wildly different between, say, Togo, Brazil, and Indonesia. Same with private- vs. public sector diffusion; within âdeveloping countriesâ, youâve got countries with massive tech hubs and their own tech billionaires, and those where most people still donât have electricity.
For me, the most important development question with regards to TAI (and the reason itâs important to distinguish) is the feasibility of the export-led development model. Generally, if countries manage to develop a high-value added export sector, they attract FDI, get foreign currency, climb up the value chain, and become richer. If they donât, they stay poor. Except for the occasional country finding insane levels of natural resources, this is the only real way that countries have become rich over the last 100 years.
If we get safe, transformative AI, we can imagine that demand for imports massively rises in the West, and middle-income countries like China, Vietnam, and Indonesia with strong export sectors (and the infrastructure to build on their existing exports) are able to take advantage of this. As these countries already have good infrastructure (e.g. electrification, internet access, land and shipping transport) they can probably also benefit from AI & Robotics to develop âIndustry 4.0â and make their export sector even more dominant.
Iâd therefore estimate that a few of these âdevelopingâ countries with existing strong export sectors will catch-up and become rich relatively soon.
But what of the poorest countries?!
Most African countries with a GDP below, say $3000 are very low down the value chain in all sectors, with little but raw materials (e.g. coffee, cocoa, oil if they have it) as exports. Theyâre struggling to compete with Asian developing powerhouses, and they havenât got the transport infrastructure, governance, or capital etc. to develop a quality export-led economy. In a world without TAI, as middle-income countries get rich, poor countries would develop the export industries and climb the ladder themselves, but this seems very unlikely with displacement of manufacturing labour by robotics.
My overall take is that (in an optimistic AI scenario) well-governed middle-income countries would probably end up more similar to rich countries. But weâd have really âkicked away the ladderâ from the very poor countries.
My ideas for posts (Iâll try to write at least one):
I recently learned that malaria causes about as many miscarriages and stillbirths as it causes live infant deaths, but we only count neonatal deaths in most cost-effectiveness estimates. Intermittent preventive treatment with Sulphadoxine-Pyrimethamine (IPTp-SP) for pregnant women seems to be more cost-effective than bed-nets for preventing malaria-related stillbirths and miscarriage. Unsure whether to write a narrow post on that, or a deeper post on âWhat are the most effective charities, given worldviews where unborn children have similar value to new-borns?â
Some Europeans have been asking me a lot about what people in smaller countries can do to make AI go better (or slow down) - especially with regards to China. I think weâve got a lot of lessons from (especially Cold War) history about third countries using their relations with superpowers to increase existential safety, but I donât think anyoneâs written an EA forum post about it.
I wrote a blog post on what I call âThe Great Happiness Stagnationââlooking at the flattening of happiness in many rich countries since they became rich. Iâve been thinking about converting it to a forum post, but it currently seems insufficiently rigorous to be worthy of the forum!
Agree with your point about the Chinese study reference, about healthy aging for elderly Chinese people. The OP uses it to make three separate points, about cognitive impairment, dose-response effects and lower overall odds of healthy aging, but itâs pretty clear that the study is basically showing the effects of poverty on health in old age.
Elderly Chinese people are mostly vegetarian or vegan because a) they canât afford meat, or b) have stopped eating meat because they struggle with other health issues, both of which would massively bias the outcomes! So their poor outcomes might be partly through diet-related effects, like nutrient/âprotein deficiency, but could also be sanitation, malnutrition in earlier life (these are people brought up in extreme famines), education (particularly for the cognitive impairment test), and the health issues that cause them to reduce meat.
The study fails to control for extreme poverty by grouping together everyone who earned <8000 Yuan a year (80% of the survey sample!), which is pretty ridiculous, because the original dataset should have continuous data...
The paper also makes it very clear that diet quality is the real driver, and that healthy plant-based diets score similarly to omnivorous diets âwith vegetarians of higher diet quality not significantly differing in terms of overall healthy aging and individual outcomes when compared to omnivoresâ.
Probably less importantly, it conditions on survival to 80, which creates a case of survivorship bias/âcollider bias. So there could be a story where less healthy omnivores tend to die earlier (you get effects like this with older smokers, sometimes), and the survivors appear healthier.
I agree with the upfront tagline âHaving children is not the most effective way to improve the worldâ, but feel I disagree pretty strongly with a bunch of these takes:
âOwingâ it to your parents. This feels a little straw-manned. Wanting to have kids for your parentsâ sake might be about feeling grateful for 16+ years of love & care, or just making someone you care about happier in their old age. From an EA perspective, you perhaps shouldnât weight this too highly. But when choosing to have kids or not, especially if your parents really want grandchildren, you are making this trade-off. One of my explicit considerations when considering having kids was thinking about my in-laws and extended family.
Donating to AMF to increase population. Donât strongly disagree with the principle here, but donating to AMF is probably not optimal. I think it would be cheaper to incentivise births directly than donating to AMF, if thatâs your goal. (Edit: I wrote something else that finds that AMF might actually incentivise births cheaply, because of maternal/âplacental malaria) Iâve written about this: (Who should we pay to increase birth rates?), where I make a toy model about choosing where you might want to generate new lives. I suggest lower-middle income countries other than Sub-Saharan Africa, mainly because of quality of life concerns.
Itâs a bad idea to make ethical arguments either way about having children. This one surprised me the most. Do you mean we shouldnât make these arguments at all, or simply that we should avoid certain impolite judgements of othersâ choices? My take: of course you shouldnât overdo it and rant to expectant mothers about the meat-eater problem, risk of population collapse, and negative utilitarianism, but itâs still one of the biggest ethical decisions in a humanâs life. Thereâs no reason why this should be less suitable for ethical debate than what job you choose or what charity you donate to.
Iâm very torn on this question, so letâs shoot for 60%.
Many people are probably thinking about the impact on animals, which most EA forum readers will probably agree is a far stronger argument for anti-natalism than climate.
Thereâs a set of arguments, recently articulated by Benthamâs Bulldog, which looks like this:
High birth rates are good because human lives tend to be good
But humans kill animals for food and incentivise factory farming, which clearly overwhelms the positives of a human life, therefore high human birth rates might be bad (meat-eater problem)
But humans also incidentally prevent the lives of billions of insects and fish, and insect lives are net-negative, therefore this clearly overwhelms both farmed animals and the value of a human life, therefore high birth rates are actually good!
And more humans are likely to make the far future even better for animals, so high birth rates are even more good!
Of course we should take this argument seriously, but:
1) Having children seems an incredibly inefficient way of maximising your destruction of insects! If insect suffering does overwhelm other effects, this fails to provide an effective utilitarian argument for human pronatalism.
2) Based on current human values and preference for environmental protection/ârewilding, it seems plausible to me that the marginal human may not decrease wild insect numbers. Similarly, I can see a far-future where more humans make the world worse for both farmed and wild animals.
3) Practically, I suspect youâll lose most ethically minded individuals, or people who have very low estimates of insect/âfish consciousness, at step 2 - the meat-eater problem. Step 3 requires taking quite a bitter pill in terms of cross-species anti-natalism and the disvalue of existence more generally. âOpen Phil-brand EAâ, which generally disregards insect and wild animal welfare, would also have to reject step 3, and may therefore have to conclude that anti-natalism is good.
4) More personally, it does seem a bit weird feeling that my wonderful little babyâs main source of value in the world is his insect-destroying potential.
Maybe you read it, maybe just a coincidence, but I wrote a blogpost that (using a toy model) found Uzbekistan to be the most promising country for incentivising birth rates!
Iâll push against this post a little bit, despite agreeing with a lot of the ideas.
Firstly, I think we can avoid the moral discomfort of âhoping for warning shotsâ by reframing as âhoping for windows of opportunityâ. We should hope and prepare for moments, where, for whatever reason, policymakers and the public are unusually attentive to what weâre saying.
Secondly, while youâre more arguing against the hand-wavy âwarning-shot as cavalryâ claims, there seems to be another claim- that we should act in a similar way regardless of whether or not the âwarning shotâ model is correct, i.e. whether we expect the policy and discourse battle to take the form of a gradual grind of persuasion vs. a very lumpy, unpredictable pattern shaped around distinct windows of opportunity.
Our strategy might look similar most of the time, and I agree that a lot of the hard persuasion work in the trenches needs to go on regardless. But I suspect there are a few ways you might act differently if the âwarning shot/âwindows of opportunityâ model is correct. For example:
Strategic preparednessâkeep some things in reserve, have a bunch of ready-to-go policy proposal binders or communication strategies deliberately for when a window opens
Take a slightly more cautious approach to preserving credibility capital. There are ways of talking about risks now that might cost you influence today, but look appropriate in the correct window.
Build relationships in anticipation of a window of opportunity opening, rather than pushing directly for change.
I agree with the âyou donât have to debate on their termsâ point here, but I think for 99% of your readers/âlisteners, it cuts far more strongly in a different way than that youâre implying.
The debate has generally been set in terms of âAnthropic vs. DoWâ, and, while I know zero people in our community who have taken the governmentâs side on this, Iâve seen many EAs and adjacent people become increasingly uncritical supporters of Anthropic, just because theyâre standing against the obviously bad actor in this situation.
I think itâs important to remember:
If you thought Anthropic was untrustworthy before this, you shouldnât update too much the other wayâespecially when they backtracked on their RSP over the same period.
If you thought that Anthropicâs decision to join the race towards AGI was perilous, you shouldnât really update your view on this based on the Pentagon being absurd and unpredictable.
Regardless of the intention and character of the government actors, itâs potentially still a worrying sign that the most powerful state in the world has tried to shut down a frontier AI lab and failed spectacularly.
The growth and popularity of Anthropic and Claude Code have since caused the AI 2027 team to shorten their AGI timelines.