RA at GovAI
Tomás Aguirre
Karma: 115
“trillions of capital are being flooded into accelerating AI development across industries” fwiw this seems only correct under a very expansive definition of accelerating AI development, e.g. all hyperscaler capex (including non-AI spend and AI inference) was ~$500 billion in 2025
″while a measly $50 million dollars” this seems outdated—Coefficient Giving’s Technical AI Safety team made 140 million USD in grants in 2025. If you consider all other funding (the UK AISI’s founding budget was ~$125M, internal lab speding on technical AI safety, other grantmakers, etc.), it’s plausible we’re over $1 billion.
I think oftentimes the relevant counterfactual is not “this person would be doing even more impactful work in a highly-impactful area” but “this person would not be working in a high-impact area whatsoever”
Some notes on OpenAI disproving the Erdős unit distance conjecture (from a non-mathematician):
First, this is big. A notorious math conjecture being disproved by AI would be sci-fi 10 years ago. In my layman’s read, this is plausibly the most prominent math result in the last 12 months—AI, centaur, human or whatever.
Second, it rebutted an Erdős conjecture, and I found it curious that the first clear math breakthrough goes against consensus. There are a few potential reads to this: a) this seems to go against the LLMs-are-sycophantic-machine claims; b) even if LLMs are that sycophantic, exploring different intellectual paths is so cheap to them that sycophancy doesn’t quite matter as much; c) it may mean that AI is sycophantic at the user-level but not at the literature-level, which actually may be great for finding novel solutions, but is also the very thing that enables e.g. AI psychosis.
Third, it’s hard to wrap my head around having an intelligence that is probably at the level of a very promising Terence Tao graduate student—but not Tao-level yet. It allows exploring many hypotheses/conjectures/counter-examples/constructions that go against intuitive human ~quick evaluation/priors of what is promising, simply because they can be so exhaustive in their exploration. It’s the country part of a “country of geniuses in a datacenter”
Fourth, the solution combines insights/techniques from different fields. It pulled off an answer that used algebraic number theory to solve a combinatorial geometry problem. Mathematicians seem to think how it did it may unlock more. In a world where specialization is deemed necessary structurally/institutionally, AIs have a special advantage even with “mere” cross-field interpolation [tbc, in this case there seems to be substantial extrapolation in my layman’s read]. Also, the constraint here may not be human intelligence per se. Surely we don’t have a current Riemann-level mathematician partly because of bottlenecks of human intelligence as things specialized, but also institutionally/organizationally we may have incentivized specialization too much besides what was intellectually needed—and organizational innovations may actually be the bottleneck for solving some big Millennium-level math problems [e.g. focused research organizations that allow for interdisciplinary moonshoots]
Fifth, we don’t know much about how many other math problems OpenAI explored. Is this the first [prominent] one they got a solution to after running through all (relatively prominent) ~Erdős problems? I don’t want to come across as moving goalposts—again, this is really big. But what does it mean if they did an extensive evaluation of various problems and this particular one is the first one to land? My best sense right now is that they probably ran a search across various problems, had internal employees [that include very bright mathematicians] to verify what was most promising, then got Gowers/other prominent mathematicians involved to double-check. This may mean that various other problems have been solved but is currently bottleneck by expert human verification.
My sense is that you’re right. IIRC diminishing returns are more salient for AMF than for GiveDirectly, and one of the key arguments pro GiveDirectly would be that flat returns persist longer—but probably when they make this argument they’re thinking on the scale of $100M–$1B, not hundreds of billions.
Perhaps the best case in point would be Bolsa Família: ~$30B yearly budget, one of the largest conditional cash transfer programs in the world, increased ~5-fold over the past few years, noticeably turned into a less effective program imho, but still seems like one of the most effective programs from the Brazilian government
I wouldn’t think of this as a matter of thresholds but continuously decreasing returns, tho
I agree. Also, my sense is that MATS and GovAI’s fellowships are generally more senior, or require you to be more well-versed in the AI safety/governance universe, than Pivotal, ERA, and Talos—at least, that’s how I strongly perceived it as an applicant.
My current view of that is best summarized by: “Drop the ‘AI’. Just policy. It’s cleaner”
Lighting has been getting ridiculously cheaper. And for the most part we seem to be not taking advantage of that positive externality: reducing crime through better lighting. This has been battle-tested as one of the effective ways for public security, see Chalfin, Hansen, Lerner & Parker (2022), an RCT in NYC public housing finding ~36% reductions in nighttime outdoor index crimes from added street lighting. Many, many major cities still haven’t copied this at the right levels!
But we’re also getting substantially negative externalities of bright lighting. Office buildings that never turn off their lights because why would they care. Apropos the new office building that just opened next to my housing. This may alimentate NIMBY spirits in me, God forbid. Kyba et al. (2017) document that Earth’s artificially lit outdoor area grew 2.2% per year from 2012 to 2016, with the LED transition producing a rebound effect instead of getting savings. Jevons paradox and such.
Also, this has all sorts of annoyances. I think malls, pharmacies, and hospitals have all become much brighter since my childhood. I may be more sensorially overloaded than most people, but this does meaningfully affect my qualia, so much that Pigou himself would collect taxes from the pharmacies with dozens and dozens of LEDs, while Coase would advocate that I have the natural property right of not being assaulted with that much lumen while buying a Tylenol. This does affect wellbeing of more than just me (Cho et al. 2015). But lightly enough, ha, to not be a topic of discussion.
Hi Peter! I often see the MIT FutureTech positions, and as an economics major who is about to graduate, I would love to apply, but it’s a pity they don’t sponsor visas. From what I understand, this seems to be a common issue across MIT, as I’ve noticed that many of their predoc and similar positions also don’t sponsor visa. I know this might not be something you can change, but it would be great if there could be options for visa sponsorship, or even remote independent contracting. In any case, I think the work you all do is fantastic!
Thanks for the answer; I think I understood your point better, but I still have a different view. You’re correct that guaranteed income can be described as a UBI with a 100% marginal tax rate on labor income up to a certain threshold. Both these policies are indeed equivalent in terms of incentives and have the same cost. However, this is quite different from the straightforward UBI that the OP was describing!
You also make a good point that most means-tested income programs are incentive-equivalent (or quite similar, though typically with more design details) to a UBI program with a less-than-100% marginal tax rate on labor income up to the threshold. This helps explain why we don’t observe such high substitution effects in practice.
That said, guaranteed income still tends to be much less costly than simple UBI. This becomes evident when comparing a low level of guaranteed income versus a low level of UBI [as defined by OP]. If you offer a guaranteed monthly income of 100 USD [or UBI + 100% marginal tax rate up to 100 USD of labor income] , some people might leave the workforce entirely, but very few would likely do so. This is much less costly than providing 100 USD to every person! The point at which their economic costs [fiscal costs + behavioral costs due to changing incentives] will be similar is largely an empirical question involving labor elasticity, reservation wages, etc.
Draghi’s report signal a less safety-focused European Union on AI
I’m not sure I follow your argument entirely. While your thought experiment is interesting for considering substitution vs. income effects, I believe UBIs are still far more costly than guaranteed incomes.
Yes, guaranteed, means-tested incomes have some costs due to substitution effects that UBIs don’t. People near the income threshold might work less or even quit the labor force to avoid losing benefits, potentially increasing program costs. UBI avoids this by giving everyone the same amount regardless of work.
However, empirically, these effects seem small. Studies on conditional transfer programs like Brazil’s Bolsa Familia and Mexico’s Progresa/Prospera show modest impacts on labor supply. Most find only slight reductions in work hours, if any. These aren’t strictly guaranteed income programs as they have some strings attached, but they seem to be good indicators of how people might respond to similar cash transfer programs. So while the OP overlooked substitution effects, real-world evidence suggests they do not seem nearly significant enough to make guaranteed income programs comparable in cost to full UBI systems.
For reference, I think these papers from an Annual Review UBI symposium are quite good: https://www.annualreviews.org/content/journals/10.1146/annurev-ec-11
Brazil has been dealing with massive criminal wildfires for the last few weeks, and the air quality is record-breakingly bad. Besides other obvious issues (ineffective government response in going after the criminals setting fires, climate change making everything worse), hardly anyone is talking about how to deal with the immediate air quality problem. It’s a bit bizarre.
People aren’t widely adopting PFF2 masks and air purifiers. These remain somewhat niche topics even though pretty much everyone is suffering. To be fair, there are occasional media reports and government alerts about how to deal with the situation, but these feel too little, and one only gets them if actively looking for them.
It’s affecting tens of millions of people (scale ✓)
Barely anyone is really addressing it (neglectedness ✓)
We have simple solutions that could help a lot (tractability ✓)
It feels like there’s potential for some serious impact if one approaches this right. It may be a severe case of availability bias, but all this is making me value air quality more as an EA cause area.
My guess is that the gains from professional management practices are generally much more limited for micro enterprises or self-employed individuals because: 1) there is less internal friction. If it’s just you, it’s easier to monitor money inflow and outflow even without bookkeeping. This changes significantly when you have around 15 people; and 2) there are fewer economies of scale to achieve. I’d guess the ideal future envisioned by small companies, like those with 15 people, is to grow into a 100-person company, with much more business and new locations. For a self-employed person, the best scenario [as seen by themselves] seems to be to keep the business running, increase some profit but not change things much, or find a new job. Since many professional management practices seem to be somewhat of a fixed cost, there is little incentive for them to adopt these practices if they don’t see growth as their default path to success. I think Duflo and Banerjee wrote a bit about this in their work on microcredit for small companies, but I’m doing guesswork here haha
Fantastic work organizing this event, Leo and Juana =)
“I usually estimate that the U.S. government saves about one life per $10 million that it spends well.”
I’m curious, why do you think this? The value of statistical life (VSL) you linked is about the benefit of saving a life, not about the cost. If we assume the government consistently uses this $10 million VSL threshold for interventions, it could reflect the (marginal) cost of saving a life. But this feels like a weird approach. It might be useful to get a sense of order of magnitude, but I’m somewhat skeptical
[comment crossposted] I agree with the main point of the text. As someone from a developing country (Brazil), I feel people often worry too much about brain drain, overlooking these equilibrium effects you mentioned. That said, I’d still be concerned about friction costs. The supply of many highly skilled professionals, like specialized physicians, is pretty inelastic in the short term. If a substantial part of them suddenly migrates, it could cause a real shortage, leading to significant economic costs during the adjustment period. I think policymakers do have a role here, but it should be more about easing this transition rather than trying to stop migration outright.
Securitization in the international relations sense, that is, framing frontier AI development as a major (international/national) security threat that requires top cybersecurity, people with experience in the department of defense, etc.
I wonder how do they plan to get GPUs at scale while remaining “insulated from short-term commercial pressures”
Securitization is indeed coming
“OpenAI appoints Retired U.S. Army General Paul M. Nakasone to Board of Directors”
https://openai.com/index/openai-appoints-retired-us-army-general/
very low confidence, but I think 1) orgs tend to advertise research roles more widely than ops roles, 2) with longer decision times, and 3) research roles outnumber ops roles by something like 2:1 or even 4:1, so I wouldn’t read too much into raw numbers of job postings