MIRI’s business model relies on the opposite narrative. MIRI pays Eliezer Yudkowsky $600,000 a year. It pays Nate Soares $235,000 a year. If they suddenly said that the risk of human extinction from AGI or superintelligence is extremely low, in all likelihood that money would dry up and Yudkowsky and Soares would be out of a job.
[...] The kind of work MIRI is doing and the kind of experience Yudkowsky and Soares have isn’t really transferable to anything else.
$235K is not very much money [edit: in the context of the AI industry]. I made close to Nate’s salary as basically an unproductive intern at MIRI. $600K is also not much money. A Preparedness researcher at OpenAI has a starting salary of $310K – $460K plus probably another $500K in equity. As for nonprofit salaries, METR’s salary range goes up to $450K just for a “senior” level RE/RS, and I think it’s reasonable for nonprofits to pay someone with 20 years of experience, who might be more like a principal RS, $600K or more.
In contrast, if Mechanize succeeds, Matthew Barnett will probably be a billionaire.
If Yudkowsky said extinction risks were low and wanted to focus on some finer aspect of alignment, e.g. ensuring that AIs respect human rights a million years from now, donors who shared their worldview would probably keep donating. Indeed, this might increase donations to MIRI because it would be closer to mainstream beliefs.
MIRI’s work seems very transferable to other risks from AI, which governments and companies both have an interest in preventing. Yudkowsky and Soares have a somewhat weird skillset and I disagree with some of their research style but it’s plausible to me they could still work productively in a mathy theoretical role in either capabilities or safety.
However, things I agree with
If the Mechanize co-founders wanted to focus on safety rather than capabilities, they could.
the Mechanize co-founders decided to start the company after forming their views on AI safety.
The Yudkowsky/Soares/MIRI argument about AI alignment is specifically that an AGI’s goals and motivations are highly likely to be completely alien from human goals and motivations in a way that’s highly existentially dangerous.
I understand the point being made (Nate plausibly could get a pay rise from an accelerationist AI company in Silicon Valley, even if the work involved was pure safetywashing, because those companies have even deeper pockets), but I would stress that these two sentences underline just how lucrative peddling doom has become for MIRI[1] as well as how uniquely positioned all sides of the AI safety movement are.
There are not many organizations whose messaging has resonated with deep pocketed donors to the extent that they can afford to pay their [unproductive] interns north of $200k pro rata to brainstorm with them.[2] Or indeed up to $450k to someone with interesting ideas for experiments to test AI threats, communication skills and at least enough knowledge of software to write basic Python data processing scripts. So the financial motivations to believe that AI is really important are there on either side of the debate; the real asymmetry is between the earning potential of having really strong views on AI vs really strong views on the need to eliminate malaria or factory farming.
tbf to Eliezer, he appears to have been prophesizing imminent tech-enabled doom/salvation since he was a teenager on quirky extropian mailing lists, so one thing he cannot be accused of is bandwagon jumping.
Outside the Valley bubble, plenty of people at profitable or well-backed companies with specialist STEM skillsets or leadership roles are not earning that for shipping product under pressure, never mind junior research hires for nonprofits with nominally altruistic missions
I think this misses the point: The financial gain comes from being central to ideas around AI in itself. I think given this baseline, being on the doomer side tends to carry huge opportunity cost financially. At the very least it’s unclear and I think you should make a strong argument to claim anyone financially profits from being a doomer.
The opportunity cost only exists for those with a high chance of securing comparable level roles in AI companies, or very senior roles at non-AI companies in the near future. Clearly this applies to some people working in AI capabilities research,[1] but if you wish to imply this applies to everyone working at MIRI and similar AI research organizations, I think the burden of proof actually rests on you. As for Eliezer, I don’t think his motivation for dooming is profit, but it’s beyond dispute that dooming is profitable for him. Could he earn orders of magnitude more money from building benevolent superintelligence based on his decision theory as he once hoped to? Well yes, but it’d have to actually work.[2]
Anyway, my point was less to question MIRI’s motivations or Thomas’ observation Nate could earn at least as much if he decided to work for a pro-AI organization and more to point out that (i) no, really, those industry norm salaries are very high compared with pretty much any quasi-academic research job not related to treating superintelligence as imminent and especially to roles typically considered “altruistic” and (ii) if we’re worried that money gives AI company founders the wrong incentives, we should worry about the whole EA-AI ecosystem and talent pipeline EA is backing. Especially since that pipeline incubated those founders.
The context of this quote, which you have removed, is discussion of the reasonableness of wages for specific people with specific skills. Since neither Nate nor Eliezer’s counterfactual is earning the median global wage, your statistic seems irrelevant.
I agree. But the reason I agree is that I think the relevant metric of what counts as a lot of money here is not whether it is a competitive salary in an ML context, but whether it would be perceived as a lot of money in a way that could plausibly threaten Eliezer’s credibility among people who would otherwise be more disposed to support AI safety, e.g. if cited broadly. I believe the answer is that it is, and so in a way that even a sub-$250k salary would not be (despite how insanely high a salary that is by the standard of even most developed countries), and I would guess this expected effect to be bigger than the incentive benefits of guaranteeing his financial independence. For this reason, accepting this level of income struck me as unwise, though I’m happy to be persuaded otherwise.
One should stick to the original point that raised the question about salary.
Is $600K a lot of money for most people and does EY hurt his cause by accepting this much? (Perhaps, but not the original issue)
Does EY earning $600K mean he’s benefitting substantially from maintaining his position on AI safety? E.g. if he was more pro AI development, would this hurt him financially? (Very unlikely IMO, and that was the context Thomas was responding to)
On a global scale I agree. My point is more that due to the salary standards in the industry, Eliezer isn’t necessarily out of line in drawing $600k, and it’s probably not much more than he could earn elsewhere; therefore the financial incentive is fairly weak compared to that of Mechanize or other AI capabilities companies.
Thanks for the reply. I agree with your specific point but I think it’s worth being more careful with your phrasing. How much we earn is an ethically-charged thing, and it’s not a good thing if EA’s relationship with AI companies gives us a permission structure to lose sight of this.
Edit: to be clear, I agree that “it’s probably not much more than he could earn elsewhere” but disagree that “Eliezer isn’t necessarily out of line in drawing $600k”
Outside view: If I got WID data right: net personal wealth of US top percentile increased from $.59 Million in 1820 to $13.53 Million in 2024. For the bottom two deciles of India it increased from $58 to $228.
The industrial revolution made some people very rich, but not others. Why would transformative AI make everybody incredibly rich? See also https://intelligence-curse.ai/
I used: Average net personal wealth, all ages, equal split, Dollar $ ppp constant (2024) (I’m new to WID database and did not have time to read the data documentation. Let me know if I interpret data wongly.) Source: https://wid.world/
Global wealth would have to increase a lot for everyone to become billionaire. There are 10 billion people. So everyone being a billionaire would require a global wealth of 10^19 $ (= 10*10^9*1*10^9) for perfect distribution. Global wealth is 600 T$. So it would have to become 16.7 k (= 10^19/(600*10^12)) times as large. For a growth of 10 %/year, it would take 102 years (= LN(16.7*10^3)/LN(1 + 0.10)). For a growth of 30 %/year, it would take 37.1 years (= LN(16.7*10^3)/LN(1 + 0.30)).
I strongly disagree with a couple of claims:
$235K is not very much money [edit: in the context of the AI industry]. I made close to Nate’s salary as basically an unproductive intern at MIRI. $600K is also not much money. A Preparedness researcher at OpenAI has a starting salary of $310K – $460K plus probably another $500K in equity. As for nonprofit salaries, METR’s salary range goes up to $450K just for a “senior” level RE/RS, and I think it’s reasonable for nonprofits to pay someone with 20 years of experience, who might be more like a principal RS, $600K or more.
In contrast, if Mechanize succeeds, Matthew Barnett will probably be a billionaire.
If Yudkowsky said extinction risks were low and wanted to focus on some finer aspect of alignment, e.g. ensuring that AIs respect human rights a million years from now, donors who shared their worldview would probably keep donating. Indeed, this might increase donations to MIRI because it would be closer to mainstream beliefs.
MIRI’s work seems very transferable to other risks from AI, which governments and companies both have an interest in preventing. Yudkowsky and Soares have a somewhat weird skillset and I disagree with some of their research style but it’s plausible to me they could still work productively in a mathy theoretical role in either capabilities or safety.
However, things I agree with
I understand the point being made (Nate plausibly could get a pay rise from an accelerationist AI company in Silicon Valley, even if the work involved was pure safetywashing, because those companies have even deeper pockets), but I would stress that these two sentences underline just how lucrative peddling doom has become for MIRI[1] as well as how uniquely positioned all sides of the AI safety movement are.
There are not many organizations whose messaging has resonated with deep pocketed donors to the extent that they can afford to pay their [unproductive] interns north of $200k pro rata to brainstorm with them.[2] Or indeed up to $450k to someone with interesting ideas for experiments to test AI threats, communication skills and at least enough knowledge of software to write basic Python data processing scripts. So the financial motivations to believe that AI is really important are there on either side of the debate; the real asymmetry is between the earning potential of having really strong views on AI vs really strong views on the need to eliminate malaria or factory farming.
tbf to Eliezer, he appears to have been prophesizing imminent tech-enabled doom/salvation since he was a teenager on quirky extropian mailing lists, so one thing he cannot be accused of is bandwagon jumping.
Outside the Valley bubble, plenty of people at profitable or well-backed companies with specialist STEM skillsets or leadership roles are not earning that for shipping product under pressure, never mind junior research hires for nonprofits with nominally altruistic missions
I think this misses the point: The financial gain comes from being central to ideas around AI in itself. I think given this baseline, being on the doomer side tends to carry huge opportunity cost financially.
At the very least it’s unclear and I think you should make a strong argument to claim anyone financially profits from being a doomer.
The opportunity cost only exists for those with a high chance of securing comparable level roles in AI companies, or very senior roles at non-AI companies in the near future. Clearly this applies to some people working in AI capabilities research,[1] but if you wish to imply this applies to everyone working at MIRI and similar AI research organizations, I think the burden of proof actually rests on you. As for Eliezer, I don’t think his motivation for dooming is profit, but it’s beyond dispute that dooming is profitable for him. Could he earn orders of magnitude more money from building benevolent superintelligence based on his decision theory as he once hoped to? Well yes, but it’d have to actually work.[2]
Anyway, my point was less to question MIRI’s motivations or Thomas’ observation Nate could earn at least as much if he decided to work for a pro-AI organization and more to point out that (i) no, really, those industry norm salaries are very high compared with pretty much any quasi-academic research job not related to treating superintelligence as imminent and especially to roles typically considered “altruistic” and (ii) if we’re worried that money gives AI company founders the wrong incentives, we should worry about the whole EA-AI ecosystem and talent pipeline EA is backing. Especially since that pipeline incubated those founders.
including Nate
and work in a way that didn’t kill everyone, I guess...
This is false.
The context of this quote, which you have removed, is discussion of the reasonableness of wages for specific people with specific skills. Since neither Nate nor Eliezer’s counterfactual is earning the median global wage, your statistic seems irrelevant.
What do you think their counterfactual is? I don’t think any of what they’ve been doing is really transferable.
I agree. But the reason I agree is that I think the relevant metric of what counts as a lot of money here is not whether it is a competitive salary in an ML context, but whether it would be perceived as a lot of money in a way that could plausibly threaten Eliezer’s credibility among people who would otherwise be more disposed to support AI safety, e.g. if cited broadly. I believe the answer is that it is, and so in a way that even a sub-$250k salary would not be (despite how insanely high a salary that is by the standard of even most developed countries), and I would guess this expected effect to be bigger than the incentive benefits of guaranteeing his financial independence. For this reason, accepting this level of income struck me as unwise, though I’m happy to be persuaded otherwise.
Thanks for the good point, Paul. I tend to agree.
One should stick to the original point that raised the question about salary.
Is $600K a lot of money for most people and does EY hurt his cause by accepting this much? (Perhaps, but not the original issue)
Does EY earning $600K mean he’s benefitting substantially from maintaining his position on AI safety? E.g. if he was more pro AI development, would this hurt him financially? (Very unlikely IMO, and that was the context Thomas was responding to)
On a global scale I agree. My point is more that due to the salary standards in the industry, Eliezer isn’t necessarily out of line in drawing $600k, and it’s probably not much more than he could earn elsewhere; therefore the financial incentive is fairly weak compared to that of Mechanize or other AI capabilities companies.
Thanks for the reply. I agree with your specific point but I think it’s worth being more careful with your phrasing. How much we earn is an ethically-charged thing, and it’s not a good thing if EA’s relationship with AI companies gives us a permission structure to lose sight of this.
Edit: to be clear, I agree that “it’s probably not much more than he could earn elsewhere” but disagree that “Eliezer isn’t necessarily out of line in drawing $600k”
It’s true Mechanize are trying to hire him for 650k...
If Mechanize succeeds in its long-term goal of “the automation of all valuable work in the economy”, then everyone on Earth will be a billionaire.
Outside view: If I got WID data right: net personal wealth of US top percentile increased from $.59 Million in 1820 to $13.53 Million in 2024. For the bottom two deciles of India it increased from $58 to $228.
The industrial revolution made some people very rich, but not others. Why would transformative AI make everybody incredibly rich?
See also https://intelligence-curse.ai/
I used: Average net personal wealth, all ages, equal split, Dollar $ ppp constant (2024)
(I’m new to WID database and did not have time to read the data documentation. Let me know if I interpret data wongly.) Source: https://wid.world/
Hi Matthias. Thanks for linking to the World Inequality Database (WID). I had never checked it out, and it has very interesting data.
Global wealth would have to increase a lot for everyone to become billionaire. There are 10 billion people. So everyone being a billionaire would require a global wealth of 10^19 $ (= 10*10^9*1*10^9) for perfect distribution. Global wealth is 600 T$. So it would have to become 16.7 k (= 10^19/(600*10^12)) times as large. For a growth of 10 %/year, it would take 102 years (= LN(16.7*10^3)/LN(1 + 0.10)). For a growth of 30 %/year, it would take 37.1 years (= LN(16.7*10^3)/LN(1 + 0.30)).