A 10% chance of transformative AI this decade justifies current EA efforts to make AI go well. That includes the opportunity costs of that money not going to other things in the 90% worlds. Spending money on e.g. nuclear disarmament instead of AI also implies harm in the 10% of worlds where TAI was coming. Just calculating the expected vale of each accounts for both of these costs.
It’s also important to understand that Hendrycks and Yudkowsky were simply describing/predicting the geopolitical equilibrium that follows from their strategies, not independently advocating for the airstrikes or sabotage. Leopold is a more ambiguous case, but even he says that the race is already the reality, not something he prefers independently. I also think very few “EA” dollars are going to any of these groups/individuals.
A 10% chance of transformative AI this decade justifies current EA efforts to make AI go well.
Not necessarily. It depends on
a) your credence distribution of TAI after this decade,
b) your estimate of annual risk per year of other catastrophes, and
c) your estimate of the comparative longterm cost of other catastrophes.
I don’t think it’s unreasonable to think, for example, that
there’s a very long tail to when TAI might arrive, given that its prospects of arriving in 2-3 decades are substantially related to to its prospects of arriving this decade) it arriving this decade (e.g. if we scale current models substantially and they still show no signs of becoming TAI, that undermines the case for future scaling getting us there under same paradigm); or
the more pessimistic annual risk estimates I talked about in the previous essay of 1-2% per year are correct, and that future civilisations will have a sufficiently increased difficulty for a collapse to cost to have near 50% the expected cost of extinction
And either of these beliefs (and others) would suggest we’re relatively overspending on on AI.
It’s also important to understand that Hendrycks and Yudkowsky were simply describing/predicting the geopolitical equilibrium that follows from their strategies, not independently advocating for the airstrikes or sabotage.
This is grossly disingenuous. Yudkowsky frames his call for airstrikes as what we ‘need’ to do, and describes them in the context of the hypothetical ‘if I had infinite freedom to write laws’. Hendrycks is slightly less direct in actively calling for it, claiming that it’s the default, but the document clearly states the intent of supporting it ‘we outline measures to maintain the conditions for MAIM’.
These aren’t the words of people dispassionately observing a phenomenon—they are both clearly trying to bring about the scenarios they describe when the lines they’ve personally drawn are crossed.
I would add that it’s not just extreme proposals to make “AI go well” like Yudkowsky’s airstrike that potentially have negative consequences beyond the counterfactual costs of not spending the money on other causes. Even ‘pausing AI’ through democratically elected legislation enacted as a result of smart and well-reasoned lobbying might be significantly negative in its direct impact, if the sort of ‘AI’ restricted would have failed to become a malign superintelligence but would have been very helpful to economic growth generally and perhaps medical researchers specifically.
This applies if the imminent AGI hypothesis is false, and probably to an even greater extent it if it is true.
(The simplest argument for why it’s hard to justify all EA efforts to make AI go well based purely on its neglectedness as a cause is that some EA theories about what is needed for AI to go well directly conflict with others; to justify the course of action one needs to have some confidence not only that AGI is possibly a threat but that the proposed approach to it at least doesn’t increase the threat. It is possible that both donations to a “charity” that became a commercial AI accelerationist and donations to lobbyists attempting to pause AI altogether were both mistakes, but it seems implausible that they were both good causes)
A 10% chance of transformative AI this decade justifies current EA efforts to make AI go well. That includes the opportunity costs of that money not going to other things in the 90% worlds. Spending money on e.g. nuclear disarmament instead of AI also implies harm in the 10% of worlds where TAI was coming. Just calculating the expected vale of each accounts for both of these costs.
It’s also important to understand that Hendrycks and Yudkowsky were simply describing/predicting the geopolitical equilibrium that follows from their strategies, not independently advocating for the airstrikes or sabotage. Leopold is a more ambiguous case, but even he says that the race is already the reality, not something he prefers independently. I also think very few “EA” dollars are going to any of these groups/individuals.
Not necessarily. It depends on
a) your credence distribution of TAI after this decade,
b) your estimate of annual risk per year of other catastrophes, and
c) your estimate of the comparative longterm cost of other catastrophes.
I don’t think it’s unreasonable to think, for example, that
there’s a very long tail to when TAI might arrive, given that its prospects of arriving in 2-3 decades are substantially related to to its prospects of arriving this decade) it arriving this decade (e.g. if we scale current models substantially and they still show no signs of becoming TAI, that undermines the case for future scaling getting us there under same paradigm); or
the more pessimistic annual risk estimates I talked about in the previous essay of 1-2% per year are correct, and that future civilisations will have a sufficiently increased difficulty for a collapse to cost to have near 50% the expected cost of extinction
And either of these beliefs (and others) would suggest we’re relatively overspending on on AI.
This is grossly disingenuous. Yudkowsky frames his call for airstrikes as what we ‘need’ to do, and describes them in the context of the hypothetical ‘if I had infinite freedom to write laws’. Hendrycks is slightly less direct in actively calling for it, claiming that it’s the default, but the document clearly states the intent of supporting it ‘we outline measures to maintain the conditions for MAIM’.
These aren’t the words of people dispassionately observing a phenomenon—they are both clearly trying to bring about the scenarios they describe when the lines they’ve personally drawn are crossed.
I would add that it’s not just extreme proposals to make “AI go well” like Yudkowsky’s airstrike that potentially have negative consequences beyond the counterfactual costs of not spending the money on other causes. Even ‘pausing AI’ through democratically elected legislation enacted as a result of smart and well-reasoned lobbying might be significantly negative in its direct impact, if the sort of ‘AI’ restricted would have failed to become a malign superintelligence but would have been very helpful to economic growth generally and perhaps medical researchers specifically.
This applies if the imminent AGI hypothesis is false, and probably to an even greater extent it if it is true.
(The simplest argument for why it’s hard to justify all EA efforts to make AI go well based purely on its neglectedness as a cause is that some EA theories about what is needed for AI to go well directly conflict with others; to justify the course of action one needs to have some confidence not only that AGI is possibly a threat but that the proposed approach to it at least doesn’t increase the threat. It is possible that both donations to a “charity” that became a commercial AI accelerationist and donations to lobbyists attempting to pause AI altogether were both mistakes, but it seems implausible that they were both good causes)