noteâeverything i say here is based on my own interpretation of the AI scene. I have no special knowledge, so this may be inaccurate, out-of-date, or both
So this post has provoked a lot of reaction, and Iâm not going to defend all of it. I do think, however, that debating what is lying/âwhite lying/âlying by omission/âbeing misleading is not as important as the other issues. (I think the tone is overly combative and hurts its cause, for instance). But from the perspective of someone like Connor, I can see why they may think that the AI Safety Community (especially those at the larger labs) are deliberately falsifying their beliefs.
On the May 16th Senate Judiciary Committee hearing where the witnesses are under oath[1], Blumenthal asks Sam Altman about his quote that âdevelopment of superhuman machine intelligence is probably the greatest threat to the continued existence of humanityâ and leads on to talk about the effect of AI on jobs and employment. Sam answers this by also talking about jobs, but Gary Marcus then calls him out by saying âSamâs worst fear I do not think is employment and he never told us what his worst fear actually isâ. When asked again, Sam says âmy worst fears are that...we cause significant harm to the worldâ[2]
I think thatâs a lie of omissionâSamâs worst fear is the extinction of humanity or perhaps some S-risk scenario, and he did not say that. Does this violate Section 1001 of the United States Code? I donât know. I think itâd be good to get more concrete estimates from Sam of what he thinks the risks are, and I think as the CEO of the worldâs leading AI company, itâd be something the US Senate should interested in.
In a recent podcast, Dario Amodei implied his p(doom) was between 10-25%, though itâs not a clear prediction with a timeframe, resolution criteria, or whether itâs conditional or unconditional on Anthropicâs actions (listen and judge for yourself). But Darioâs been aware of xRisk for a long time, so this isnât a new belief, but it didnât make it into his senate testimony either. If my p(doom) was that high, and I was leading one of the worldâs top AI labs, Iâd be making a lot more noise. But, instead, theyâve recently announced partnerships with and taken money from Google and Amazon with the intent to develop frontier AI and, presumably, commercialise it.
Now, many people see this and go âthat doesntâ add upâ. Itâs what leads many critics of AI x-risk to go âthis is just a galaxy brained marketing strategyâ, and I hope this comment has given some clarification as to why they think so.
Connor, on the other hand, has been consistently against scaling up AI systems and has argued for this in public appearances on the news, on podcasts, and in public hearings. I donât agree with his pessimism or framing of the positions, but he has authenticity regarding his position. Itâs what people expect people to act like if they think the industry they work in has a significant chance of destroying the world, whereas I canât make sense of Amodeiâs position, for example, without guessing additional beliefs or information which he may or may not actually have.
So, tying it back to the intense scepticism of RSPs from the Connor/âctrl.ai/âConjecture side of the AI Safety space is because given a track record like the above they donât trust that RSPs are actually a mechanism that will reduce the risk from developing unaligned AGI. And I can see why theyâd be scepticalâif Iâd just accepted the promise of $4 billion from Amazon, thatâs a big incentive to put your thumb on the scale and say âweâre not at ASL-3 yetâ, for example.
The alternative might be titotalâs explanation that Sam/âDario have much lower p(doom) than Connor, and I think thatâs probably true. But that might not be true for every employee at these organisations, and even then I think 10% is a high enough p(doom) that one ought to report that estimate fully and faithfully to your democratically elected government if they ask you about it.
I hope this makes the âRSP scepticalâ position more clear to people, though I donât hold it myself (at least to the same extreme as Connor and Gabe do here), and Iâm open to corrections from either Connor or Gabe if you think Iâve interpreted what youâre trying to say incorrectly.
noteâeverything i say here is based on my own interpretation of the AI scene. I have no special knowledge, so this may be inaccurate, out-of-date, or both
So this post has provoked a lot of reaction, and Iâm not going to defend all of it. I do think, however, that debating what is lying/âwhite lying/âlying by omission/âbeing misleading is not as important as the other issues. (I think the tone is overly combative and hurts its cause, for instance). But from the perspective of someone like Connor, I can see why they may think that the AI Safety Community (especially those at the larger labs) are deliberately falsifying their beliefs.
On the May 16th Senate Judiciary Committee hearing where the witnesses are under oath[1], Blumenthal asks Sam Altman about his quote that âdevelopment of superhuman machine intelligence is probably the greatest threat to the continued existence of humanityâ and leads on to talk about the effect of AI on jobs and employment. Sam answers this by also talking about jobs, but Gary Marcus then calls him out by saying âSamâs worst fear I do not think is employment and he never told us what his worst fear actually isâ. When asked again, Sam says âmy worst fears are that...we cause significant harm to the worldâ[2]
I think thatâs a lie of omissionâSamâs worst fear is the extinction of humanity or perhaps some S-risk scenario, and he did not say that. Does this violate Section 1001 of the United States Code? I donât know. I think itâd be good to get more concrete estimates from Sam of what he thinks the risks are, and I think as the CEO of the worldâs leading AI company, itâd be something the US Senate should interested in.
In a recent podcast, Dario Amodei implied his p(doom) was between 10-25%, though itâs not a clear prediction with a timeframe, resolution criteria, or whether itâs conditional or unconditional on Anthropicâs actions (listen and judge for yourself). But Darioâs been aware of xRisk for a long time, so this isnât a new belief, but it didnât make it into his senate testimony either. If my p(doom) was that high, and I was leading one of the worldâs top AI labs, Iâd be making a lot more noise. But, instead, theyâve recently announced partnerships with and taken money from Google and Amazon with the intent to develop frontier AI and, presumably, commercialise it.
Now, many people see this and go âthat doesntâ add upâ. Itâs what leads many critics of AI x-risk to go âthis is just a galaxy brained marketing strategyâ, and I hope this comment has given some clarification as to why they think so.
Connor, on the other hand, has been consistently against scaling up AI systems and has argued for this in public appearances on the news, on podcasts, and in public hearings. I donât agree with his pessimism or framing of the positions, but he has authenticity regarding his position. Itâs what people expect people to act like if they think the industry they work in has a significant chance of destroying the world, whereas I canât make sense of Amodeiâs position, for example, without guessing additional beliefs or information which he may or may not actually have.
So, tying it back to the intense scepticism of RSPs from the Connor/âctrl.ai/âConjecture side of the AI Safety space is because given a track record like the above they donât trust that RSPs are actually a mechanism that will reduce the risk from developing unaligned AGI. And I can see why theyâd be scepticalâif Iâd just accepted the promise of $4 billion from Amazon, thatâs a big incentive to put your thumb on the scale and say âweâre not at ASL-3 yetâ, for example.
The alternative might be titotalâs explanation that Sam/âDario have much lower p(doom) than Connor, and I think thatâs probably true. But that might not be true for every employee at these organisations, and even then I think 10% is a high enough p(doom) that one ought to report that estimate fully and faithfully to your democratically elected government if they ask you about it.
I hope this makes the âRSP scepticalâ position more clear to people, though I donât hold it myself (at least to the same extreme as Connor and Gabe do here), and Iâm open to corrections from either Connor or Gabe if you think Iâve interpreted what youâre trying to say incorrectly.
I canât find a recording or transcript of what the oath was, but I assume it included a promise to tell the whole truth
The hearing is here, this exchange kicks off around 35:38