Hey Steven! As always I really appreciate your engagement here, and Iâm going to have to really simplify but I really appreciate your links[1] and Iâm definitely going to check them out đ
I think François is right, but I do think that work on AI safety is overwhelmingly valuable.
Hereâs an allegory:
I think the most relevant disagreement that we have[2]is the beginning of your allegory. To indulge it, I donât think we have knowledge of the intelligent alien species coming to earth, and to the extent we have a conceptual basis for them we canât see any signs of them in the sky. Pair this with the EA concern that we should be concerned about the counterfactual impact of our actions, and that there are opportunities to do good right here and now,[3] it shouldnât be a primary EA concern.
Now, what would make it a primary concern is if Dr S is right and that the aliens are spotted and that theyâre on their way, but I donât think heâs right. And, to stretch the analogy to breaking point, Iâd be very upset that after I turned my telescope to the co-ordinates Dr S mentions and seeing meteors instead of spaceships, that significant parts of the EA movement were still wanting to have more funding to construct the ultimate-anti-alien-space-laser or do alien-defence-research instead of buying bednets.
(When I say âAGIâ I think Iâm talking about the same thing that you called digital âbeingsâ in this comment.)
A secondary crux I have is that a âdigital beingâ in the sense I describe, and possibly the AGI you think of, will likely exhibit certain autopoietic properties that make it significantly different from either the paperclip maxermiser or a âfoom-ingâ ASI. This is highly speculative though, based on a lot of philosophical intuitions, and I wouldnât want to bet humanityâs future on it at all in the case where we did see aliens in the sky.
To be clear, you can definitely find some people in AI safety saying AGI is likely in <5 years, although Ajeya is not one of those people. This is a more extreme claim, and does seem pretty implausible unless LLMs will scale to AGI.
My take on it, though I admit driven by selection bias on Twitter, is that many people in the Bay-Social-Scene are buying into the <5 year timelines. Aschenbrenner for sure, Kokotajlo as well, and even maybe Amodei[4] as well? (Edit: Also lots of prominent AI Safety Twitter accounts seem to have bought fully into this worldview, such as the awful âAI Safety Memesâ account) However, I do agree itâs not all of AI Safety for sure! I just donât think it that, once you take away that urgency and certainy of the probelm, it ought to be considered the worldâs âmost pressing problemâ, at least without further controversial philosophical assumptions.
Iâd argue through increasing human flourishing and reducing the suffering we inflict on animals, but you could sub in your own cause area here for instance, e.g. âpreventing nuclear warâ if you thought that was both likely and an x-risk
See the transcript with Dwarkesh at 00:24:26 onwards where he says that superhuman/âtransformative AI capabilities will come within âa few yearsâ of the interviewâs date (so within a few years of summer 2023)
Pair this with the EA concern that we should be concerned about the counterfactual impact of our actions, and that there are opportunities to do good right here and now,[3] it shouldnât be a primary EA concern.
As in, your crux is that the probability of AGI within the next 50 years is less than 10%?
I think from an x-risk perspective it is quite hard to beat AI risk even on pretty long timelines. (Where the main question is bio risk and what you think about (likely temporary) civilizational collapse due to nuclear war.)
Itâs pretty plausible that on longer timelines technical alignment/âsafety work looks weak relative to other stuff focused on making AI go better.
As in, your crux is that the probability of AGI within the next 50 years is less than 10%?
Iâm essentially deeply uncertain about how to answer this question, in a true âKnightian Uncertaintyâ sense and I donât know how much it makes sense to use subjective probability calculus. It is also highly variable to what we mean by AGI though. I find many of the arguments Iâve seen to be a) deference to the subjective probabilities of others or b) extrapolation of straight lines on graphsâneither of which I find highly convincing. (I think your arguments seem stronger and more grounded fwiw)
I think from an x-risk perspective it is quite hard to beat AI risk even on pretty long timelines.
I think this can hold, but it holdâs not just in light of particular facts about AI progress now but in light of various strong philosophical beliefs about value, what future AI would be like, and how the future would be post the invention of said AI. You may have strong arguments for these, but I find many arguments for the overwhelming importance of AI Safety do very poorly to ground these, especially in the light of compelling interventions to good that exist in the world right now.
It is also highly variable to what we mean by AGI though.
Iâm happy to do timelines to the singularity and operationize this with âwe have the technological capacity to pretty easily build projects as impressive as a dyson sphereâ.
(Or 1000x electricity production, or whatever.)
In my views, this likely adds only a moderate number of years (3-20 depending on how various details go).
Hey Steven! As always I really appreciate your engagement here, and Iâm going to have to really simplify but I really appreciate your links[1] and Iâm definitely going to check them out đ
I think the most relevant disagreement that we have[2]is the beginning of your allegory. To indulge it, I donât think we have knowledge of the intelligent alien species coming to earth, and to the extent we have a conceptual basis for them we canât see any signs of them in the sky. Pair this with the EA concern that we should be concerned about the counterfactual impact of our actions, and that there are opportunities to do good right here and now,[3] it shouldnât be a primary EA concern.
Now, what would make it a primary concern is if Dr S is right and that the aliens are spotted and that theyâre on their way, but I donât think heâs right. And, to stretch the analogy to breaking point, Iâd be very upset that after I turned my telescope to the co-ordinates Dr S mentions and seeing meteors instead of spaceships, that significant parts of the EA movement were still wanting to have more funding to construct the ultimate-anti-alien-space-laser or do alien-defence-research instead of buying bednets.
A secondary crux I have is that a âdigital beingâ in the sense I describe, and possibly the AGI you think of, will likely exhibit certain autopoietic properties that make it significantly different from either the paperclip maxermiser or a âfoom-ingâ ASI. This is highly speculative though, based on a lot of philosophical intuitions, and I wouldnât want to bet humanityâs future on it at all in the case where we did see aliens in the sky.
My take on it, though I admit driven by selection bias on Twitter, is that many people in the Bay-Social-Scene are buying into the <5 year timelines. Aschenbrenner for sure, Kokotajlo as well, and even maybe Amodei[4] as well? (Edit: Also lots of prominent AI Safety Twitter accounts seem to have bought fully into this worldview, such as the awful âAI Safety Memesâ account) However, I do agree itâs not all of AI Safety for sure! I just donât think it that, once you take away that urgency and certainy of the probelm, it ought to be considered the worldâs âmost pressing problemâ, at least without further controversial philosophical assumptions.
I remember reading and liking your âLLM plateau-istâ piece.
I canât speak for all the otheres you mention, but fwiw I do agree with your frustrations at the AI risk discourse on various sides
Iâd argue through increasing human flourishing and reducing the suffering we inflict on animals, but you could sub in your own cause area here for instance, e.g. âpreventing nuclear warâ if you thought that was both likely and an x-risk
See the transcript with Dwarkesh at 00:24:26 onwards where he says that superhuman/âtransformative AI capabilities will come within âa few yearsâ of the interviewâs date (so within a few years of summer 2023)
As in, your crux is that the probability of AGI within the next 50 years is less than 10%?
I think from an x-risk perspective it is quite hard to beat AI risk even on pretty long timelines. (Where the main question is bio risk and what you think about (likely temporary) civilizational collapse due to nuclear war.)
Itâs pretty plausible that on longer timelines technical alignment/âsafety work looks weak relative to other stuff focused on making AI go better.
Iâm essentially deeply uncertain about how to answer this question, in a true âKnightian Uncertaintyâ sense and I donât know how much it makes sense to use subjective probability calculus. It is also highly variable to what we mean by AGI though. I find many of the arguments Iâve seen to be a) deference to the subjective probabilities of others or b) extrapolation of straight lines on graphsâneither of which I find highly convincing. (I think your arguments seem stronger and more grounded fwiw)
I think this can hold, but it holdâs not just in light of particular facts about AI progress now but in light of various strong philosophical beliefs about value, what future AI would be like, and how the future would be post the invention of said AI. You may have strong arguments for these, but I find many arguments for the overwhelming importance of AI Safety do very poorly to ground these, especially in the light of compelling interventions to good that exist in the world right now.
Iâm happy to do timelines to the singularity and operationize this with âwe have the technological capacity to pretty easily build projects as impressive as a dyson sphereâ.
(Or 1000x electricity production, or whatever.)
In my views, this likely adds only a moderate number of years (3-20 depending on how various details go).