(1) says that recursive self-improvement [RSI] has already begun
It is unclear to me what this means. Dario says āThis dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have describedā. I have gone through the instances of ārecursive self-improvementā and āRSIā in the linked sources, and I did not find any concrete description of what it means for RSI to start. There is a sense in which humanity has always been building on past knowledge.
(2) says that rogue agents could take over āthe entire Internetā within six to 12 months
Dario says āitās my worry that in 6ā12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrailsā. I doubt humans will lose control over the entire internet to bots over the next 12 months. I am open tobetsagainst short timelines for transformative AI (TAI), or what they supposedly imply, up to 10 k$.
Hi Vasco. I also feel unclear about what it means for RSI to properly begin; Iām assuming humans are still in the loop at Anthropic. Iām reminded of Toby Ordās analogy:
I like to think of it in terms of a group of hikers seeing a mountain in the distance, towering up into the clouds and beyond, with its snowy peak catching the sunās light. They talk animatedly about how amazing it would be to climb so high that they are inside a cloud. Or imagine being above the clouds, looking over them like an angel. After many hours of climbing, they notice there is a faint haze. Are they inside the cloud now? The mist gradually gets thicker until they can only see 10 metres ahead. Are they inside it now? Then it drops to 9 metres. Then 8. Then visibility starts to increase again. After an hour there is only the slightest haze. Are they above the clouds now? Another 30 minutes and there is no haze, and they can all agree they are above the clouds.
It is clear that at some point they were inside the cloud and sometime later were above it. And it is clear that these were sensible and useful concepts. For example, they took precautions like roping themselves together for the journey through the cloud due to the low visibility and took cameras with them because they knew they could take beautiful photos above the clouds. A lack of sharp boundaries doesnāt make these concepts useless. But they were admittedly a lot more useful when the hikers were on the ground, planning their route, and a lot less useful in the debatable boundary zones.
Has anybody taken you up on the bet yet? Iām not betting; I have no idea whatās going on!
I bet Greg Colbourn 10 kā¬ that AI will not kill us all by the end of 2027
Good luck, I hope you win!
If for some reason I am not able to decide (e.g. if I die before 2028), the transfer must be made to my lastly stated organisation of choice, currently The Humane League (THL).
Thanks. Me too. I am thinking about suggesting to Greg doing a similar bet resolving at the end of 2030, where I would initially donate to Gregās preferred charity what I win from the 1st bet plus some more money. I could probably bet like 40 k$ in total, and then win 80 k$ adjusted for inflation or growth in stocks at the end of 2030.
Where would you now like the donation to go?
I would make a donation to Rethink Priorities (RP) restricted to research on moral weights led by Bob Fischer. Here is some context.
Hi Ben. Thanks for sharing that.
It is unclear to me what this means. Dario says āThis dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have describedā. I have gone through the instances of ārecursive self-improvementā and āRSIā in the linked sources, and I did not find any concrete description of what it means for RSI to start. There is a sense in which humanity has always been building on past knowledge.
Dario says āitās my worry that in 6ā12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrailsā. I doubt humans will lose control over the entire internet to bots over the next 12 months. I am open to bets against short timelines for transformative AI (TAI), or what they supposedly imply, up to 10 k$.
Hi Vasco. I also feel unclear about what it means for RSI to properly begin; Iām assuming humans are still in the loop at Anthropic. Iām reminded of Toby Ordās analogy:
Has anybody taken you up on the bet yet? Iām not betting; I have no idea whatās going on!
I have this and this bets resolving at the end of 2027.
Fair and funny. I suggested the bet having other readers in mind.
Good luck, I hope you win!
Where would you now like the donation to go?
Thanks. Me too. I am thinking about suggesting to Greg doing a similar bet resolving at the end of 2030, where I would initially donate to Gregās preferred charity what I win from the 1st bet plus some more money. I could probably bet like 40 k$ in total, and then win 80 k$ adjusted for inflation or growth in stocks at the end of 2030.
I would make a donation to Rethink Priorities (RP) restricted to research on moral weights led by Bob Fischer. Here is some context.