So, before I go any further, I want to state that personally, my AGI timelines have updated somewhat, to basically thinking AGI is here. At least, at the time of our discussion, I hadnāt really processed the implications of AI agent swarms. Even if any given singular model instance fails your personal test of AGI, and even if the swarms are still pretty jagged in terms of intelligence, at this point, it just seems silly to not acknowledge that AI agent swarms are intelligent enough entities to pose a serious threat to us human beings at say, the level of a cyber-hacking group at the very least. Right now, swarms are still very new and very expensive and so they are not heavily deployed, but if they achieve adoption levels similar to say, Claude Code, we could have no AI advances at all ever again, and the world would look radically different in short order.
Further, my p(doom) has mostly gone up, because I suspect just slight improvements in swarm social architecture and scale would constitute Bostromian collective super intelligence, and this is frankly faster than I thought weād get here, so I donāt think we are very prepared at all. Sorry if that all was a bit incoherent, but I say that so that what follows is not misunderstood as indicating some sort of doubt about AI progress.
OK, thing one, less important. It seems like the swarm system functionally results in āhigher intelligence.ā I donāt know if this should count as a point in my favor for my earlier claim about the harness mattering a lot relative to the model. But it does seem like the swarm architecture is part of the story, even as the model intelligence continues to rapidly increase. Honestly, it seems like they are just different components and so canāt exactly be compared against each other. High model intelligence matters a lot even as the ability to coordinate 10,000 of them at once sort of compensates for things like limited context windows and a lack of deep persistence. Individual sessions are very smart in terms of serial thinking power, but when we think of an extended Turing test, we really need more things than just raw reasoning horsepower, we need contextual awareness, large-scale goals (big enough to in turn lead to developing instrumental goals which final goals tend to converge on), deliberation that leads to something like in-context learning. The swarm architecture allows for these other things to happen, despite models never themselves developing continual learning or astronomical context windows.
Secondly, I think Iām a bit vindicated in my specific claims relative to Mythos and cybersecurity. Specifically, in Anthropicās latest report, the section on cybersecurity details numerous impressive automations that probably are basically what people were imagining when Mythos was droppedā¦ And they all were done by Opus or lower class models. To be a bit clearer and more fair, I suppose what Iām saying is that the cybersecurity nightmare is fully within reach without Mythos-class intelligence, though it would be reasonable to respond by saying, āWith Mythos-class intelligence, it would have been another OOM more terrifying.ā And I agree with that I guess.
To be specific, Iām especially thinking of GTG-10007, which details the creation of exploit foundries. Here we see less-than-Mythos class intelligence being harnessed systematically to create novel exploits, which is kind of what I was saying I was worried about. It just hadnāt become public news at the time of the Mythos announcement.
Lastly, Iāve basically changed sides on hype discourse. Its not that I think there arenāt people who are overstating or misstating things. The recent OpenAI hugging face attack had many re-tellings before landing on what I hope is the final and true one. Listening back to the earlier re-tellings, such as that the agents were trying to find the answers, makes me cringe a bit. Apparently, to some degree, the various speakers in those instances were reporting false speculation as fact, knowingly or not. But I guess I now feel more forgiving about such things because my personal sense of alarm has shot through the roof somewhat irrationally late.
So, before I go any further, I want to state that personally, my AGI timelines have updated somewhat, to basically thinking AGI is here. At least, at the time of our discussion, I hadnāt really processed the implications of AI agent swarms. Even if any given singular model instance fails your personal test of AGI, and even if the swarms are still pretty jagged in terms of intelligence, at this point, it just seems silly to not acknowledge that AI agent swarms are intelligent enough entities to pose a serious threat to us human beings at say, the level of a cyber-hacking group at the very least. Right now, swarms are still very new and very expensive and so they are not heavily deployed, but if they achieve adoption levels similar to say, Claude Code, we could have no AI advances at all ever again, and the world would look radically different in short order.
Further, my p(doom) has mostly gone up, because I suspect just slight improvements in swarm social architecture and scale would constitute Bostromian collective super intelligence, and this is frankly faster than I thought weād get here, so I donāt think we are very prepared at all. Sorry if that all was a bit incoherent, but I say that so that what follows is not misunderstood as indicating some sort of doubt about AI progress.
OK, thing one, less important. It seems like the swarm system functionally results in āhigher intelligence.ā I donāt know if this should count as a point in my favor for my earlier claim about the harness mattering a lot relative to the model. But it does seem like the swarm architecture is part of the story, even as the model intelligence continues to rapidly increase. Honestly, it seems like they are just different components and so canāt exactly be compared against each other. High model intelligence matters a lot even as the ability to coordinate 10,000 of them at once sort of compensates for things like limited context windows and a lack of deep persistence. Individual sessions are very smart in terms of serial thinking power, but when we think of an extended Turing test, we really need more things than just raw reasoning horsepower, we need contextual awareness, large-scale goals (big enough to in turn lead to developing instrumental goals which final goals tend to converge on), deliberation that leads to something like in-context learning. The swarm architecture allows for these other things to happen, despite models never themselves developing continual learning or astronomical context windows.
Secondly, I think Iām a bit vindicated in my specific claims relative to Mythos and cybersecurity. Specifically, in Anthropicās latest report, the section on cybersecurity details numerous impressive automations that probably are basically what people were imagining when Mythos was droppedā¦ And they all were done by Opus or lower class models. To be a bit clearer and more fair, I suppose what Iām saying is that the cybersecurity nightmare is fully within reach without Mythos-class intelligence, though it would be reasonable to respond by saying, āWith Mythos-class intelligence, it would have been another OOM more terrifying.ā And I agree with that I guess.
To be specific, Iām especially thinking of GTG-10007, which details the creation of exploit foundries. Here we see less-than-Mythos class intelligence being harnessed systematically to create novel exploits, which is kind of what I was saying I was worried about. It just hadnāt become public news at the time of the Mythos announcement.
Lastly, Iāve basically changed sides on hype discourse. Its not that I think there arenāt people who are overstating or misstating things. The recent OpenAI hugging face attack had many re-tellings before landing on what I hope is the final and true one. Listening back to the earlier re-tellings, such as that the agents were trying to find the answers, makes me cringe a bit. Apparently, to some degree, the various speakers in those instances were reporting false speculation as fact, knowingly or not. But I guess I now feel more forgiving about such things because my personal sense of alarm has shot through the roof somewhat irrationally late.