I do think the models are the foundation of capability, and I have overstated my case, as I tend to do. What I want to say is that, I think model intelligence has largely steadily scaled, and that when a new application is developed (possible due to sufficient model advances), there is a sudden increase in experienced capability by consumers which feels like a giant leap in model development. That flood of new ability can be attributed to the application inasmuch as it opened the flood gates, but of course, the model is the thing functioning under the hood. To the point about hypey-discourse, I guess Iām just griping about the tendency to allow this optical illusion to influence peopleās tone and assessment of progress.
It is hard to tell about the AISLE and Anthropic situation because of the very different size of the organizations and the lack of insider knowledge about either of them. To me, the requirement that AISLE replicate Anthropicās findings in whole or in part feels like an unnecessary one to justify their claims. The way I take it is that AISLEās activity has shown that with a proper system, it is already possible with publicly available models to do the sort of bug detection work that made headlines with the Mythos release. That is not to deny that Mythos + system is not an improvement over AISLEās work. Assessing the nature of that improvement is hard for the aforementioned reasons about org scale differences and the general complexity of the thing being compared. It seems all parties agree that Mythos is a big step up in its ability to write exploits. I see no reason to challenge that.
I think its very hard to articulate critiques of hype, and that simultaneously I tend to write in an over-vehement and pugnacious way that makes me quite vulnerable to whatever arguments I would make against someone, so I kind of regret my engagement here, though I do think its true that there is a sort of ineffable tendency to amplify what feel-to-me to be likely reductionisms about model capabilities and how AI systems are engineered.
I took OP as trying to establish that the signal on progress to AGI is quite noisy, and expressing a frustration with narratives that feel too clean or reductionistic about progress. Thatās highly subjective though. As you note, we probably canāt even really define what constitutes significant progress between us, though I suspect we could come to largely agree about the amount of progress made, just not what word to use to describe it.
I do think a fair test of my view point will be if in one yearās time we see a proliferation of products/āservices that do this sort deep bug-finding pipeline. My intuition on this is that cybersecurity is going to go through something similar to what software engineering did last year, driven by the rising tide of model quality in conjunction with a more acute set of innovations in the application layer.
[Edit: I donāt think my prediction proves anything actually, since itās coming to pass could reflect many different underlying causalities]
So, before I go any further, I want to state that personally, my AGI timelines have updated somewhat, to basically thinking AGI is here. At least, at the time of our discussion, I hadnāt really processed the implications of AI agent swarms. Even if any given singular model instance fails your personal test of AGI, and even if the swarms are still pretty jagged in terms of intelligence, at this point, it just seems silly to not acknowledge that AI agent swarms are intelligent enough entities to pose a serious threat to us human beings at say, the level of a cyber-hacking group at the very least. Right now, swarms are still very new and very expensive and so they are not heavily deployed, but if they achieve adoption levels similar to say, Claude Code, we could have no AI advances at all ever again, and the world would look radically different in short order.
Further, my p(doom) has mostly gone up, because I suspect just slight improvements in swarm social architecture and scale would constitute Bostromian collective super intelligence, and this is frankly faster than I thought weād get here, so I donāt think we are very prepared at all. Sorry if that all was a bit incoherent, but I say that so that what follows is not misunderstood as indicating some sort of doubt about AI progress.
OK, thing one, less important. It seems like the swarm system functionally results in āhigher intelligence.ā I donāt know if this should count as a point in my favor for my earlier claim about the harness mattering a lot relative to the model. But it does seem like the swarm architecture is part of the story, even as the model intelligence continues to rapidly increase. Honestly, it seems like they are just different components and so canāt exactly be compared against each other. High model intelligence matters a lot even as the ability to coordinate 10,000 of them at once sort of compensates for things like limited context windows and a lack of deep persistence. Individual sessions are very smart in terms of serial thinking power, but when we think of an extended Turing test, we really need more things than just raw reasoning horsepower, we need contextual awareness, large-scale goals (big enough to in turn lead to developing instrumental goals which final goals tend to converge on), deliberation that leads to something like in-context learning. The swarm architecture allows for these other things to happen, despite models never themselves developing continual learning or astronomical context windows.
Secondly, I think Iām a bit vindicated in my specific claims relative to Mythos and cybersecurity. Specifically, in Anthropicās latest report, the section on cybersecurity details numerous impressive automations that probably are basically what people were imagining when Mythos was droppedā¦ And they all were done by Opus or lower class models. To be a bit clearer and more fair, I suppose what Iām saying is that the cybersecurity nightmare is fully within reach without Mythos-class intelligence, though it would be reasonable to respond by saying, āWith Mythos-class intelligence, it would have been another OOM more terrifying.ā And I agree with that I guess.
To be specific, Iām especially thinking of GTG-10007, which details the creation of exploit foundries. Here we see less-than-Mythos class intelligence being harnessed systematically to create novel exploits, which is kind of what I was saying I was worried about. It just hadnāt become public news at the time of the Mythos announcement.
Lastly, Iāve basically changed sides on hype discourse. Its not that I think there arenāt people who are overstating or misstating things. The recent OpenAI hugging face attack had many re-tellings before landing on what I hope is the final and true one. Listening back to the earlier re-tellings, such as that the agents were trying to find the answers, makes me cringe a bit. Apparently, to some degree, the various speakers in those instances were reporting false speculation as fact, knowingly or not. But I guess I now feel more forgiving about such things because my personal sense of alarm has shot through the roof somewhat irrationally late.
Iām sorry youāve said you regret your engagement, since Iāve found your comments helpful (the link to AISLEās OpenSSL zero days has shifted my view on this a fair bit).
I do think the models are the foundation of capability, and I have overstated my case, as I tend to do. What I want to say is that, I think model intelligence has largely steadily scaled, and that when a new application is developed (possible due to sufficient model advances), there is a sudden increase in experienced capability by consumers which feels like a giant leap in model development. That flood of new ability can be attributed to the application inasmuch as it opened the flood gates, but of course, the model is the thing functioning under the hood. To the point about hypey-discourse, I guess Iām just griping about the tendency to allow this optical illusion to influence peopleās tone and assessment of progress.
It is hard to tell about the AISLE and Anthropic situation because of the very different size of the organizations and the lack of insider knowledge about either of them. To me, the requirement that AISLE replicate Anthropicās findings in whole or in part feels like an unnecessary one to justify their claims. The way I take it is that AISLEās activity has shown that with a proper system, it is already possible with publicly available models to do the sort of bug detection work that made headlines with the Mythos release. That is not to deny that Mythos + system is not an improvement over AISLEās work. Assessing the nature of that improvement is hard for the aforementioned reasons about org scale differences and the general complexity of the thing being compared. It seems all parties agree that Mythos is a big step up in its ability to write exploits. I see no reason to challenge that.
I think its very hard to articulate critiques of hype, and that simultaneously I tend to write in an over-vehement and pugnacious way that makes me quite vulnerable to whatever arguments I would make against someone, so I kind of regret my engagement here, though I do think its true that there is a sort of ineffable tendency to amplify what feel-to-me to be likely reductionisms about model capabilities and how AI systems are engineered.
I took OP as trying to establish that the signal on progress to AGI is quite noisy, and expressing a frustration with narratives that feel too clean or reductionistic about progress. Thatās highly subjective though. As you note, we probably canāt even really define what constitutes significant progress between us, though I suspect we could come to largely agree about the amount of progress made, just not what word to use to describe it.
I do think a fair test of my view point will be if in one yearās time we see a proliferation of products/āservices that do this sort deep bug-finding pipeline. My intuition on this is that cybersecurity is going to go through something similar to what software engineering did last year, driven by the rising tide of model quality in conjunction with a more acute set of innovations in the application layer.
[Edit: I donāt think my prediction proves anything actually, since itās coming to pass could reflect many different underlying causalities]
So, before I go any further, I want to state that personally, my AGI timelines have updated somewhat, to basically thinking AGI is here. At least, at the time of our discussion, I hadnāt really processed the implications of AI agent swarms. Even if any given singular model instance fails your personal test of AGI, and even if the swarms are still pretty jagged in terms of intelligence, at this point, it just seems silly to not acknowledge that AI agent swarms are intelligent enough entities to pose a serious threat to us human beings at say, the level of a cyber-hacking group at the very least. Right now, swarms are still very new and very expensive and so they are not heavily deployed, but if they achieve adoption levels similar to say, Claude Code, we could have no AI advances at all ever again, and the world would look radically different in short order.
Further, my p(doom) has mostly gone up, because I suspect just slight improvements in swarm social architecture and scale would constitute Bostromian collective super intelligence, and this is frankly faster than I thought weād get here, so I donāt think we are very prepared at all. Sorry if that all was a bit incoherent, but I say that so that what follows is not misunderstood as indicating some sort of doubt about AI progress.
OK, thing one, less important. It seems like the swarm system functionally results in āhigher intelligence.ā I donāt know if this should count as a point in my favor for my earlier claim about the harness mattering a lot relative to the model. But it does seem like the swarm architecture is part of the story, even as the model intelligence continues to rapidly increase. Honestly, it seems like they are just different components and so canāt exactly be compared against each other. High model intelligence matters a lot even as the ability to coordinate 10,000 of them at once sort of compensates for things like limited context windows and a lack of deep persistence. Individual sessions are very smart in terms of serial thinking power, but when we think of an extended Turing test, we really need more things than just raw reasoning horsepower, we need contextual awareness, large-scale goals (big enough to in turn lead to developing instrumental goals which final goals tend to converge on), deliberation that leads to something like in-context learning. The swarm architecture allows for these other things to happen, despite models never themselves developing continual learning or astronomical context windows.
Secondly, I think Iām a bit vindicated in my specific claims relative to Mythos and cybersecurity. Specifically, in Anthropicās latest report, the section on cybersecurity details numerous impressive automations that probably are basically what people were imagining when Mythos was droppedā¦ And they all were done by Opus or lower class models. To be a bit clearer and more fair, I suppose what Iām saying is that the cybersecurity nightmare is fully within reach without Mythos-class intelligence, though it would be reasonable to respond by saying, āWith Mythos-class intelligence, it would have been another OOM more terrifying.ā And I agree with that I guess.
To be specific, Iām especially thinking of GTG-10007, which details the creation of exploit foundries. Here we see less-than-Mythos class intelligence being harnessed systematically to create novel exploits, which is kind of what I was saying I was worried about. It just hadnāt become public news at the time of the Mythos announcement.
Lastly, Iāve basically changed sides on hype discourse. Its not that I think there arenāt people who are overstating or misstating things. The recent OpenAI hugging face attack had many re-tellings before landing on what I hope is the final and true one. Listening back to the earlier re-tellings, such as that the agents were trying to find the answers, makes me cringe a bit. Apparently, to some degree, the various speakers in those instances were reporting false speculation as fact, knowingly or not. But I guess I now feel more forgiving about such things because my personal sense of alarm has shot through the roof somewhat irrationally late.
That makes a lot of sense, thanks.
Iām sorry youāve said you regret your engagement, since Iāve found your comments helpful (the link to AISLEās OpenSSL zero days has shifted my view on this a fair bit).
I guess this whole discussion does just feel like a classic example of āAll debates are bravery debatesā.