Thanks for the detailed reply, I understand your point clearly now I think!
But $20,000 for *all* of the OpenBSD bugs (not just the published ones) doesn’t sound like that much to spend on inference compute to me. If AISLE could have spent the same and made an equally impressive announcement, unearthing enough bugs at once that government ministers around the world start issuing statements about it, then shouldn’t they have been able to find the investors to fund that? That would have been incredible publicity for them.
The crux for me seems to be whether they have made equally impressive announcements, as you suggest they might have done. Maybe they’re just worse at marketing. I don’t know enough to evaluate that claim properly, but that does seem the relevant question here: have Anthropic been able to use Mythos to go significantly beyond what the best harnesses could already achieve with existing models for the same inference spend? I thought the answer was a clear yes, and I didn’t find the original linked AISLE writeup very convincing at all. Your comment has made me more uncertain, but has still not convinced me, and I’d be really interested to read something more in depth on that question. (Maybe we also would disagree about what the word ‘significantly’ means here, since I guess you are acknowledging it probably represents some improvement).
(Also, I’d push back a bit on your characterization of AI progress. I agree the scaffolding is extremely important, but in my experience the “paradigm shifts” in capability over the last two and a half years I’ve been working with them have come from the models)
(And extra comment: the fact that cybersecurity capabilities might not imply imminent superintelligence takeoff seems an entirely independent point that I don’t necessarily disagree with)
I wanted to follow up on this thread and bring in some additional evidence on the whole AISLE thing. It isn’t definitive or anything, but both AISLE and Mythos have been used to scan curl and the results are interesting.
- AISLE identified five CVEs and 24 bugs (plus two more CVEs in a dependency). - Mythos identified 1 CVE and potentially 20 bugs.
Now, Mythos scanned after AISLE. We don’t know what would have happened if Mythos had come first. But here’s some quotes by the maintainer of curl about the Mythos results:
> curl is certainly getting better thanks to this report, but counted by the volume of issues found, all the previous AI tools we have used have resulted in larger bugfix amounts.
> I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing.
> Any project that has not scanned their source code with AI powered tooling will likely find huge number of flaws, bugs and possible vulnerabilities with this new generation of tools. Mythos will, and so will many of the others.
Daniel’s blog post has some not great English, and also his tone is a bit less than objective. And of course, this is not a rigorous scientific comparison. But he is an expert in software and writing secure software, and he has demonstrated a willingness to change his mind on AI for cybersecurity,[1] so I think he’s worth listening to.
Interestingly, the same harness is not as successful at recreating AISLE’s own results, but this is all a bit selective.
To my earlier claim that harnesses and pipelines enabling Mythos like results can and will be built in the near future, AISLE open-sourced nano-analyzer, the harness used in this blog to discover maybe as many as 40 bugs in FreeBSD, so it would appear that my prediction was fulfilled at the time of writing...
AISLE quotes $100 in spend to find the 40 bugs in FreeBSD and to recreate a Mythos result. That is presumably going to be 1/100th or less of compute spend by Anthropic. To me, this causes me to think that the compute spend by Anthropic is playing an important role in the whole assessment of the Mythos results. My belief now is that Anthropic’s high compute spend is probably due to an inefficient pipeline, which is somewhat of a reversal of my earlier belief that the pipeline was the key determinant in model capabilities. However, this doesn’t mean that I think model capabilities are hugely improved—it just means that I think that with a better pipeline, they would have spent less to get these results, given that AISLE was able to find some of these results for $100 with dumber models.
Sorry to turn this into an infinitely extended thread, but I wanted to post yet more data, namely CloudFlare’s recent write up of their chance to work with Project Glasswing.
They do not provide numbers on bugs/vulnerabilities found, but they do provide some interesting commentary. Like others, they note that where Mythos stands out is in its ability to put together working exploits, and they elaborate on the value of this: proof-of-concept exploits are obviously worth reviewing; they are far less likely to be false positives.
They talk about the inadequacy of simply pointing a model at a codebase, and advocate for building pipelines and harnesses that enable the model to stay on task and counteract some of its reward-seeking behaviors. In an aside, they do make a passing comparison of Mythos to other frontier LLMs.
> When we ran other frontier models through the same harness, they found a fair number of the same underlying bugs, and in some cases they got further than we expected on the reasoning side too. Where they fell short was at the point of stitching the pieces together. A model would identify an interesting bug, write a thoughtful description of why it mattered, and then stop, leaving the actual chain unfinished and the question of exploitability open. What changed with Mythos Preview is that a model can now take those low-severity bugs (which would traditionally sit invisible in a backlog) and chain them into a single, more severe exploit.
To me, this write up is only so valuable. On the one hand, it is evidence for what I’ve been saying—that an important part of unlocking model capabilities in cybersecurity is the development of adequate harnesses. Raw model intelligence is not enough for such a complex task. On the other hand, as is evident in the write up, the model intelligence is the foundation of capabilities, and without an adequate supply of it, the task is impossible.
So, I don’t know that this really moves the needle on our broader discussion of whether “Mythos is overhyped,” though I do think it supports some of my intermediate claims.
I actually have an update for you on this! AISLE also came after Mythos (having found a single low-severity CVE + 4 things that turned out to be false positives), and actually got 6 (!!) new CVEs assigned that Mythos has completely missed, including the oldest issue ever to be found in curl. Check this post out by Daniel Stenberg, the lead developer of curl himself: https://mastodon.social/@bagder/116807317163361428
In addition, we are very clearly able to go into a codebase after Mythos is “done” with it, and find zero-days that it misses and have them assigned with publicly tracked CVEs (even in FreeBSD, the very codebase Anthropic chose to demonstrate Mythos’ capabilities). I make the data-driven case for it here: https://stanislavfort.substack.com/p/mythos-at-home-and-its-called-aisle, I think you might enjoy reading it!
Lastly, UC Berkeley is running a continuous academic study on the roly of AI in zero-day discovery (https://vuln.cs.berkeley.edu/). AISLE outranks Anthropic in 4 of the 8 categories they track, namely:
total number of zero-days with CVEs (the only way to actually track them reliably)
total number of critical zero-days with CVEs
coverage of vulnerability types (CWEs)
coverage of the MITRE top-25 most important vulnerability types
It is very clear that Mythos is not by any margin significantly better in finding zero-days than other, even much cheaper models, purposefully finetuned for the task + with a great harness.
Disclaimer: I’m the founder and CTO/Chief Scientist of AISLE
Thank you for jumping in and joining the discussion! I’m particularly excited to read more into the Berkeley Vulnerability Initiative, just to understand the project’s larger goals and techniques.
I think, as per other comments, that determining whether something is hyped or not is pretty subjective, but I do feel that my main prediction, that carefully scaffolded harnesses and systems would be reliably pumping out Mythos-class vulnerability discoveries by the end of the year, is pretty clearly already true.
I do think its clear that progress in AI offensive cyber capabilities is a real and present concern—I find myself actively worrying about the digital money in my bank accounts—so, I would consider that a valid takeaway from Mythos and subsequent developments. We are clearly very close to a situation where AI progress could endanger a lot of our digital infrastructure, if it just had the funding and someone with bad motivations to push it. That’s a big change from how I perceived the world one year ago, and seems worth being excited about it.
It also, to the technical point of this discussion, seems to be a capability gain that is gated on model intelligence, or at least, that’s how it looks to someone like me, with a naïve picture of the current state of model training/development.
I do think the models are the foundation of capability, and I have overstated my case, as I tend to do. What I want to say is that, I think model intelligence has largely steadily scaled, and that when a new application is developed (possible due to sufficient model advances), there is a sudden increase in experienced capability by consumers which feels like a giant leap in model development. That flood of new ability can be attributed to the application inasmuch as it opened the flood gates, but of course, the model is the thing functioning under the hood. To the point about hypey-discourse, I guess I’m just griping about the tendency to allow this optical illusion to influence people’s tone and assessment of progress.
It is hard to tell about the AISLE and Anthropic situation because of the very different size of the organizations and the lack of insider knowledge about either of them. To me, the requirement that AISLE replicate Anthropic’s findings in whole or in part feels like an unnecessary one to justify their claims. The way I take it is that AISLE’s activity has shown that with a proper system, it is already possible with publicly available models to do the sort of bug detection work that made headlines with the Mythos release. That is not to deny that Mythos + system is not an improvement over AISLE’s work. Assessing the nature of that improvement is hard for the aforementioned reasons about org scale differences and the general complexity of the thing being compared. It seems all parties agree that Mythos is a big step up in its ability to write exploits. I see no reason to challenge that.
I think its very hard to articulate critiques of hype, and that simultaneously I tend to write in an over-vehement and pugnacious way that makes me quite vulnerable to whatever arguments I would make against someone, so I kind of regret my engagement here, though I do think its true that there is a sort of ineffable tendency to amplify what feel-to-me to be likely reductionisms about model capabilities and how AI systems are engineered.
I took OP as trying to establish that the signal on progress to AGI is quite noisy, and expressing a frustration with narratives that feel too clean or reductionistic about progress. That’s highly subjective though. As you note, we probably can’t even really define what constitutes significant progress between us, though I suspect we could come to largely agree about the amount of progress made, just not what word to use to describe it.
I do think a fair test of my view point will be if in one year’s time we see a proliferation of products/services that do this sort deep bug-finding pipeline. My intuition on this is that cybersecurity is going to go through something similar to what software engineering did last year, driven by the rising tide of model quality in conjunction with a more acute set of innovations in the application layer.
[Edit: I don’t think my prediction proves anything actually, since it’s coming to pass could reflect many different underlying causalities]
I’m sorry you’ve said you regret your engagement, since I’ve found your comments helpful (the link to AISLE’s OpenSSL zero days has shifted my view on this a fair bit).
Thanks for the detailed reply, I understand your point clearly now I think!
But $20,000 for *all* of the OpenBSD bugs (not just the published ones) doesn’t sound like that much to spend on inference compute to me. If AISLE could have spent the same and made an equally impressive announcement, unearthing enough bugs at once that government ministers around the world start issuing statements about it, then shouldn’t they have been able to find the investors to fund that? That would have been incredible publicity for them.
The crux for me seems to be whether they have made equally impressive announcements, as you suggest they might have done. Maybe they’re just worse at marketing. I don’t know enough to evaluate that claim properly, but that does seem the relevant question here: have Anthropic been able to use Mythos to go significantly beyond what the best harnesses could already achieve with existing models for the same inference spend? I thought the answer was a clear yes, and I didn’t find the original linked AISLE writeup very convincing at all. Your comment has made me more uncertain, but has still not convinced me, and I’d be really interested to read something more in depth on that question. (Maybe we also would disagree about what the word ‘significantly’ means here, since I guess you are acknowledging it probably represents some improvement).
(Also, I’d push back a bit on your characterization of AI progress. I agree the scaffolding is extremely important, but in my experience the “paradigm shifts” in capability over the last two and a half years I’ve been working with them have come from the models)
(And extra comment: the fact that cybersecurity capabilities might not imply imminent superintelligence takeoff seems an entirely independent point that I don’t necessarily disagree with)
I wanted to follow up on this thread and bring in some additional evidence on the whole AISLE thing. It isn’t definitive or anything, but both AISLE and Mythos have been used to scan curl and the results are interesting.
- AISLE identified five CVEs and 24 bugs (plus two more CVEs in a dependency).
- Mythos identified 1 CVE and potentially 20 bugs.
Now, Mythos scanned after AISLE. We don’t know what would have happened if Mythos had come first. But here’s some quotes by the maintainer of curl about the Mythos results:
> curl is certainly getting better thanks to this report, but counted by the volume of issues found, all the previous AI tools we have used have resulted in larger bugfix amounts.
> I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing.
> Any project that has not scanned their source code with AI powered tooling will likely find huge number of flaws, bugs and possible vulnerabilities with this new generation of tools. Mythos will, and so will many of the others.
Quotes taken from here: https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-vulnerability/
Daniel’s blog post has some not great English, and also his tone is a bit less than objective. And of course, this is not a rigorous scientific comparison. But he is an expert in software and writing secure software, and he has demonstrated a willingness to change his mind on AI for cybersecurity,[1] so I think he’s worth listening to.
AISLE wrote a little news release about their findings vis-a-vis Mythos in the curl project: https://aisle.com/blog/curl-adopts-aisle-after-its-ai-agents-discovered-5-cves
AISLE also wrote up another blog post building on the earlier one. In this case, they showcase a simple pipeline that is able to recreate a Mythos result without any steering towards the relevant snippet of code: https://aisle.com/blog/system-over-model-zero-day-discovery-at-the-jagged-frontier
Interestingly, the same harness is not as successful at recreating AISLE’s own results, but this is all a bit selective.
To my earlier claim that harnesses and pipelines enabling Mythos like results can and will be built in the near future, AISLE open-sourced nano-analyzer, the harness used in this blog to discover maybe as many as 40 bugs in FreeBSD, so it would appear that my prediction was fulfilled at the time of writing...
AISLE quotes $100 in spend to find the 40 bugs in FreeBSD and to recreate a Mythos result. That is presumably going to be 1/100th or less of compute spend by Anthropic. To me, this causes me to think that the compute spend by Anthropic is playing an important role in the whole assessment of the Mythos results. My belief now is that Anthropic’s high compute spend is probably due to an inefficient pipeline, which is somewhat of a reversal of my earlier belief that the pipeline was the key determinant in model capabilities. However, this doesn’t mean that I think model capabilities are hugely improved—it just means that I think that with a better pipeline, they would have spent less to get these results, given that AISLE was able to find some of these results for $100 with dumber models.
Here’s Daniel’s highly negative blog post on AI cybersecurity work from last Summer: https://daniel.haxx.se/blog/2025/07/14/death-by-a-thousand-slops/ Here’s where he changed his mind just a few months later after ZeroPath started to systematically find bugs using AI: https://daniel.haxx.se/blog/2025/10/10/a-new-breed-of-analyzers/
Sorry to turn this into an infinitely extended thread, but I wanted to post yet more data, namely CloudFlare’s recent write up of their chance to work with Project Glasswing.
They do not provide numbers on bugs/vulnerabilities found, but they do provide some interesting commentary. Like others, they note that where Mythos stands out is in its ability to put together working exploits, and they elaborate on the value of this: proof-of-concept exploits are obviously worth reviewing; they are far less likely to be false positives.
They talk about the inadequacy of simply pointing a model at a codebase, and advocate for building pipelines and harnesses that enable the model to stay on task and counteract some of its reward-seeking behaviors. In an aside, they do make a passing comparison of Mythos to other frontier LLMs.
> When we ran other frontier models through the same harness, they found a fair number of the same underlying bugs, and in some cases they got further than we expected on the reasoning side too. Where they fell short was at the point of stitching the pieces together. A model would identify an interesting bug, write a thoughtful description of why it mattered, and then stop, leaving the actual chain unfinished and the question of exploitability open. What changed with Mythos Preview is that a model can now take those low-severity bugs (which would traditionally sit invisible in a backlog) and chain them into a single, more severe exploit.
To me, this write up is only so valuable. On the one hand, it is evidence for what I’ve been saying—that an important part of unlocking model capabilities in cybersecurity is the development of adequate harnesses. Raw model intelligence is not enough for such a complex task. On the other hand, as is evident in the write up, the model intelligence is the foundation of capabilities, and without an adequate supply of it, the task is impossible.
So, I don’t know that this really moves the needle on our broader discussion of whether “Mythos is overhyped,” though I do think it supports some of my intermediate claims.
I actually have an update for you on this! AISLE also came after Mythos (having found a single low-severity CVE + 4 things that turned out to be false positives), and actually got 6 (!!) new CVEs assigned that Mythos has completely missed, including the oldest issue ever to be found in curl. Check this post out by Daniel Stenberg, the lead developer of curl himself: https://mastodon.social/@bagder/116807317163361428
In addition, we are very clearly able to go into a codebase after Mythos is “done” with it, and find zero-days that it misses and have them assigned with publicly tracked CVEs (even in FreeBSD, the very codebase Anthropic chose to demonstrate Mythos’ capabilities). I make the data-driven case for it here: https://stanislavfort.substack.com/p/mythos-at-home-and-its-called-aisle, I think you might enjoy reading it!
Lastly, UC Berkeley is running a continuous academic study on the roly of AI in zero-day discovery (https://vuln.cs.berkeley.edu/). AISLE outranks Anthropic in 4 of the 8 categories they track, namely:
total number of zero-days with CVEs (the only way to actually track them reliably)
total number of critical zero-days with CVEs
coverage of vulnerability types (CWEs)
coverage of the MITRE top-25 most important vulnerability types
It is very clear that Mythos is not by any margin significantly better in finding zero-days than other, even much cheaper models, purposefully finetuned for the task + with a great harness.
Disclaimer: I’m the founder and CTO/Chief Scientist of AISLE
Thank you for jumping in and joining the discussion! I’m particularly excited to read more into the Berkeley Vulnerability Initiative, just to understand the project’s larger goals and techniques.
I think, as per other comments, that determining whether something is hyped or not is pretty subjective, but I do feel that my main prediction, that carefully scaffolded harnesses and systems would be reliably pumping out Mythos-class vulnerability discoveries by the end of the year, is pretty clearly already true.
I do think its clear that progress in AI offensive cyber capabilities is a real and present concern—I find myself actively worrying about the digital money in my bank accounts—so, I would consider that a valid takeaway from Mythos and subsequent developments. We are clearly very close to a situation where AI progress could endanger a lot of our digital infrastructure, if it just had the funding and someone with bad motivations to push it. That’s a big change from how I perceived the world one year ago, and seems worth being excited about it.
It also, to the technical point of this discussion, seems to be a capability gain that is gated on model intelligence, or at least, that’s how it looks to someone like me, with a naïve picture of the current state of model training/development.
This is all very interesting, thanks for following up!
I do think the models are the foundation of capability, and I have overstated my case, as I tend to do. What I want to say is that, I think model intelligence has largely steadily scaled, and that when a new application is developed (possible due to sufficient model advances), there is a sudden increase in experienced capability by consumers which feels like a giant leap in model development. That flood of new ability can be attributed to the application inasmuch as it opened the flood gates, but of course, the model is the thing functioning under the hood. To the point about hypey-discourse, I guess I’m just griping about the tendency to allow this optical illusion to influence people’s tone and assessment of progress.
It is hard to tell about the AISLE and Anthropic situation because of the very different size of the organizations and the lack of insider knowledge about either of them. To me, the requirement that AISLE replicate Anthropic’s findings in whole or in part feels like an unnecessary one to justify their claims. The way I take it is that AISLE’s activity has shown that with a proper system, it is already possible with publicly available models to do the sort of bug detection work that made headlines with the Mythos release. That is not to deny that Mythos + system is not an improvement over AISLE’s work. Assessing the nature of that improvement is hard for the aforementioned reasons about org scale differences and the general complexity of the thing being compared. It seems all parties agree that Mythos is a big step up in its ability to write exploits. I see no reason to challenge that.
I think its very hard to articulate critiques of hype, and that simultaneously I tend to write in an over-vehement and pugnacious way that makes me quite vulnerable to whatever arguments I would make against someone, so I kind of regret my engagement here, though I do think its true that there is a sort of ineffable tendency to amplify what feel-to-me to be likely reductionisms about model capabilities and how AI systems are engineered.
I took OP as trying to establish that the signal on progress to AGI is quite noisy, and expressing a frustration with narratives that feel too clean or reductionistic about progress. That’s highly subjective though. As you note, we probably can’t even really define what constitutes significant progress between us, though I suspect we could come to largely agree about the amount of progress made, just not what word to use to describe it.
I do think a fair test of my view point will be if in one year’s time we see a proliferation of products/services that do this sort deep bug-finding pipeline. My intuition on this is that cybersecurity is going to go through something similar to what software engineering did last year, driven by the rising tide of model quality in conjunction with a more acute set of innovations in the application layer.
[Edit: I don’t think my prediction proves anything actually, since it’s coming to pass could reflect many different underlying causalities]
That makes a lot of sense, thanks.
I’m sorry you’ve said you regret your engagement, since I’ve found your comments helpful (the link to AISLE’s OpenSSL zero days has shifted my view on this a fair bit).
I guess this whole discussion does just feel like a classic example of “All debates are bravery debates”.