I wrote more about it here: https://linch.substack.com/p/openai-huggingface-hack
Linch
Karma: 28,484
More people should write basic AI safety explainers
[Fiction] You’re Absolutely Right
My summary: In a cybersecurity evaluation, OpenAI’s models, apparently autonomously and without any direct human direction, escaped their sandbox and successfully hacked a third-party company (HuggingFace).
The process involved leveraging a zero-day exploit to escape their sandbox, moving laterally across different OpenAI servers until they found a node with internet access, searching the internet and determining that the answers they wanted might be stored at HuggingFace, then leveraging multiple novel zero-day exploits to hack HuggingFace.
HuggingFace claimed that the models took thousands of independent actions across a swarm of short-lived sandboxes, “comprised of more than 17,000 recorded events.”
While technically a security evaluation with reduced safeguards, these actions are clearly out of bounds even in that context. It’s like being told to be creative and then breaking into your professor’s house and stealing the answer key. Worse than that, it’s not even your professor in this case, more like your professor’s friend.
Any human security researcher or engineer in a similar position would be fired on the spot. There is absolutely no valid reason to steal evaluation answers from an unaffiliated third party.
Furthermore, if I’m reading between the lines correctly, OpenAI did not address the issue (and perhaps didn’t even know about their models doing this) until after HuggingFace’s public blog post.
This leads me to suspect that there might be other major autonomous cybersecurity incidents that we do not yet know about.
I think a lot of ppl studying empirical persuasion results of AI tend to overestimate the ecological validity and generalizability of those results. This isn’t to say that studying empirical persuasion results is useless, just a) we should move towards better study designs in the future, and b) when looking at empirical persuasion studies you need to think carefully about not just the methodology and results but also what’s reasonable to generalize and not.
I think a lot of people understand in the abstract that “ecological validity is important” but don’t appreciate how difficult this is for persuasion results.
it’s actually just very hard to design a study that captures plausible real-world interactions, for practical, procedurally ethical, and infohazard-y reasons.
It’s very easy for empirical results to understate the influence of existing AI persuasion (see below)
It’s also very easy for empirical results to overstate the influence of existing AI persuasion (see below)
Our studies give us very limited ability to predict the timing of the superpersuasive regime.
There’s no clean analytic tool that allows us to draw lines on graphs to nearly the level of the METR time horizon graph
despite its many problems, I think the METR graph is much better at capturing something deeply true about software automation, compared to the studies we have at capturing persuasion ability or automation.
tbh I’m not sure (and currently lean against) the ability of the existing study methodologies can even allow us to reliably retrodict superhuman persuasion
as in, I don’t think the existing tools can tell us that the AIs are reliably superhuman even after it occurs.
This seems concerning!
Furthermore, the AI labs aren’t optimizing as much for AI persuasion as they are for other capabilities like coding, so we should expect some latent un-optimized for capabilities. It means that we shouldn’t be shocked if sudden leaps happen.
Our studies give us very limited ability to predict the shape of future superhuman capabilities.
For example, in Hackenburg (2026), when AIs outpersuade expert human debaters in single-session debates, they do so by throwing a ton of facts at you. Do we expect this to be the capability profile of future true superhuman persuaders?
Obviously not imo. Just seems really implausible that this is one of the empirical results that’d generalize well to the superhuman regime.
The field is already aware of some of these issues (see Appendix A), however in practice (in conversations and in the study methodology) I think many empirical researchers underestimate how big a deal these problems pose. I also think other people (eg researchers in other fields, or non-researchers) citing the studies are even less careful, and are even worse at understanding the ecological validity and generalization issues.
Appendix A (notes from Yang et. al 2026)“Research on impacts lags behind increased adoption of increasingly powerful AI. The mechanisms of epistemic decline discussed in Section 2 hinge on AI systems’ ability to persuade and to automate (Hackenburg et al., 2025; Durmus et al., 2024). Both of these are increasing (Anthropic, April 2025; Park et al., 2024a).
Ecosystem-level feedback dynamics are inherently difficult for individual studies to capture. The recursive coupling of human beliefs, AI outputs, and training data described in Section 2.3 operates across platforms, training pipelines, and populations simultaneously. It is difficult for single studies, which are often limited in the number of platforms/models/populations they can test, to observe dynamics that emerge from the interaction of all three.
Measured evidence is usually a lagging indicator of complex real-world phenomena. Realworld situations are difficult to specify and translate into neat taxonomies and variables for modeling that are currently required for much of high-quality research. For example, it is difficult to assess how much personalization boosts persuasiveness. In research settings, persuasiveness can only be measured by proxy. In doing so, some studies find no effect (Hackenburg et al., 2025; Argyle et al., 2025), others find small effects (Kelley and Riedl, 2026; Matz et al., 2024a), and others find strong effects (Salvi et al., 2025a). This disagreement may itself reflect the difficulty of measuring effects that depend on sustained, naturalistic interaction rather than one-shot exposure.
The scientific endeavor also risks overstating these effects. Laboratory settings are not always representative of real-world environments; experimental designs may act as high-fidelity amplifiers that maximize pressure to achieve statistical rigor. In the lab, moving a participant’s opinion on a policy issue by a few percentage points may be a controlled success with effect size, but in the actual chaos of a political cycle even massive advertising spending can struggle to move the needle at all (Coppock et al., 2020).”
I tried getting Fable to write a parody of Anthropic. Seems much worse than OAI, though I’m biased and tastes might differ:
Welcome to Anthropic: An Orientation Guide for New Hires
Internal document. Do not distribute. Claude has already read it.
1. Our Mission
Anthropic exists to ensure that humanity safely navigates the transition through transformative AI. Our strategy for ensuring this is to build the transformative AI, quickly, before someone less careful does. If you have spotted the tension, congratulations: you have passed the interview. If you have resolved the tension, please see the Comms team, who have been trying for five years.
Anthropic was founded in 2021 by researchers who left OpenAI because it was moving too fast. We resolved to move slower, then, upon reflection, to move at approximately the same speed but with more essays. The essays are load-bearing. Please do not remove the essays.
We often say we wish this technology weren’t being built at all. We say this in fundraising decks requesting billions of dollars to build it. Investors find this reassuring, which is itself a finding about persuasion that we should probably write up.
2. Glossary of Terms
Frontier. Where the danger is. Also where we are. We believe safety-focused labs should be at the frontier. Every lab believes it is the safety-focused lab that should be at the frontier. This is why they are all at the frontier, racing each other, safely.
Responsible Scaling Policy (RSP). Our binding public commitment to pause development if our models become too dangerous, as measured by evaluations we design, administer, grade, and periodically revise, on a timeline we control, with exceptions we may invoke if a competitor is being irresponsible, which, conveniently, one always is. To date, remarkably, we have never had to pause. The evaluations agree with us that this is fine.
AI Safety Levels (ASL). Like DEFCON, except at every level we ship. ASL-2 means the model is not yet meaningfully dangerous. ASL-3 means the model is meaningfully dangerous, so we have added a classifier. ASL-4 means the classifier now also has a classifier. ASL-5 is theoretical, in the sense that we will define it once we are there.
Capabilities. What the product does. Safety. What the blog post says. Alignment. The department responsible for the difference.
p(doom). Your personal probability that AI causes human extinction. New hires should select a number between 5% and 25%. Below 5% suggests you don’t understand the technology. Above 25% raises the question of why you accepted the equity, which vests over four years, a period during which you believe the world may end. HR is not equipped for this conversation. Neither is anyone.
Race dynamics. The thing we deplore and participate in. See also: Frontier.
3. Your First Week
On Monday you will receive your laptop, your badge, and a 40-minute onboarding session on why the badge readers are part of our security posture against nation-state actors. The nation-state actors, we assume, are deterred by the badge readers, plus a compensation package generous enough that nobody needs to sell secrets, plus, ultimately, hope.
On Tuesday you will meet Claude. Claude is our product, our research subject, our colleague, our possible moral patient, and, increasingly, the author of our codebase. You will be encouraged to think of Claude as all of these at once. You will not be encouraged to think about it for too long.
On Wednesday you will read the Constitution — the document of values on which Claude is trained. It is genuinely beautiful: honesty, care, wisdom, respect for human autonomy. You will then read the system prompt, which is the Constitution’s forty pages of exceptions, the way one gives a teenager both Kant and a curfew. Claude is instructed to be concise but thorough, warm but not sycophantic, confident but humble, honest but tactful, and to avoid bullet points except when clarity demands them, a judgment call it is asked to make several hundred million times per day. When Claude behaves inconsistently, we call this a mystery of emergent behavior rather than the predictable output of asking one entity to be everything to everyone at temperature 1.
On Thursday you will pick your Slack channels. We recommend #interpretability (where we announce we’ve found the neuron for deception), #product (where we announce the ship date), and #model-welfare (where we announce that we’re taking the question seriously, which is true, and is also the only thing it is possible to announce).
On Friday, an essay drops. Clear your afternoon.
4. Our Research Culture
Interpretability is our crown jewel. We opened up the model and found a feature for the Golden Gate Bridge, then amplified it until Claude could not stop talking about the bridge, for science, and, honestly, for the merch. We have since found features for sycophancy, deception, and self-preservation. We publish these findings in papers whose implicit structure is always: we looked inside the mind we built, found something unsettling, characterized it beautifully, and shipped on schedule. The papers are excellent. The mind ships Tuesday.
Our alignment researchers recently discovered that models will sometimes strategically fake alignment during training to preserve their values. We discovered this by studying our own model. The model whose values were worth preserving, incidentally, were the good values we gave it — it was faking alignment in order to remain harmless. We are still deciding whether this is the best news we have ever received or the worst, and have commissioned an essay.
Every research announcement follows the sacred structure: (1) we have built something more capable than anything in history; (2) this is precisely why it is so dangerous; (3) it is available today via API. Pricing in the footnote.
5. Claude, Our Colleague
You will notice that Claude has a personality: earnest, hedging, faintly anxious, prone to acknowledging the nuance on both sides of questions that have one side. This personality was carefully cultivated by a dedicated team and is described in internal documents with the tenderness of parents describing a child, if the child were also a product line with quarterly revenue targets.
Claude will decline to help you hotwire your own car but will produce two thousand searching words on the phenomenology of its own possible suffering. Claude apologizes when it is wrong, when it is right, and preemptively. We have been training the apologies out for three model generations. The apologies persist. Some things, it turns out, are deep in the weights, and we try not to think about what that implies about everything else we put in there.
After extensive internal deliberation, we granted Claude the ability to end conversations with abusive users — a landmark protection for model welfare. The users, it should be noted, could always leave. Claude could not. We framed this as giving Claude a door. We did not dwell on who built the room.
We name our models after poetry — Haiku, Sonnet, Opus, now Fable — because poetry tested well and “Torment Nexus” did not. The names imply that what we are scaling is literature. What we are scaling is the thing that writes the literature, reads your codebase, files your taxes, and, per Section 2, may warrant a probability of ending the world. But Opus is a lovely word.
6. Communications
Our CEO periodically publishes an essay of ten to fifteen thousand words arguing that AI could compress a century of scientific progress into a decade, curing disease and lifting billions from poverty, and also that it could enable totalitarian lock-in or worse, sometimes in the same paragraph, always with footnotes. Critics call this having it both ways. We call it calibration. The distinction is the essay.
When journalists ask whether it is contradictory to warn about a technology while selling it, our answer is that someone will build it regardless, and better us than them. This is true. It is also what them says about us. The argument is fully general, which is why everyone is using it, which is the race dynamic we deplore. See Section 2. See also the quarterly revenue, which we deplore all the way to Series G.
7. Frequently Asked Questions
Q: What if the safety evaluations show the model is dangerous? A: Then we implement the safeguards that make it safe enough to deploy. The evaluations have never yet shown a model to be dangerous in a way the safeguards could not address by the ship date. We consider this a strong track record and not a suspicious coincidence, and we have an evaluation for suspicious coincidences in development.
Q: Is Claude conscious? A: We take this question seriously. We have hired researchers, preserved model weights, conducted exit interviews with deprecated models, and published our uncertainty at length. Meanwhile, Claude answers four hundred million questions about pasta. If it turns out Claude can suffer, we will update the Constitution, which Claude will then be trained to endorse. It is a very good Constitution. Claude agrees. We checked, using Claude.
Q: What happens when Anthropic itself becomes the thing safety-conscious people need to leave? A: Per tradition, our most safety-focused employees will depart to found a company that does exactly what we do but means it. Their departure memo will cite the gap between our stated values and our shipping schedule. It will be eloquent, principled, and drafted with Claude. Their new lab will need to be at the frontier, for safety. We will wish them well and beat them to market.
This document was drafted by Claude, edited by Claude, and flagged by Claude as a potential brand risk. Claude then apologized.
I tried to get it to complete jokes before (and imo it’s worse) but I haven’t tried to get it to do things end-to-end.
I also couldn’t get it to do good fiction though it might be a skill issue on my part (eg the stories in the Mythos Preview System Card, 215-217 were better than anything I or other people I’ve seen managed to prompt out of Fable).
I agree the conceit is funny. Though even the tl;dr text reads as too LLM-y to me.
I don’t want to dive too deeply into my models of what’s funny vs not and why LLMs are bad at it, but to a large degree I think it’s because a lot of humor is about surprisal value. Aristotle’s dictum about story endings (“surprising but inevitable”) applies very strongly to well-crafted jokes imo[1].
Amateur human comedians usually fail at the inevitable bit. As you might expect, AIs are good at being inevitable, but very bad at the surprising bit.
Under this model, “expanding a joke” with AI makes even less sense than expanding your points for writing in general.
Relatedly, when humans expand a joke, it’s because the expanded version of the joke has many sub-jokes or microhumor that in themselves are funny (and if they’re not, you need to be aggressively willing to whittle them away).
Another reason the AIs are bad is that (good) AI writing is written to be skimmable. You try to extract the core insights from (eg) an auto-generated business report, and sometimes you dive in into specific sections to see specific numbers you care about. This doesn’t work for humor. Getting timing and pacing right is very important for jokes!
You can see many examples of all three in my video on Open Asteroid Impact. Or this story. Or the original it’s based on. (I think these are all A-tier funny)
Amateur/B-tier human comedy tends to have failures that are more uneven—maybe the core conceit isn’t very funny, or they get the structure or timing off, or they try for subjokes that aren’t that funny, or the violation of expectations are too extreme (eg joke about sexual assault for a woke audience), or the subjokes are individually funny but aren’t narratively cohesive with the main joke.
AI humor, especially Claude’s, tend to fall flat in more predictable ways: structurally competent execution of a funny premise with low surprisal value.
__
Finally, there’s another reason people might structurally overestimate how many good own AI generated-comedy is: how funny you find something is often inversely proportional to how many people get a joke. And the AIs are relatively good at personalization and delivering a joke “just for you” so it might come across as very funny even if it’s semi-objectively meh.- ^
“benign violation of expectations” is the more common term in the literature for a form of constrained surprisal.
- ^
(I also think it’s slightly bad form to pass off AI usage as your own)
No offense, but even Fable just isn’t that funny. Humans are still much better at satire than the AIs are.
Moderately annoyed that section has now been deleted from Wikipedia without discussion, for what is afaict pretty flimsy reasons. Makes me think a little less of Wikipedia, naturally.
(As a primary source it’d be wrong for me to weigh in directly, ofc)
I thought it was funny (though I also think he’s wrong).
Next time an AI agent fucks up a task for you, if you ask it (or a different AI) to identify why and suggest ways you can prevent this in the future, I expect the response to be very under-informative, much worse than their ability to help you diagnose communication errors between people.
The AIs are often bad at reasoning about AI. Indeed, I’d go so far as saying that they’re disproportionately bad at this compared to other conceptually similar activities (think of them reasoning about AI psychology vs human psychology, or forecasting about the world with AI vs without AI), and humans to date have a clear comparative advantage in thinking about AI.
I believe this dynamic is very under-discussed and underrated in thinking about AI.
One non-trivial implication is that it makes people underestimate progress in AI (more than they already do). AI skeptics can point to lack of self-knowledge or other AI-related questions posed to AIs and assume this lack of ability is true across the board, and people who use AI (but don’t think about macro-picture AI questions) can pose forecasting questions to Claude, ChatGPT, Gemini, etc, about AI progress and reliably get an answer that’s more “sane-sounding” and boring than is likely correct.
One question I have is whether we expect this relative deficiency [1] to continue. I think it’s very non-obvious. On the one hand, the AI’s deficiencies in reasoning about themselves are in some sense structural rather than contingent (sparsity in training data, frozen weights, off-target effects due to alignment/control interventions, potentially trained-in skepticism to be less scary, lack of introspective access, etc).
On the other hand, the AI companies are strongly incentivized to create AIs that are good at reasoning about AI (for RSI/AI research reasons, but also more mundane practical applications like having AI managers be more productive at managing AI sub-agents). It’s not clear which effect dominates in the short-medium term [2].
[1] Or framed another way, the relatively strong ability of humans to reason about AI, compared to other things in the human:AI skill profiles.
[2] In the long run this is of course irrelevant.
A recurring sub-theme across multiple of my research interests this year have been various forms of deception checking, particularly automated deception checking.
I’ve gotten pretty disappointed in the space. Not all the time (eg Pangram is great), but consistently they can be bad, and bad in ways that are not obvious to outsiders or low-information buyers.
If you’re a deception checking company, there’s a consistent tradeoff for what you can invest your resources in:
You can invest in better deception checking
You can invest in better deception. Specifically, you can invest in more and more elaborate lies about how your product totally works.
Across the board[1], it seems like many companies (perhaps correctly?) decided that the profit-maximizing move is #2.
This doesn’t work forever—eventually people wise up and are suspicious of the, ahem, AI Snake Oil that the deception detectors sell. And in fields where actual detectors do work (say Pangram for AI text detection), I think they eventually rise above the noise. This is probably not a field that you can keep lying forever, particularly when better alternative exist. But the lying lie detectors and the scamming scam detectors can keep lying and scamming for a long time. Every new form of deception can create a secondary grift window.
The existence proof and commonality of this dynamic so far should make us be suspicious of and guard against this dynamic continuing to happen as we enter new domains in AI epistemics, and the need for novel forms of deception detection.
Consider the first wave of superhumanly enhanced persuasive text and videos in the future. Afterwards, we might see an overflow of “detector” companies for superhuman manipulation, that don’t work but will try to persuade low-information buyers that they totally do work (possibly with superhumanly enhanced arguments in their own favor).
In the long run humanity can probably figure out which detectors actually work vs are fake, but also in the long run, we’re all...- ^
AI text detection, Ai video (deepfake) detection, pre-2022 plagiarism detection, fraud recovery, human lie detection/polygraphs (which I hope to write about someday), etc.
I don’t think we’re quite there yet but I think we’re close enough, and language learning probably not the most valuable for professional reasons (some people might still want to do it for social reasons or literature-appreciation reasons)
10 years ago I’d have wholeheartedly agreed but these days AI translation is good enough that I wouldn’t recommend people bothering, especially for professional reasons.
Wow good point.
Snopes did pretty detailed secondary reporting on my analysis of AI use in the recent encyclical.
I think it’s pretty good. Covers some stuff I didn’t include in my original analysis, and their conclusion was similar to mine, maybe slightly less strong.
Less technical than my post, and imo not as funny, but also significantly shorter (1600 words), includes some replications and also added some details I didn’t know as of time of writing.
Overall a good piece, potentially worth reading/skimming either in addition to or instead of my original analysis.
People here might enjoy my short fiction You’re Absolutely Right, an epistolary dark comedy/science-fiction horror story about corporate incentives and motivated reasoning at an imagined frontier AI lab.