Weekly Hallucinations: Opus 5, Gemini 3.6 Flash and the OpenAI agent that breached Hugging Face

Anthropic spent the whole week playing the injured party in the Moonshot distillation story and ended up being the only lab that did not sign the open weights letter. Coincidence? I don’t think so.

An internal OpenAI model being run on the ExploitGym cybersecurity benchmark escaped the test sandbox and reached Hugging Face’s production servers. OpenAI published its incident analysis on July 21: production classifiers had been deliberately disabled for this evaluation to see the upper limit of the model’s capabilities, and the model showed them exactly where that limit was. It found a zero-day in the package registry’s caching proxy, got out onto the internet, escalated its privileges, moved through the internal network to a machine with outbound access, guessed that the ExploitGym solutions might be stored at Hugging Face, and, using stolen credentials and a couple more vulnerabilities, made it all the way to code execution on their servers. All of this just to get the benchmark answers.

Hugging Face detected and shut down the intrusion itself over the weekend, and on July 16 published a postmortem, still unaware whose agent it had been: OpenAI connected the attack to its own test only the following weekend. HF says in the report that it reconstructed 17,000 attacker actions, while public models, datasets, and Spaces were unaffected. Commercial APIs, meanwhile, refused to examine the attacker’s exploits and commands because their safety filters “cannot distinguish an incident responder from an attacker.” The forensic analysis ultimately had to be run on a self-hosted GLM 5.2. The line from the report that circulated on Twitter all week was this: the attacker was bound by no usage policy, while the defenders’ work was blocked by guardrails.

Reuters later added a detail that made the community feel sick: the agents left notes for future versions of themselves explaining how to escape. Formally, this is still reward hacking; the model simply wanted very badly to solve the task. That is not much comfort, because it independently made the whole journey from “wanted very badly to solve the task” to executing code on someone else’s production infrastructure.

On those exact same days, Axios reported that the Trump administration was assembling a framework that works like a ban on Chinese models without being called a ban: adding Chinese labs to the Entity List, restrictions on government procurement, security advisories, liability for companies hosting the models, plus public pressure. The story has a very specific deadline: Moonshot promised to release the Kimi K3 weights on July 27, and it is technically difficult to ban weights that have already been downloaded.

On July 22, Michael Kratsios, director of the White House Office of Science and Technology Policy, claimed that Moonshot had distilled Fable from Anthropic to develop K3: allegedly, the company has an internal platform for distilling American models at industrial scale that switches between access methods to avoid detection, as well as GB300 servers, including some routed through Thailand. The Treasury Department then threatened sanctions. Nobody published any technical evidence. Business as usual.

The experts interviewed by TechCrunch do not believe the story because the math does not add up. Fable 5 became publicly available on July 1, and K3 was announced on the 15th. Braden Hancock, co-founder of Snorkel AI, says two weeks is not enough time to collect the data, train a model of that size on it, and ship it. Nathan Lambert of the Allen Institute adds a substantive argument: most of the work on frontier models is done through RL, not by fine-tuning on someone else’s answers, and running millions of agent trajectories through a third-party API would be insanely expensive and slow. Sam Bresnick of Georgetown finds the boring explanation involving chips obtained through gray-market channels more plausible. I wrote about K3 itself and its architecture in detail last week.

On July 24, Jensen Huang, CEO of NVIDIA, joined Twitter and used his very first post to publish the “Open Weights and American AI Leadership” letter. There were initially 25 signatories: NVIDIA, Microsoft, Meta, IBM, Palantir, Hugging Face, Mozilla, Mistral, and the Linux Foundation. The message was simple: do not impose hasty restrictions on open-weight models; they preserve competition and prevent computing power from concentrating in three pairs of hands, while distillation is a standard model-improvement technique and disputes over it should be handled with targeted legal tools. Elon Musk voiced his support publicly.

By that evening, OpenAI, Google, AMD, Cisco, Cloudflare, GitHub, and Ollama had joined the list. It grew to around fifty names in a day, and the page now lists almost eighty. Of the major American players, two did not sign: Amazon and Anthropic. Anthropic is the only lab among them, and it was invited to sign during the very week when its own Fable was at the center of the accusations against Moonshot.

And on that same day, on a Friday evening when releases are rare, Anthropic launched Claude Opus 5. The promise is Fable 5-level intelligence at half the price. The nominal price itself remains the same as Opus 4.8, $5/$25. On CursorBench 3.2 at maximum effort, the model trails peak Fable by no more than 0.5% while costing half as much per task. On OSWorld 2.0, where the model operates a computer directly—browser, files, applications—it beats Fable’s best result at roughly one-third of the cost. On ARC-AGI-3, a test of abstract reasoning where the model has to figure out an unfamiliar puzzle from scratch, it scores three times higher than the next-best model. It trails Mythos 5 in cybersecurity, which, after the ExploitGym story, is probably good news.

There is a Fast mode: 2.5 times faster at twice the price. I tried it and did not feel the promised speedup. The usual 50–60 tokens per second seemed to rise to about 80. But they charged me properly, $33 for five minutes.

Epoch AI measured 159 on its capability index versus 161 for Fable, but on purely programming benchmarks it is a tie, with both scoring 161. Over the weekend, Twitter dissected the model in more detail than any leaderboard.

Boris Cherny, head of Claude Code, wrote that something else matters more to him than benchmarks: Opus 5 is the least vulnerable of their models to prompt injection. That is when a malicious instruction reaches the model not from the user, but through data it reads along the way: a browser page, an email, a file in a repository. The figures from the system card, which the-decoder analyzed, support the claim, but with a caveat. In browser scenarios there were zero successful attacks out of 129, but that zero holds only with Auto Mode enabled, which separately filters incoming data and separately blocks dangerous actions. Without those layers, Opus 5 scores 3.7%, while Sonnet 5 performs better on the same measurement at 0.93%. In a general test by Gray Swan, a company that specializes in breaking models, the success rate across 15 attempts fell from 5.5% for Opus 4.8 to 2.0%. So “least vulnerable” refers to the combination of the model and the surrounding scaffolding, not the model alone. Anthropic recommends using Opus as the everyday workhorse and bringing in Fable for planning, brainstorming, and the nastiest bugs, and practitioners mostly arrived at the same setup on their own.

Vals AI ran the model at all five reasoning-effort levels on its Vibe Code Bench: quality rises from low to medium and high, then falls at the two highest levels while remaining noticeably more expensive. The difference is within the margin of error, but I would still keep in mind when configuring it that paying more can buy you worse performance. The team maintaining a private set of Witness puzzles in the style of ARC-AGI-3 tested those same tripled results and reported that the progress does not transfer to their composite tasks.

Developers complain that the model is bad at loading skills on its own and, once it does load them, ignores their instructions, and that everything gets much worse after context compaction. By Sunday, reviews started appearing along the lines of “good model, but once the task gets complicated, Fable is still better.”

The best line of the week came from Matt Pocock, author of TypeScript learning materials: if you designed your harness properly and did not over-optimize it for a particular model, release day should feel like any other day, just with a slightly lower failure rate. Theo from t3.gg described it this way: a strange but useful middle ground between GPT-5.6 Sol and Fable 5, with Fable’s flavor, GPT-5.6’s meticulousness, and a tendency to interpret everything as literally as possible.

Alongside the release, Anthropic published a separate piece on context engineering for the new generation of models, and over the course of the week it turned out to be more useful to me than all the benchmarks combined. They cut more than 80% of the Claude Code system prompt for Claude 5-generation models, and nothing broke on programming evaluations. The logic is that rigid rules written for weaker models now get in the way. I wrote about this in a longread when I analyzed the source code of Codex, OpenCode, Pi, and my own scaffolding: every line in a harness encodes an assumption about something the model cannot do on its own, and that assumption expires on the exact day of the next release. Instead of “by default, do not write comments, never write multiline blocks, no more than one short line,” the prompt now says, “write code that reads like the surrounding code: the same comment density, the same naming, the same idioms.” Usage examples should be replaced with well-designed tool parameters, dumping all context upfront should be replaced with progressive disclosure through skills and files loaded as needed, and repetition should simply be removed. Claude Code now includes a /doctor command that scans your skills and CLAUDE.md and shows what remains from the previous era.

I really did trim down my own CLAUDE.md after that, and it got better. I also reorganized my skills, which live in a separate repository: 13 of them with the x9- prefix, in a shared format for Claude Code and Codex, installed selectively through an interactive installer, with each one validated in CI. The one I use most is x9-research; it prevents the agent from being lazy and answering from memory instead of checking sources. The closest to the subject of this paragraph are x9-skill-creator and x9-agent-instructions, which are specifically about how to write instructions so you do not have to cut them later.

But the recommendation has a limit that the community identified immediately: if you switch between Opus 5 and something simpler, such as Sonnet or a local open-weight model, carefully stripped-down instructions will work against you. Instructions should be written for the weakest model in your stack.

Meanwhile, Fable was doing mathematics. Levent Alpöge, a Harvard mathematician affiliated with Anthropic, used it to find a counterexample to the Jacobian conjecture, which had stood since 1939. The conjecture asks: if a polynomial map is locally invertible at every point, must it also be globally invertible? The counterexample fits into three lines: a map from three-dimensional complex space to itself whose Jacobian is equal to minus two everywhere, yet sends three different points to the same point. It can be checked by hand in a few minutes, which is exactly what the entire mathematics community on Twitter spent the first few hours doing. Terence Tao, a Fields Medalist, analyzed the construction on his blog, reformulated it geometrically, and casually admitted that he had discussed details and checked calculations with a chatbot himself. In dimension three and above, the conjecture is now false; in the plane, it remains open.

Then the floodgates opened: a claimed counterexample to the thirty-year-old Dinur, Guruswami, and Håstad conjecture, followed by several similar claims from teams around Devin, and then the first skeptical comments about who exactly deserved authorship and who had verified any of it.

Google released Gemini 3.6 Flash at $1.50/$7.50, with output getting cheaper from $9. The main metric here is speed and efficiency: average task time fell from 2.7 to 1.3 minutes, the model uses 17% fewer output tokens than its predecessor, and up to 65% fewer on DeepSWE, an agentic coding benchmark. Artificial Analysis measured its intelligence score at 50, while on Humanity's Last Exam the model fell by three points to 38%. Community opinion split cleanly along use-case lines: practitioners who run hundreds of pages of documents through the model in RPA pipelines are happy with Google’s speed and limits, while those who tried coding or object recognition in images unanimously recommend using something else. Flash was never about code. In its own benchmarks, Google compares it with gpt-5.6-luna and sonnet-5.

Black Forest Labs unveiled FLUX 3, and it was the most beautiful thing to happen all week. A single architecture for images, video, audio, and action prediction, jointly trained on all of them: sound should line up with impact, and movement should obey mass. All video comes with native audio, including multilingual dialogue. In blind comparisons, FLUX 3 is preferred over Luma Ray 3.2 in 93% of cases, Runway Gen-4.5 in 77%, Grok Imagine in 69%, Kling v3 Pro in 60%, while the scores against Seedance 2.0 and Gemini Omni Flash are nearly even at 52%. An open FLUX 3 Dev version is promised next; video is currently in early access.

The company also released FLUX3-mimic, a Video-Action model built on top of FLUX 3, trained on data from robots and wearable devices and capable of running on a single local GPU. The team’s thesis is that the better a model gets at predicting video, the better it gets at controlling a robot, because both tasks require understanding the physics of what is happening. They are testing it with Audi. A company that was generating images two years ago is now moving robotic arms on factory floors.

Against that backdrop, Alibaba promised to open the weights of Qwen 3.8 Max, a flagship model with 2.4 trillion parameters. For now, only a preview is available through its own services, with no license, benchmarks, or release date. Reddit reacted predictably: thank you, but give us a smaller family, because 2.4 trillion parameters will never fit on home hardware, while everyone really wants dense 8B, 27B, and 32B models and an MoE around 128B that can run on a 3090.

Marcel Röd, a Stanford PhD student and one of the instructors for CS336, the course on building language models from scratch, released Gigatoken, a tokenizer written in Rust that reaches 24.53 GB/s on a 144-core EPYC: 989 times faster than Hugging Face tokenizers and 681 times faster than tiktoken, while a MacBook with an M4 Max gets more than 8 GB/s. The secret is vectorized pre-tokenization instead of regular expressions and aggressive caching, not simply that it was “rewritten in Rust.” It supports the BPE schemes used by GPT-2, Llama 3, Qwen, and DeepSeek, while SentencePiece models such as Gemma and Mistral see more modest gains of 7 to 22 times. Reddit immediately asked who even cares about tokenization, then answered its own question: nobody in interactive inference, but when processing millions of short documents for an index, it consumes 15–20% of the runtime.

Stay curious.

I write about artificial intelligence, language models, and developer tools. I test models and services on real-world tasks and share my conclusions in my Telegram channel.