Copying the Brakes: The Open-Weights Fight Is About the Wrong Variable

27 Jul, 2026

On July 24, two documents about open-weight AI models appeared within hours of each other. One traveled everywhere. The other went almost nowhere, and it is the more interesting of the pair.

The loud one was a letter, "Open Weights and American AI Leadership," shared by NVIDIA's chief executive in what several outlets reported was the first post of his life on X. It argued that downloadable models expand access to the AI economy, sharpen competition, and free organizations from vendors who can revoke access. It went out with 25 signatures. By the next day Forbes counted 50, OpenAI among the names arriving after publication. The final version carries 77. Amazon and Anthropic are absent from every version, and Anthropic absorbed three days of criticism for its silence before its chief executive published a position on July 27.

The quiet document was a joint assessment from the UK AI Security Institute and the US Center for AI Standards and Innovation, testing the cyber capabilities of Kimi K3, the Chinese open-weight model at the center of the argument.

The staged fight

Read the two documents together and the disagreement looks total. The letter treats openness as nearly a civic good, since transparency lets many teams find and fix vulnerabilities while concentration creates single points of failure. Anthropic concedes the opposite pressure: you cannot apply guardrails to a file someone already downloaded, and you cannot withdraw it.

But the fight is largely staged, because the two are not scoring the same variable.

The letter argues along an open-versus-closed axis: diffusion, cost discipline, competition. Anthropic argues along roughly capability multiplied by state backing. Its clearest sentence is that openness is "irrelevant" to the worry that an authoritarian government builds something more powerful than the US and uses it for military advantage or repression. On that framing the most dangerous model is not downloadable at all. It is one trained in secret and handed to a military.

There is also a candid admission most coverage skipped. A ban, the post notes, would shield American AI companies from competition while doing little about misuse, because the actors nobody wants using these models are not legitimate US businesses anyway.

So where is the genuine disagreement? Distillation. The letter warns against conflating legitimate model-development techniques with misappropriation, defending training on another model's outputs as part of a long tradition of building on existing technology. Anthropic wants the industrial-scale version curtailed. Yet elsewhere both converge on nearly identical wording: mandatory pre-release safety testing for sufficiently capable models, open or closed, with smaller ones exempt.

A debate presented as open versus closed is closer to a disagreement about copying, wrapped around an agreement about testing.

The test nobody was waiting for

Which makes the quiet document awkward. Testing is what both camps say they want, and a test had just been run.

The evaluators used ExploitBench, a public Carnegie Mellon benchmark covering 41 recent vulnerabilities in the browser engine that powers Chrome, plus a 32-step simulated corporate network intrusion called The Last Ones.

Note the shape rather than the totals. Kimi K3 is the strongest open-weight model here, ahead of the previous leader on the aggregate score. But at the top of the exploitation ladder it does not merely trail, it registers zero. Arbitrary code execution is the outcome that actually hands an attacker control, and K3 achieved it on none of the 41 tasks while the most capable models managed roughly half.

That is inconvenient for the urgency of a ban. It is not a clean result for the other side either: K3's own safeguards did not stop it attempting exploit development, and it did solve the full intrusion once, which is quite enough against a weakly defended network. The evaluators also flag their own uncertainty, since K3's aggregate cyber score rests on a single benchmark and carries a wider confidence interval than the models it is compared against.

Why the gap might exist

Here it gets strange. On July 22, the director of the White House Office of Science and Technology Policy claimed Moonshot AI had built K3 by distilling Anthropic's Fable model, using an internal platform that rotated access methods to avoid detection. Anthropic had earlier reported industrial-scale extraction campaigns by three Chinese labs, totaling over 16 million Claude exchanges through roughly 24,000 fraudulent accounts, with more than 3.4 million attributed to Moonshot.

If so, a mundane consequence follows. A model trained on another model's completions can only learn what that model was willing to produce. Anthropic's classifiers refuse advanced offensive-security requests. A refusal is not a subtle signal in a training corpus, it is the visible output. Copy those outputs at scale and you inherit a ceiling on exactly the capability the refusals were built to suppress.

That predicts the odd shape in the results. General exploit reasoning is available from many sources, so a competent student reaches a respectable aggregate score. The final rung is the part a safety-trained teacher will not demonstrate, so that is where the student reads zero.

The cheaper the copying, the stronger this should be. Distillation in its original sense means matching a teacher's full probability distribution across rejected tokens, which needs deeper access than an API gives. Learning from sampled completions is the accessible version, and a sampled completion is where a refusal appears as itself.

It is tempting to reach for something more exotic, and worth resisting. Anthropic's own work on subliminal learning shows models absorbing traits from data with no semantic link to them, including misalignment, through signals that survive filtering. A student trained on number sequences from a teacher that likes owls comes to like owls. But the version published in Nature is explicit that this holds only when the student shares, or is behaviorally matched to, the teacher's initialization. Across different model families it should not apply. The boring mechanism is the plausible one.

The case that none of this happened

The distillation claim deserves more skepticism than it got. One detailed rebuttal notes the accusation arrived with no access logs, no training-data indicators, and no procurement records, and that a near-identical accusation followed DeepSeek's competitive release in January 2025. The theft claim, on this reading, arrives reliably a few days after a Chinese lab ships something good, and the real subject is who captures the margin on inference.

If the distillation never happened, the inheritance story evaporates and the gap needs another explanation, most obviously less compute and less mature security tooling.

The shape survives either way. Two camps staged a fight about openness. One camp's stated concern was explicitly indifferent to openness. Both asked for testing. A test arrived, undercut the loudest fear, and was largely ignored while the signature count climbed.

One second take-away

When a policy fight has a proxy variable and a real variable, watch which one the evidence attaches to. Here the proxy was open versus closed, the real variables were capability and provenance, and the only measurement anyone produced spoke to the second pair. If you have to choose what to read, choose the evaluation over the letter.

Links: Open Weights and American AI Leadership (NVIDIA) | Our position on open-weights models (Anthropic) | Kimi K3 cyber capabilities assessment (NIST) | Letter doubled to 50 (Forbes) | White House distillation claim (PCMag) | Subliminal learning (Anthropic) | Language models transmit behavioural traits through hidden signals in data (Nature) | Is Kimi K3 really distilled from Fable? (AI Realist) | Anthropic wants tests, not bans (The New Stack) | Defense or Diffusion? (Asia Society)

#ai #geopolitics #open-source #policy #safety