Frontier AI Now Gets a 30-Day Government Review. The Test Is Classified. | Edition 314
Edition 314 — The framework for testing whether frontier AI can find software vulnerabilities is done. The criteria are classified.
On June 2, an executive order gave the federal government 60 days to build a process for testing whether frontier AI models can find and exploit software vulnerabilities.
The deadline was August 1. The framework was finished. On August 4 the White House walked a room full of AI companies through it — Meta, Nvidia, Microsoft, OpenAI, Anthropic, and a number of smaller firms.
Then it declined to publish it.
The criteria are classified. The labs being tested know what they are being measured against. Regulators, security researchers, and the businesses deploying these models do not, and there is currently no plan to tell them.
What the order actually requires
The text is public even though the framework built from it is not, and it is worth reading plainly rather than through the coverage.
Executive Order 14409, signed June 2, directs officials to develop and maintain a classified benchmarking process to assess the advanced cyber capabilities of AI models. Classified is not an interpretation of the order. It is the order.
Alongside it sits the voluntary side: developers would give the federal government access to covered frontier models for a period of up to 30 days before they plan to release such models to other trusted partners, subject to confidentiality, cybersecurity, insider-risk, and intellectual property protections.
And then the sentence that defines the whole arrangement:
Nothing in this section shall be construed to authorize the creation of a mandatory governmental licensing, preclearance, or permitting requirement for the development, publication, release, or distribution of new AI models, including frontier models.
That is not boilerplate. It is the load-bearing beam. There is no permit, no approval, no gate a model must clear before shipping. A lab that does not want a 30-day federal review can decline and release anyway.
The order also stands up an AI cybersecurity clearinghouse under the Treasury Department, coordinating with the National Cyber Director, the NSA, and CISA, to deconflict vulnerability scanning and prioritize remediation.
Why a lab would say yes to something optional
A voluntary program with no penalty for refusing sounds like a program nobody joins. The reason the labs are in the room is ten weeks old.
On June 12, Anthropic received a government letter at 5:21pm Eastern. It was an export-control directive suspending all access to its Fable 5 and Mythos 5 models by any foreign national — anywhere in the world, inside or outside the United States, including Anthropic's own foreign-national employees.
There was no practical way to comply selectively. The company disabled both models for every customer on earth.
By Anthropic's account, the letter cited national security authorities but did not provide specific details of its national security concern. The dispute later surfaced as a disagreement over a jailbreak technique: the administration considered it severe, Anthropic said the demonstration it reviewed surfaced a small number of previously known, minor vulnerabilities, and objected that a narrow potential jailbreak should not be cause for recalling a model deployed to hundreds of millions of people.
The restrictions were lifted around July 1, in a letter from Commerce Secretary Howard Lutnick citing Anthropic's cooperation.
That is the incentive. Not a fine, not a license — the memory of a flagship model going dark globally, overnight, with no published criteria and no appeals process. Against that, a voluntary 30-day review looks less like regulation and more like insurance.
The category the framework appears to leave out
According to reporting by Politico, citing three people familiar with the discussions, the framework broadly exempts open-weight and open-source models and is expected to primarily affect the most capable closed models from OpenAI, Anthropic, and Google. Only state-of-the-art systems judged to pose national security risks would fall under it.
Treat that as well-sourced but unconfirmed — it comes from anonymous accounts of a classified document, which is the only form this information can currently take.
If it holds, it matters more than the secrecy does. Because the models businesses actually run are increasingly not the frontier closed ones.
On July 27, eight days before that meeting, NVIDIA open-sourced NOOA as its first named technical contribution to a new Open Secure AI Alliance formed with the Linux Foundation. It is an Apache 2.0 Python framework that collapses an AI agent into a single Python class — methods become the actions the model can take, docstrings become prompts, type annotations become contracts the runtime enforces. It is deliberately model-agnostic. You point it at whatever model you like.
That is the shape of the gap. A governance framework aimed at a handful of closed frontier models, and a tooling ecosystem built so that the model underneath is an interchangeable part.
| Question | Public | Classified or unstated |
|---|---|---|
| What is being tested? | Advanced cyber capabilities - whether a model can find and exploit software vulnerabilities | The actual benchmarks, tasks, and scoring method |
| Which models are covered? | Covered frontier models, defined by the order | The threshold itself - shared with developers and researchers only as appropriate |
| How long does review take? | Up to 30 days before release to other trusted partners | What happens if a model fails |
| Is participation required? | No. The order bars any licensing, preclearance, or permitting requirement | What a lab gives up by declining |
| Can outsiders audit it? | No public release is planned | Whether independent researchers will ever see the criteria |
Why this lands on your desk
You are probably not submitting a frontier model for federal review. The relevance is downstream of that, and it is concrete.
When a vendor tells you their model has been evaluated for security risk, you now cannot check what that sentence means. Not because the vendor is hiding something — because the standard itself is classified. The usual move for a diligence question, going and reading the benchmark, is unavailable.
This is a new kind of gap. Enterprises are used to assurance they can inspect: SOC 2 reports, penetration test summaries, published benchmarks, model cards. Those are all still there. Underneath them now sits a government evaluation that is real, is happening, and is unreadable.
The practical consequence is not that you should distrust the models. It is that this particular claim cannot carry weight in a risk assessment, because it cannot be examined. If a vendor cites federal review as a reason you should be comfortable, the honest response is that you have no way to evaluate that, and it should be scored as unverifiable rather than as a pass.
Most teams have never written down which of their AI assurances are verifiable and which are being taken on faith. That distinction just got sharper, and it is worth about ten minutes.
What is genuinely unresolved
Three things are worth holding loosely.
Whether the secrecy is justified. There is a real argument for it: a public list of the exact capabilities the government tests for is also a roadmap for anyone building a model to evade them, and detailed cyber-capability benchmarks are dual-use by construction. There is an equally real argument against, made bluntly by Chris McGuire, Senior Fellow for China and Emerging Technologies at the Council on Foreign Relations, who called the decision baffling and wrote that we can't have secret, voluntary rules to regulate the most important tech in the world. Both positions are serious. Neither has been publicly adjudicated, because the material needed to adjudicate them is the material being withheld.
Whether voluntary holds. The order forbids a licensing regime today. It does not prevent a future order, or a procurement rule, from making participation the practical price of selling to the federal government.
Whether the scope survives contact with reality. If the open-weight exemption is real, the framework covers a shrinking share of deployed AI. Capable open-weight models and model-agnostic agent tooling are both moving fast, and a framework scoped to closed frontier models in June 2026 may be scoped to a minority of the risk by 2028.
The takeaway
A classified vetting framework is better than no framework. The capability it targets — models that can find and exploit software vulnerabilities at scale — is one of the few AI risks that is concrete, measurable, and already partly here.
But it creates something genuinely new: accountability without transparency. The labs know the test. The government knows the test. Everyone whose business depends on the outcome is asked to accept it on trust.
That may be the right trade. It is impossible to say from outside, which is precisely the point. What you can do is stop treating it as assurance. Know which of your AI risk claims you can verify and which you cannot, and never let the second category quietly get filed as the first.