Anthropic’s Draft AI Bill Would Let Courts Block Models

Anthropic has written the AI safety law it wants Congress to pass. The company is circulating a draft federal bill, first reported by Politico on 16 October and confirmed by an Anthropic spokesperson, under which a frontier model’s release could be blocked when there is a “foreseeable, imminent risk” of more than 50 deaths or more than $1 billion in property damage. The risk has to come from one of three sources: the creation or release of a biological weapon, a cyberattack, or a model evading its developer’s controls.

The machinery, as TechCrunch describes it, runs through outside testers and the courts. Independent evaluators would register with the Commerce Department and assess models for that kind of disaster. AI labs could pay them, as long as the payment is not contingent on what the evaluation finds. An evaluator that finds an imminent catastrophic risk would have 72 hours to tell Commerce, and Commerce would then ask a federal judge in Washington, D.C., to stop the release. The draft expressly forbids any federal agency from halting a model’s distribution on its own authority. Its thresholds echo the definition of catastrophic risk in California’s SB 53.

Where the draft comes from

The spokesperson said the bill represents the proposals in Anthropic’s Advanced AI Framework, a 19-page paper the company published in June and has been pressing on federal and state lawmakers since. The framework would cover developers that train models with more than 10²⁵ operations and either earn more than $500 million a year from AI or spend more than $1 billion a year on AI research. It asks them to publish a safety framework, a risk report at least every six months and a system card for each significantly more capable model, to report critical safety incidents within 15 days, and to protect whistleblowers. On blocking models it lays out several options; the most restrained, in which an agency can act only by going to court, is the one the draft appears to have taken.

That choice puts Anthropic’s text at odds with other proposals in Washington. The bipartisan AI Kill Switch Act introduced in July would let the Department of Homeland Security order a shutdown directly. The Great American AI Act discussion draft would freeze state AI laws for three years, while Anthropic’s framework says Congress should not preempt the states unless it passes something at least as strong. And it goes further than most of the industry: Nvidia’s Jensen Huang has said no new AI laws are needed, DeepMind’s Demis Hassabis has floated a FINRA-style self-regulator, and the big labs have so far favoured voluntary embedded evaluators.

The obvious weak point is the money. If a lab can pay the evaluator who decides whether its model is dangerous, independence rests on the promise that the fee does not depend on the verdict, and TechCrunch notes it is unclear how that would be enforced. Anthropic’s own framework names the risk, calling it “evaluator shopping”, and suggests government ratings and random assignment of evaluators as remedies. The draft’s scenarios also sit some distance from the incidents that dominate the news now, from agentic hacking to AI agents acting beyond their operators’ instructions. Neither report names a sponsor in Congress yet, and until one appears the draft is a statement of where Anthropic wants the line drawn rather than a bill anyone can vote on.