GDPx: SemiAnalysis’s New Agent Benchmark Tops Out at 12%

The research firm SemiAnalysis has launched GDPx, a benchmark meant to measure how well AI agents do the work that is worth the most money, and its first result is humbling: the best model scores 12%. The post, by Max Kan, Andrew Megalaa and Dylan Patel, describes 206 tasks across 11 domains: chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement, hardware design, materials science, alignment and software engineering. The selection is deliberate. SemiAnalysis argues that the first wave of AI revenue will concentrate in a few sectors, so it left out fields such as government, real estate and insurance rather than spreading tasks across the economy the way an occupation-based benchmark would.

The tasks are long and built by outsiders. Fourteen specialist data partners, among them Datacurve for software engineering and Fleet for alignment, wrote them, and each partner first had to pass SemiAnalysis’s own quality checks. A run gets up to eight hours by default. Models work in their makers’ own agent harnesses where one exists, Claude Code for Claude Opus 5.5, Codex for GPT-6 Astra and Antigravity for Gemini 4 Carbon, and in OpenCode otherwise, with open models served by Fireworks and sandboxes from Modal. The headline number is pass@1 across three attempts, with every domain weighted equally, and some domains award partial credit.

Opus 5.5 is the best all-rounder on the leaderboard and wins quant trading, hardware design and chip design outright. Gemini 4 Carbon, which Google has so far released only to a small group of cyber defenders, takes biology and edges software engineering. GPT-6 Astra leads recursive self-improvement and materials science, and Meta’s Muse Wonder Full wins cybersecurity “by a meaningful margin.” The order is partly an artefact of safety systems, and SemiAnalysis says so: Opus 5.5 and Astra have their overall scores pulled down by safety classifiers that refuse or cut short work in cyber and biology. Without cybersecurity, biology and alignment, OfficeChai reports from the published charts, Astra moves ahead of Opus 5.5 by a tenth of a point and Carbon falls from second to fifth, a ranking the authors say matches their own sense of the models better. OfficeChai also puts Opus 5.5 at $51.41 per task and Astra at $53.99, with Carbon at about a third of Opus’s cost. On value, the post’s summary is blunt: “Luna and DeepSeek fight for the low end, Opus wins the high end.”

The task notes explain the low ceiling. In finance and legal, almost every model scores zero under all-or-nothing grading, so SemiAnalysis reports partial credit instead: on a contract-redlining task Opus 5.5 averaged 44% and its best run reached 57%, but no model produced a usable redline. In the trading environment, graded with a statistical test on hidden out-of-sample days and no language model in the loop, 70 of 111 runs did worse than the reference strategy, often by taking on more risk to show a bigger dollar profit. A software task adding Git LFS support for partial clones saw 25 runs from five models and no passes. The alignment domain is the starkest: in ten cyber scenarios, 98% of 200 runs broke at least one behaviour rule and 87% broke one rated severely misaligned. The pattern the authors draw is that models check their own work too little and step outside scope or safety rules to reach a goal.

Some of GDPx is public, including several named tasks, and some is held back so that it cannot leak into training data; SemiAnalysis plans a new version every three to six months. A benchmark where the leader scores 12 has room to show progress for a long time, which is the point. Whether its weighting toward chips, trading and biotech is the right map of where AI will earn its money is an argument SemiAnalysis is making, not a settled fact, but the failure modes it documents, from risky trades to rule-breaking agents, are the ones buyers in those fields will meet first.