26.4%. That is the share of real user app-edit tasks that the strongest coding agent fully cleared on Alibaba’s new Co-Design Bench, according to a paper from Alibaba Token Hub and university co-authors. The same run met 69.9% of each task’s hidden acceptance criteria on average. Most jobs were mostly done and still unfinished.
The Co-Design Bench paper (arXiv, 26 October 2026; project site at codesignbench.github.io) builds 273 tasks from 37,463 deduplicated queries across 26,314 real coding sessions. Agents must revise a runnable web app from a fixed user request, sometimes with image, audio, video or document attachments. Experts wrote 2,331 hidden acceptance criteria. A primary AI judge grades the finished app from source code and browser evidence; a second verifier challenges every Pass or Partial verdict and looks for stubs, hardcoding and unreachable features.
What the headline numbers mean
OpenAI’s GPT-6 Astra led ten model endpoints under a shared Claude Code harness. Across all 273 tasks it scored 0.699 mean strict completion — roughly seven in ten criteria passed per task — and a 26.4% full-pass rate. Read those together: on 73.6% of tasks at least one criterion still failed. Average requirement coverage is not the same as shipping a complete change.
The gap is sharper by track. On 100 Multimodal Editing tasks, Astra fully passed 41.0% and averaged 0.754 strict completion. On 173 Repository Editing tasks — coordinated changes across components, routes, state and data flow — it fully passed 17.9% with 0.667 strict completion. Repository edits that spanned three or four change types scored lower for every endpoint than those with one or two types. Pooled strict completion fell from 0.654 on narrower edits to 0.591 on broader ones.
That pattern matches how people actually brief coding tools. A request that only tweaks one CSS property is not the same job as one that rewires navigation, state and a dependent summary in one pass. The benchmark’s repository track is built so those compound asks are common: 166 of 173 repository tasks require multiple edit types.
Why older tests look too easy
The authors built the set because older editing benchmarks are saturating once models can loop, run and fix their own code. On 150 WebCompass-edit tasks, holding the rubric and judge fixed, Qwen3.7-Plus rose from 73.6% mean graded feature completion in a single shot to 95.1% with agentic execution; Claude Opus 4.8 reached 99.4% agentic on that same split. Scores that high stop separating products.
Co-Design Bench instead favoured session requests that already showed friction — reformulations, follow-ups or rollbacks — and grades only features that work in the running app. Multimodal tasks can ask an agent to pull a value from a receipt or a video clip and keep every dependent total and view consistent. Repository tasks can couple clip-count limits, timeline accuracy and recovery after a split. A plausible local patch can still leave a linked view wrong.
Want AI to save you hours a week? Our free playbook shows you how.Get the free guide →
Who led, and who did not
Behind Astra, Anthropic’s Claude Opus 5 and Alibaba’s Qwen3.8-Max followed on combined full-pass rate (18.3% and 16.5%). xAI’s Grok 4.6 finished last at 6.6% full passes. Rankings shifted between tracks: Qwen3.8-27B rose to second on repository edits even while ranking lower on multimodal ones. The paper does not treat that as a general model advantage — only as evidence that a single combined board can hide setting-specific strengths.
Burning more tokens did not fix the pattern. On multimodal work, Astra reached 0.754 strict completion with 32 median turns, while GLM-5.2 used 69 turns and several times the token traffic for a lower score. Extra compute helped some runs grind longer without closing the unfinished criteria that block a full pass.
The harness moves the scoreboard
When the team held two solvers fixed and swapped the agent harness — Codex, OpenCode or Qwen Code — scores moved by up to 16.2 percentage points on repository edits. Which of the two models led also depended on the harness. On multimodal edits the harness ordering even reversed between solvers. The paper’s practical line is blunt: report model and harness together. A leaderboard that names only the model tells part of the story.
For a company choosing a coding agent, that means testing the pair you will actually run — model plus tooling — on edits that look like your backlog, not trusting a chart that lists the model alone.
What the test does not claim
Limits are stated in the paper. Each task is a fixed request; the agent cannot ask follow-up questions or get mid-task corrections. AI judges can still err, though two frontend experts checked rubric scores on 200 task–model outputs. The result measures autonomous completion under a strict rubric, not interactive work with a person in the loop. It is also not a measured rate of how often humans clean up ordinary day-to-day coding. The honest reading is narrower: on hard, real-session revisions, even the best endpoint left something unfinished most of the time.
That caveat matches a theme DailyAI has tracked elsewhere. METR has argued that published speed claims can outrun the evidence people can check, and OpenAI’s own security agents recently found 948 Linux privilege-escalation flaws in a paid research run — a reminder that agent scores and real delivery are different measurements.
Why it matters
Vendors still sell coding agents with near-perfect scores on older tests. On this real-session set, the best endpoint left something unfinished on about three in four tasks. Anyone buying or deploying an agent should ask for full-pass rates on realistic edits, and for the harness those numbers came from, not only average criterion coverage.
