Co-Design Bench Shows Real-World Coding Agents Finish Only 26.4 Percent of Tasks Fully

Benchmarks at the limit of usefulness

A joint team from Alibaba and university partners released Co-Design Bench in November 2026, a stress test aimed at coding agents that try to deliver complete software artifacts—not just correct snippets. The benchmark’s premise is that older leaderboards saturated: models ace HumanEval-style function completion while still failing multi-file refactors, dependency wiring, and product requirements documents that real teams hand to junior engineers.

Co-Design Bench encodes thirty-seven thousand natural-language queries spanning mobile UI mockups, API services, data pipelines, and small full-stack applications. Agents must interpret loosely structured specs, iterate with tool use, and emit repositories that compile, run tests where provided, and satisfy explicit requirement checklists. The headline number stings: full end-to-end completion across the suite sits at 26.4 percent among contemporary agents, a reminder that “almost right” is ubiquitous in generative coding.

Why thirty-seven thousand queries matter

Scale in benchmarks is not vanity; it resists overfitting and reveals variance across domains. Co-Design’s query generators combine human-authored seeds with programmatic mutations—changing frameworks, error-handling clauses, accessibility requirements—to produce families of related tasks. That design makes it harder for vendors to memorize answers via training contamination, though maintainers still scrub against public commit histories before inclusion.

Each task carries layered evaluation: static lint and unit tests when available, requirement tag matching for features described only in prose, and optional human audit queues for ambiguous UI tasks. The multi-signal scoring avoids单一 reliance on execution, which agents sometimes game via hard-coded outputs that pass tests without generalizing.

GPT-6 Astra’s paradoxical score

OpenAI’s GPT-6 Astra variant drew attention for satisfying 69.9 percent of enumerated requirements on average while achieving relatively low full-completion rates. Practitioners read that split as diagnostic: Astra understands specs and produces plausible stubs, configs, and partial modules, but struggles to knit them into shippable repos within interaction budgets.

Frontier labs argue requirement coverage is the harder cognitive step—decomposition, prioritization, edge-case spotting—while integration is an engineering loop solvable with more tool time. Skeptics counter that incomplete repos are failed deliveries in production settings, regardless of checklist percentages. Co-Design Bench makes that debate quantitative rather than anecdotal.

Saturated legacy benchmarks

HumanEval, MBPP, and similarly scoped sets top out near ceiling performance for top-tier models, compressing vendor differentiation into marketing noise. Security-focused extensions and university variants temporarily restored spread, but agents adapted quickly via fine-tuning on synthetically similar problems.

Industry buyers now demand benchmarks mirroring ticket backlogs: “Add SSO and logging to this legacy service,” “Port this Qt desktop tool to Electron,” not “Implement quicksort twice.” Co-Design joins SWE-bench, AppWorld, and enterprise-private eval suites as part of a shift toward repository-scale evaluation—expensive to run, revealing in ways snippet tasks are not.

Alibaba’s stake in measurement

Alibaba Cloud sells coding assistants integrated with domestic IDEs and DevOps suites. Publishing Co-Design signals technical leadership and pressures global rivals on agent completeness, not just autocomplete latency. University collaborators lend methodological credibility—important where Chinese and Western labs cross-cite benchmark papers despite geopolitical friction.

The release includes a subset open to academic use while withholding some proprietary UI assets tied to commercial design systems. That split mirrors how other vendors balance openness with competitive caution.

Agent architectures under the hood

Teams submitting results pair language models with planners, linters, test runners, and retrieval over documentation corpora. Best-performing stacks spend tool calls budget on iterative test-fix cycles rather than single-shot mega-prompts. Co-Design’s logs show agents that pause to read existing repo structure outperform those that overwrite files blindly—a lesson aligned with human practice.

Multi-agent setups—planner, coder, reviewer—sometimes beat monolithic prompts, but coordination overhead rises. The 26.4 percent full completion rate leaves room for organizational innovation, not just bigger models.

Economic implications for software labor

If full completion lingers below thirty percent on realistic specs, agents augment rather than replace feature teams. Product managers may still spend review cycles fixing partial handoffs, shifting costs rather than eliminating them. Co-Design gives CFOs a sober metric for pilot ROI instead of demo videos.

Outsourcing firms and consultancies may reprice fixed-bid projects using benchmark-derived risk adjustments. Insurance for software delivery could eventually reference third-party eval scores—a speculative but plausible second-order effect.

Gaming and contamination controls

Maintainers enforce submission rules: frozen model weights, disclosed tool access, capped compute. Hidden holdout sets and cryptographic task signing reduce leakage. Nevertheless, agents trained on GitHub’s firehose may internalize stylistic patterns that accidentally help on benchmark subsets; ongoing refreshes are mandatory.

Vendors tempted to cherry-pick domain slices for press releases face backlash from maintainers publishing per-category breakdowns. Co-Design’s narrative is intentionally humbling—26.4 percent is the headline for a reason.

Relation to safety and compliance

Incomplete code can be dangerous: partially implemented auth checks look finished in demos but fail open. Benchmark requirement tags include security clauses—rate limiting, secrets handling—to score silent omissions. Regulators studying AI in critical infrastructure may prefer such granular scoring over aggregate pass rates.

Enterprises running private Co-Design forks can tailor requirement libraries to internal standards—HIPAA logging, PCI segmentation—turning the benchmark into a continuous compliance drill for coding agents deployed internally.

What developers should do

Treat public leaderboard scores skeptically until vendors publish trajectories on full completion, not auxiliary metrics. Pilot agents on internal tickets instrumented like Co-Design tasks: explicit requirement lists, automated gates, human sign-off. Invest in repo hygiene—tests, CI, small modules—because agents leverage structure humans already neglect at peril.

Research agenda ahead

Closing the gap between 69.9 percent requirement satisfaction and 26.4 percent full delivery invites work on long-horizon planning, memory across sessions, and realistic cost models for tool use. Multimodal inputs—Figma frames, architecture whiteboards—will expand query types beyond prose.

November 2026’s release resets expectations after years of benchmark inflation. Co-Design Bench does not crown a winner; it maps how far coding agents remain from trustworthy autonomous developers. For an industry selling transformation, 26.4 percent full completion is the honest floor from which the next generation of systems must climb.

Procurement checklists for enterprise buyers

Chief information officers evaluating coding-agent vendors in late 2026 should embed Co-Design-style acceptance tests in pilot statements of work. Require vendors to run frozen model snapshots on a representative slice of queries, publish full-completion rates alongside requirement-coverage metrics, and disclose tool budgets used per task. Contracts that tie renewal to improvement on those metrics align vendor incentives with outcomes buyers actually experience in ticketing systems, not demo theater.

Security reviews should examine whether agents granted repository write access can exfiltrate secrets during multi-step runs; Co-Design’s security tags offer a template for red-team scenarios bundled with functional acceptance. Legal teams may insist on indemnities when generated code infringes third-party licenses—a risk magnified when agents remix stack-overflow patterns at scale.

Lessons for computer-science education

University programs teaching software engineering can adopt Co-Design derivatives as capstone rubrics, grading students and agents on the same requirement matrices. That pedagogy makes visible where human novices still outperform machines—often on taste, product judgment, and stakeholder communication—and where machines already accelerate grunt work. Faculty senates debating academic integrity can reference public completion rates when setting policies on AI assistance in coursework.