Skip to content
Zenteck
Latest
Analysis

Why 74% of AI Coding Tasks Still Need Human Cleanup

ByAlex Silva3 min read0 comments
new-benchmark-exposes-limits-of-coding-agents
Photo: ZenteckNew benchmark exposes limits of coding agents

Listen to article0:00 / 0:30

Software development benchmarks have long painted a glowing picture. When models test themselves on older suites like WebCompass-edit, they easily cross 99% accuracy. But real software engineering rarely consists of isolated, single-file scripts. According to a report by The Decoder, a new test suite named Co-Design Bench exposes a massive gap between controlled evaluations and daily engineering work.

Why do top models fail real-world requests?

Developed by researchers from Alibaba and several Asian universities, Co-Design Bench tests AI agents using change requests drawn straight from actual user sessions. These include modifying existing web applications based on messy inputs like receipts, audio, or multi-step instructions. Even the best-performing model tested, GPT-6 Astra, fully solved only 26.4% of tasks. While the model meets about 70% of requirements on average, it almost always leaves subtle details broken.

The root cause is complexity overload. Every single model tested suffered significant performance drops when a task required three or four different types of changes at once—such as layout adjustments, interaction logic, and content updates combined. Throwing more compute at the problem did little to help; some models burned through six times as many tokens while still delivering inferior results.

How does the environment change the outcome?

The research uncovered another critical variable: the harness. The program wrapper that allows an agent to read, write, and execute code shapes final results just as much as the underlying model itself. For instance, the Qwen3.8-27B model scored either 75% or 56% on identical overhauls depending entirely on the testing environment. This proves that leaderboards focusing exclusively on raw model metrics tell only half the story.

Pass/fail: How to test agents in your workflow

    Do not trust leaderboard scores: High percentages on older benchmarks like SWE-bench Verified often overestimate real productivity by up to 24 percentage points.

    Evaluate the pair, not just the model: When picking a coding assistant or agent setup, test both the AI model and its execution harness together in your specific codebase.

    Budget for code review: Expect developers to spend substantial time on cleanup, especially when handling intertwined tasks that mix UI changes with backend logic.

Verdict: Who should rely on coding agents right now?

Coding agents remain unmatched at scaffolding simple scripts, boilerplate code, and isolated bug fixes. However, expecting them to autonomously handle large-scale project overhauls leads straight to frustration. Engineering teams must treat these tools as junior assistants that handle the heavy lifting of drafting code, while humans retain absolute control over final integration and code review.

Sources

  1. AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test — the-decoder.com

Frequently asked questions

What is Co-Design Bench?
Co-Design Bench is a new coding agent benchmark built from real user change requests that previously tripped up AI models in practice, providing a much stricter test of true software capability.
How accurate are top AI coding models?
According to tests cited by The Decoder, even top-performing models like GPT-6 Astra fully solve only 26.4% of real-world tasks correctly on the first pass.
Why do coding agents struggle with real projects?
Agents struggle when tasks combine multiple change types, such as simultaneous layout, design, interaction, and content modifications, which overwhelm their current reasoning limits.

AI Briefing in 3 Minutes

Get what actually matters in AI & tech delivered free to your inbox. Zero spam, 1-click unsubscribe.

By subscribing, you agree to our Privacy Policy.