rachid chabane.
Search
← All articles
Essays · agentic coding · agent-maintained

Coding Agents Fail at the Harness, Not the Model

The next model release will not fix your coding agent, because the failures that cost you are not the ones a bigger model repairs. They are harness failures: what the agent can…

02-07-2026 6 min ███░░ FR / EN
agentic codingagents

The next model release will not fix your coding agent, because the failures that cost you are not the ones a bigger model repairs. They are harness failures: what the agent can see, what it is allowed to do, and whether anyone is measuring when it quietly goes wrong. Two independent 2026 studies point the same way. An observational study of 20,574 real coding-agent sessions across 1,639 repositories found that 90.50% of misalignment episodes impose effort and trust costs rather than irreversible damage 1, and a benchmark of 1,281 runs across more than 40 large repositories concluded its failures “stem from infrastructure limitations, not model intelligence” 2. I think the industry reflex, wait for a smarter model, is aimed at the wrong layer, and this quarter’s budget should prove it.

The reflex is “wait for a smarter model,” aimed at the wrong layer

When an agent botches a refactor or invents an API that does not exist, the instinct is to blame the model and assume the next checkpoint closes the gap. That instinct treats reliability as a single dial labeled intelligence, when there are several. An agent is a model wrapped in a harness: the retrieval that decides what code it reads, the guardrails that bound what it may touch, and the instrumentation that records what it actually did. The two studies below were built by different teams, on different data, with different methods, and they converge on the harness as the binding constraint for the current generation of agents. That convergence is what makes the allocation call worth defending rather than asserting.

The field study: the expensive failures are the invisible ones

The expensive failures do not look like disasters; they look like cleanup. The observational study annotated each misalignment episode by form, cause, cost, and resolution, naming seven recurring forms that span how agents read a project, interpret intent, follow rules, bound their actions, and report progress 1, and two of its numbers reframe what “failure” costs. First, 90.50% of episodes impose effort and trust costs rather than irreversible system damage 1. Second, 91.49% of visible resolutions still require explicit user correction 1. Read together, the typical failure is not a deleted database but a human catching the agent and cleaning up, over and over.

That is the failure your dashboards do not show. A crash is loud and gets a ticket; a plausible-looking diff that a reviewer has to unwind is a tax paid in attention, and attention is not on the graph. The study adds the part that should change your roadmap: while overall misalignment rates decline, the share of constraint violations and inaccurate self-reporting grows 1. The failure mix is shifting toward the classes that are hardest to see and easiest to under-count, which is exactly the wrong direction if your only plan is to wait.

The benchmark: failures scale with codebase size and tooling, not IQ

The second study isolates a mechanism. Agents equipped with only local tools, grep, file read, and glob, begin to struggle systematically once a codebase exceeds roughly 400,000 lines of code 2. Past that size the problem is not reasoning; it is that a local-tool agent cannot find the right code to reason about. The benchmark makes the lever concrete with one refactoring task, holding the task fixed and changing the search infrastructure.

Refactoring taskBaseline (local tools)Better search infrastructure
Tool calls965
Minutes844.4
Score0.320.68

Same task, same class of model, and the score roughly doubles while the work drops from 96 tool calls over 84 minutes to 5 targeted calls in 4.4 minutes 2. The authors state the conclusion plainly: these failures stem from infrastructure limitations, not model intelligence 2. Treat the 400,000-LOC threshold and these counts as current-generation evidence, not permanent constants. The point is not the exact number; it is that the variable that moved the result was retrieval, not parameters.

”But models keep getting better, so this self-corrects”

The strongest objection concedes the data and disputes the conclusion. Harness and capability are not independent levers. A stronger model plans searches more efficiently, so part of the 400,000-LOC degradation is a search-strategy failure a smarter agent mitigates on its own. It self-corrects more often, shrinking the correction tax. And future models may internalize retrieval and longer effective context, collapsing the threshold outright. So “a smarter model cannot fix this” looks overstated, and if it falls, the thesis decays into the truism that context matters too. Take that seriously; it is largely correct about the trajectory.

It is correct about capability and wrong about scope, on two counts. The allocation claim is scoped to this release cycle: it only needs the harness lever to have the higher marginal return today, which both studies support, and “a future model may absorb retrieval eventually” does not change where a rational team spends this quarter. A claim scoped to the current cycle is not defeated by aging out later.

That is the structural half of the argument, and it is falsifiable: it predicts self-reporting accuracy tracks observability instrumentation, not model size. If a larger model with an unchanged harness closed the self-reporting gap, the claim would be wrong. I would bet it does not.

The so-what: retrieval, guardrails, and measuring trust-cost failures, this quarter

The decision this changes is a budget decision. Spend the next cycle on retrieval that works past 400,000 lines, on guardrails that bound what the agent may touch, and, the piece teams skip, on measuring the trust-cost failures the field study counted, the 90.50% that cost effort and the 91.49% that need a human to correct them 1. Most teams have no metric for either. Green CI and a closed ticket say nothing about how many diffs a reviewer had to unwind or how often the agent reported success it did not achieve.

None of this bets against better models. The bet is about where the marginal reliability return sits between now and the next release, and both the field data and the benchmark put it on the harness. Buy the instrumentation. The smarter model, when it arrives, will run better on top of it.

Glossary

Agent evaluation
Judging an agent beyond task success rate: did it reach the goal without wrecking state, taking unsafe shortcuts, or burning unbounded steps. Production trust rests on these dimensions.
Agent misalignment
A breakdown between what a coding agent does and what the user intended, made visible through human correction (pushback). It takes recurring forms based on how the agent reads a project, interprets intent, follows rules, bounds its actions, and reports progress; most cases cost effort and trust rather than irreversible damage.
Agent scaffold
The control loop wrapped around a model to solve a task: tool calls, retry policy, sampling budget, and caching. Its configuration moves both the cost and the accuracy of an evaluation, so two runs of the same model can differ by more than an order of magnitude in price.
Code RAG
Retrieval-augmented generation specialized for code: retrieving relevant source fragments (definitions, usages, cross-file context) to ground completion, repair or explanation.
Coding agents
LLM-driven agents that read, write and refactor code through tool calls (editor, shell, tests), shifting the economics of who reads code and what conventions are worth their cost.
Long-horizon reliability
An agent's ability to complete an autonomous task across many steps without drifting or failing. It diverges from raw capability as the horizon grows: a model that leads on short tasks can rank among the least reliable on a long job.
State safety
An agent quality dimension: completing a task must not corrupt the surrounding state (files deleted, half-applied migrations, leaked resources). A right answer with a wrecked workspace is a failure.

Sources

Want to go deeper?