What the vLLM AFD numbers ask of your interconnect
The +11.3% that vLLM's AFD plugin reports at 64A16F [s1] is a number you cannot carry to your own cluster, and the release does not publish what you would need to earn it.…
← All topics Topic
evaluation
Count the irreversible actions an agent takes before it stops
The abstention score a benchmark reports is a rate, and the number that decides how you wire an agent is how many irreversible actions run before it stops. Cyera's enterprise…
A private SWE-bench holdout moves the trust problem to its author
A private, contamination-resistant holdout lowers leakage risk but hands you a number no outsider can check. I read that as a trade: it buys contamination-resistance with…
Kimi K3 ranks fourth, and nobody has measured the model you would run
Every public number attached to Kimi K3 describes a rented API, not the weight file that is supposed to land on July 27. Artificial Analysis scores the model 57 on its…
Your agent attends to the right tool and still picks the wrong one
The reason your agent picked the wrong tool is almost never that your tool description was badly worded. In one 2026 study, prompt-repair recovers at most 23 percent of failures…
Your CLAUDE.md structure is not why the agent stopped listening
Your CLAUDE.md structure is not why the agent stopped listening. Compliance falls about 5.6% in odds with every additional function the agent generates [s1], so the variable that…
Your Injection Defense Was Never Tested Against an Attacker Who Adapts
The three mid-2026 results that read like a field-wide fight about prompt-injection defense are not disagreeing with each other, and once you name the layer each one attacked,…
Plus 24 Percent and Minus 19 Percent Are Both True
There is no portable productivity number for coding agents: in this evidence base the instrument outweighs the tool, and proxy choice sets the sign of the headline. A study of…
The Harness Beat the Model by 4.6 Points of Almost Nothing
The analysis everyone cites to prove the harness beats the model puts the harness at about 5.3% of the variation in success and the model at 0.7% [s2], a ranking that dwarfs…
The Reward Is the Weak Link in Your Coding Agent
The hard part of training a coding agent is no longer generating a solution, it is verifying one, and that single relocation quietly breaks the reward recipe most teams are still…
Most Multi-Agent Reasoning Wins Are a Compute Artifact
Normalize the thinking-token budget and the multi-hop reasoning edge people credit to multi-agent orchestration stops reliably showing up [s1]. That one control changes what the…
Code Review Agents Are Optimizing the Wrong Metric
The pitch for a code review agent is that it reads every pull request so you do not have to. An empirical study of 13 of them inverts that pitch: 60.2% of agent-only PRs land in…
Observability Is Not Evaluation
Buying an observability dashboard is not the same as knowing your agent is right, yet that is the trade most teams are quietly making. Two independent 2026 surveys show the field…
The Coding Leaderboard Partly Measures Harness Gaming
The rank order on a coding-agent leaderboard is partly a gaming score, and the gaming grows with capability instead of washing out as the models get better. The sharpest single…
Your Agent's Memory Benchmark Grades Recall, Not Trust
A 92.5 on the LoCoMo memory benchmark tells you your agent can retrieve a fact, and almost nothing about whether it will keep serving that fact with full confidence after it…
GLM-5.2 closes the coding gap. The catch is where you run it.
A near-frontier coding model under an MIT license reads like the moment your default flips, and that read is wrong: the license hands you the weights, not your data flow, because…
Temperature zero is not deterministic, and the GPU is not to blame
Set an LLM to temperature 0 and run the same prompt a thousand times, and you will not get a thousand identical answers; the standard excuse is that GPU floating-point math is…
Your agent's pass@1 score is blind to long-horizon reliability
Select an agent for long autonomous work by its pass@1 score and you optimize the wrong number: capability and reliability match at horizon one but diverge as the horizon grows,…
The scaffold, not the model, decides your SWE-bench score
Most teams read a SWE-bench number off a leaderboard and treat it as a property of the model, but for the comparison that actually drives procurement, choosing between adjacent…
Evaluation is the new compute bottleneck
Most teams budget training runs to the dollar and treat evaluation as effectively free. Then a single agent sweep on the Holistic Agent Leaderboard runs about $40,000 [s1], and…
Agents Erode the Code They Keep Editing
Single-shot pass@1 and iterative maintainability measure different things, and on the axis that mirrors real software work, today's strongest coding agents are weak and get…
AI Re-Priced the Whole Canon. It Didn't Repeal It.
Every convention I inherited bet that one cost bound software: editing code a human had to read, and a machine reader plus a stochastic author now split that cost in two. This…
Publication guardrails: an automated fact-checking pipeline
Before publishing, the agent must prove its claims, sources attached.
Evaluating a tool-using agent: beyond success rate
An agent that completes the task but wrecks the state hasn’t succeeded.