What changed
Relay-Bench went up on arXiv on 20 July 2026, posted by Liam Swayne against benchmark saturation 12. Its author calls it an “unsaturated, holistic, text-only benchmark” 1. The item shape is the interesting part: the test set consists entirely of composite problems, two to thirteen single-domain subproblems strung into one prompt, with further layers added through prompt encoding and deliberate context bloat 1. Domains span visual reasoning, coding, math, web-search-heavy information extraction, problem-solving, general knowledge and data analysis, and the harness restricts nothing; code execution, web search and every available tool are explicitly encouraged 1. GPT-5.5 at xHigh effort leads at 43.3% 12.
The number to compare it against
That 43.3% only takes on meaning next to what the same model, GPT-5.5 at xHigh, scores when the work arrives one task at a time. Zaikei Shimbun set up exactly that comparison when it covered the paper on 23 July 2.
| Benchmark | Score | Reported by |
|---|---|---|
| Relay-Bench | 43.3% | the paper 12 |
| Terminal-Bench 2.0 | 82.7% | Zaikei Shimbun 2 |
| FrontierMath Tiers 1 to 3 | 51.7% | Zaikei Shimbun 2 |
I read that spread as a fact about task shape rather than about raw capability. The model that clears a terminal-agent benchmark is the same model that stalls once its own step three becomes step four’s premise. That reading is mine; neither source names a mechanism.
What this changes in an eval suite
Teams copy per-capability suites because vendors publish them and they are cheap to assemble. They measure a model doing one thing with a clean context, which is not the shape anything you ship actually runs in. The fix is not a bigger suite. Add one composite item: three or four of your real subtasks chained into a single prompt, scored end to end, pass or fail on the final answer only. Per-step accuracy will look healthy while that item fails; that gap is the number worth tracking.
Impact on your team
If your agent is gated on a per-step accuracy target today, that gate is the thing to change this quarter, before the next model upgrade decision. Two concrete moves: one composite end-to-end item in the suite, and a verification checkpoint between chained steps so a wrong intermediate result never becomes the premise of the next one. What to ignore: 43.3% as a model-ranking signal. One preprint, one author and one score rank nothing.