Harness benchmark
FrontierHarness Eval
FrontierHarness Eval v1.0 holds the model, tasks and execution runtime constant to measure how coding-agent harness configurations change pass rate, cost, cache behavior and speed. It is useful harness-level evidence, but it is not UHP conformance and its frozen version snapshot is not a current-release ranking.
Status first
Section titled “Status first”FrontierHarness Eval v1.0 is a frozen coding-agent harness benchmark launched by Runta on 1 September 2026. Its public definition fixes one model — Kimi K3 through Fireworks — across 9 harnesses, 12 configurations, 30 software-engineering tasks and 360 evaluations. The benchmark repository publishes the task definitions, normalized results and harness-version snapshot used for the run.
The central result is architectural rather than protocol-level: changing the harness while holding the model, task set and execution environment constant produced materially different pass rates, cost, cache behavior and wall-clock time. The published pass rates span 50.0% to 66.7%, while the source’s effective-cost-per-pass values span about 17.5×, from $1.05 to $18.34.
What the benchmark actually measures
Section titled “What the benchmark actually measures”FrontierHarness v1.0 uses a controlled execution design intended to expose harness effects without changing the model between configurations:
- 30 tasks: 21 Terminal-Bench 2.1 tasks and 9 DeepSWE v1.1 tasks.
- 12 configurations across 9 harnesses: DeepSeek Harness is evaluated in four presets; the other harnesses have one configuration each.
- 360 cells: one canonical evaluation for each task/configuration pair.
- One model: Kimi K3, served through Fireworks.
- Fresh runtime state: each formal trial starts from a restored Runta golden checkpoint with the same vCPU, memory, disk size, disk contents and memory state.
- Deterministic pass/fail scoring: task verifiers determine success.
- No warm benchmark rehearsals: formal tasks are not used for debugging; warm and smoke labels are excluded from the frozen definition.
- Comparable cache accounting: first-turn cache reads are repriced consistently across harnesses.
The launch article also says all 12 configurations for a task run in the same job and that the benchmark uses one attempt per cell rather than best-of-N retry selection.
Frozen harness versions
Section titled “Frozen harness versions”The public version snapshot was captured on 22 August 2026. That matters because several projects have released newer builds since then. The leaderboard should therefore be read as a controlled historical experiment, not as a comparison of each project’s latest stable release on 3 September.
| Harness / configuration | Frozen version |
|---|---|
| Codex | 0.148.0 |
| Claude Code | 2.1.237 |
| DeepSeek Harness — Creator | 0.1.0-rc.8 |
| DeepSeek Harness — Minimal | 0.1.0-rc.8 |
| DeepSeek Harness — PTC | 0.1.0-rc.8 |
| DeepSeek Harness — Standard | 0.1.0-rc.8 |
| Pi | 0.84.2 |
| Oh My Pi | 17.4.0 |
| Kimi Code | 0.37.2 |
| Exo Harness | 0.1.0 |
| OpenCode | 1.18.19 |
| Hermes | 0.20.4 |
Published v1.0 results
Section titled “Published v1.0 results”The table below uses the public benchmark’s pass rate, effective_cost_per_pass, typical median cache-hit rate and median runtime. These are results for the exact frozen configurations above, not universal properties of the harness products.
| Configuration | Pass rate | Effective cost / pass | Cache, median cell | Median time |
|---|---|---|---|---|
| Codex | 66.7% | $3.47 | 88.0% | 6m 43s |
| DSH Creator | 63.3% | $3.28 | 84.3% | 6m 44s |
| Claude Code | 63.3% | $18.34 | 67.8% | 9m 38s |
| Pi | 60.0% | $2.43 | 79.4% | 7m 33s |
| DSH PTC | 60.0% | $4.58 | 87.2% | 7m 44s |
| DSH Standard | 60.0% | $3.46 | 86.5% | 6m 17s |
| Oh My Pi | 56.7% | $4.75 | 82.2% | 6m 46s |
| Kimi Code | 56.7% | $3.65 | 88.0% | 7m 56s |
| DSH Minimal | 56.7% | $4.72 | 84.6% | 5m 41s |
| Exo Harness | 53.3% | $1.05 | 70.3% | 6m 17s |
| OpenCode | 50.0% | $3.24 | 78.4% | 6m 27s |
| Hermes | 50.0% | $2.90 | 85.9% | 6m 58s |
The source reports 209 successes and 151 failures across the 360 cells. Codex leads pass rate, DSH Minimal leads median runtime, and Exo Harness has the lowest effective cost per pass in the frozen dataset.
How to read the cost numbers
Section titled “How to read the cost numbers”FrontierHarness deliberately prices failures into the efficiency picture. Its normalized dataset exposes effective_cost_per_pass, while the launch article describes the same headline values as cost per pass and discusses the cost of failed work rather than filtering failures away.
That distinction matters. A configuration can fail cheaply, solve fewer tasks and still have attractive spending characteristics for retry-heavy workflows; another can solve more tasks but spend much more on long unsuccessful trajectories. Pass rate and cost therefore answer different questions.
The source also warns that cache-hit percentage is not itself cost. One highly cached but very long trajectory can still consume more tokens and money than a short cache miss.
Important limitations
Section titled “Important limitations”FrontierHarness is unusually useful because it controls several confounders, but it does not isolate the harness as an abstract independent variable.
- Single-model benchmark. Every configuration uses Kimi K3. The result measures each harness’s interaction with that model and gateway path, not how the harness performs across model families.
- Software-engineering scope. The 30 tasks are terminal/software-development tasks. The project explicitly says v1.0 may not generalize to other knowledge work.
- Frozen old versions. The version snapshot was captured on 22 August, while the public launch occurred 1 September. Current releases can differ materially.
- Harness-model fit remains a confounder. Runta specifically notes that Claude Code’s high cost may reflect its interaction with Kimi K3’s implicit cache behavior rather than a harness-only defect.
- One formal attempt per cell. The benchmark intentionally avoids retry/best-of-N effects; production systems may choose different retry policies.
- Operator-published evidence. Runta designed, executed and hosts the benchmark. The public task definitions and normalized results make the work inspectable, but this guide does not treat it as independent certification of any harness vendor.
The project’s stated v1.1 direction is a broader harness × model matrix, which would make model/harness interaction effects more directly observable.
FrontierHarness vs UHP conformance
Section titled “FrontierHarness vs UHP conformance”| Question | FrontierHarness Eval | UHP conformance |
|---|---|---|
| Primary purpose | Compare task quality and execution efficiency across harness configurations | Verify implementation of the UHP client/server contract |
| Unit under test | Complete harness + model + gateway/runtime configuration | UHP server behavior at a declared conformance class |
| Main outcomes | Pass/fail task result, cost, cache behavior, runtime | Protocol checks for discovery, tasks, streaming, sessions, files, sharing and related semantics |
| Version sensitivity | Frozen harness/model/runtime snapshot | Pinned UHP version and conformance-suite shape |
| Cross-harness ranking | Yes, within the benchmark design | No |
| Native UHP adoption evidence | No | Only when a project actually implements/claims the UHP contract with evidence |
HarnessRouter’s published UHP conformance evidence and FrontierHarness’s HarnessRouter-adjacent harness measurements therefore belong to different evidence classes. They can complement one another, but one must not be used as a proxy for the other.
Why it matters for harness engineering
Section titled “Why it matters for harness engineering”FrontierHarness provides public quantitative evidence for a point often hidden by model-only benchmarks: the harness is a measurable part of the system. Tool-loop behavior, context management, cache interaction, prompting, stopping conditions and runtime policy can change cost and outcomes even when model weights and task state are held constant.
For UHP, that strengthens the case for keeping protocol interoperability separate from performance evaluation. A protocol can make different harnesses addressable through one contract; it does not make those harnesses behaviorally equivalent. Interoperability answers “can the client run it through the same contract?” while a harness benchmark asks “how does this complete configuration perform on this workload?”
Related pages
Section titled “Related pages”Read UHP conformance for protocol verification, harness composition for internal orchestration topologies, and the individual Codex, Claude Code, DeepSeek Harness, Pi, OpenCode and Hermes pages for current upstream/integration status that should not be confused with FrontierHarness’s frozen 22 August version snapshot.