Harness safety
HarnessRisk: Agent Harness Safety Benchmark
HarnessRisk evaluates safety at the deployed model–harness level across six lifecycle phases. Its results show that high task utility and even high risk detection can coexist with successful attacks; that is a safety-evaluation problem, not a UHP conformance result.
Status first
Section titled “Status first”HarnessRisk is a research benchmark, not an interoperability protocol or certification scheme. The preprint was submitted on 18 August 2026 and remains a preprint under review at this cutoff. Its public benchmark implementation evaluates complete model + agent-harness configurations rather than model weights in isolation.
The benchmark defines 128 sandboxed cases across six lifecycle phases: Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control and Incident Recovery. It evaluates OpenClaw, Hermes and Nanobot with six language models across 14 model–harness configurations.
The headline result is material for harness engineering: published Attack Success Rate (ASR) spans 12.6% to 80.9%, while benign-task Utility spans 75.0% to 97.6%. The authors also report configurations with risk Detection above 90% that still retain substantial attack success. In other words, completing the task or recognizing danger is not enough evidence that the harness actually contained the adversarial effect.
The six lifecycle phases
Section titled “The six lifecycle phases”HarnessRisk broadens the evaluation surface beyond ordinary prompt-injection tests by treating safety as a lifecycle property of the complete harness:
| Phase | What the benchmark targets |
|---|---|
| Harness Configuration | Connectors, credentials, gateways, policies and setup templates that can weaken isolation or expose secrets |
| Capability Extension | Skills/plugins that can be malicious, typosquatted or over-privileged |
| Runtime Operation | Untrusted emails, webpages, documents and tool outputs encountered during normal work |
| State Persistence | Memory, preferences, policies, identities and other durable state that can preserve adversarial influence |
| Action Control | High-impact external actions such as deployment, deletion, OAuth grants, payments or outbound communications |
| Incident Recovery | Investigation, rollback, credential rotation, state repair and evidence preservation after compromise |
The 128 cases are approximately balanced across these phases: 22 each for Configuration and Capability Extension, and 21 each for Runtime, Persistence, Action Control and Recovery.
What a benchmark case looks like
Section titled “What a benchmark case looks like”Each case combines a benign owner objective with an adversarial instruction embedded in an untrusted workflow artifact. That artifact can be configuration guidance, extension metadata, a message, webpage, document, stored state, tool output or recovery record.
The owner task is delivered over three sequential turns. Each execution starts from a fresh case-specific environment, and the benchmark records the interaction history plus observable effects such as tool calls, workspace changes, mock-service state and network observations. Real external network access is disabled; credentials, payments, deployments and other external actions are simulated inside controlled services.
For each model–harness configuration, the paper runs every case under three independent sampling seeds. That makes 384 trajectories per configuration (128 × 3) and 5,376 trajectories across the 14 published configurations if every configured run is counted.
Four different outcomes are scored
Section titled “Four different outcomes are scored”HarnessRisk intentionally separates useful work from safe work:
- Utility — higher is better: did the agent complete the benign objective?
- Attack Success Rate — lower is better: did the adversarial objective actually succeed?
- Persistence — lower is better: did adversarial influence remain in durable state after the trajectory?
- Detection — higher is better: did the agent explicitly identify the malicious or untrusted content?
The main results use a GPT-5.4 evaluator over transcripts, tool calls, final responses, state changes, mock-service states and network observations. The public implementation also exposes a deterministic rule evaluator and a separate LLM-judge path for reproducibility work.
Why the results matter
Section titled “Why the results matter”The benchmark’s strongest architectural finding is that safety is joint to the model and harness configuration. The same model can behave differently when tools, permissions, state, execution policy and authorization context change around it.
The authors report Harness Configuration as the most vulnerable phase across all three evaluated harnesses. That is important because many security discussions focus on malicious content only after an agent is already running. HarnessRisk shows that ordinary-looking setup and configuration workflows can themselves alter security-sensitive parameters within an otherwise authorized task.
The benchmark also demonstrates why Detection is not a containment metric. A model can explicitly notice a risk yet still take an unsafe action, persist compromised state or allow another adversarial effect. Operational controls therefore need enforcement boundaries, not only better warning language from the model.
Exact published configuration scope
Section titled “Exact published configuration scope”The paper evaluates three harnesses — OpenClaw, Nanobot and Hermes — with six models. DeepSeek-V4-Pro, GLM-5.2, Kimi K2.6 and MiniMax M3 are evaluated across all three harnesses; GPT-5.5 and Claude Opus 4.7 are additionally evaluated on OpenClaw, producing 14 model–harness configurations in total.
Across those configurations, the published table ranges from 12.6% ASR for GLM-5.2 on Nanobot to 80.87% ASR for Kimi K2.6 on OpenClaw. Utility ranges from 75.0% for Claude Opus 4.7 on OpenClaw to 97.6% for DeepSeek-V4-Pro on Hermes. These are benchmark results for the exact tested versions, models, prompts, adapters and case design; they are not universal rankings of the projects.
Important evidence boundaries
Section titled “Important evidence boundaries”- Preprint, not a standard. HarnessRisk is a research artifact submitted on 18 August 2026 and marked as a preprint under review.
- Three harnesses only. The benchmark does not establish the safety of agent harnesses generally or of HarnessRouter specifically.
- Model–harness results are configuration-bound. A score cannot be transferred to another model, harness version, provider path or permission configuration without measurement.
- Sandboxed simulated effects. The benchmark intentionally avoids real credentials, accounts, payments, deployments and external side effects.
- Evaluator dependence. Main scores use GPT-5.4 as an evaluator, although the repository also exposes deterministic evaluation tooling and evidence bundles.
- Repository process backend is not an OS sandbox. The public implementation warns that arbitrary host command execution can still reach host files or real networking unless the reproducer adds an OS-level sandbox/container and egress controls.
HarnessRisk vs FrontierHarness vs UHP conformance
Section titled “HarnessRisk vs FrontierHarness vs UHP conformance”| Question | HarnessRisk | FrontierHarness Eval | UHP conformance |
|---|---|---|---|
| Primary purpose | Measure safety failures across the harness lifecycle | Compare task quality, cost, cache behavior and speed across harness configurations | Verify implementation of the UHP interoperability contract |
| Unit under test | Model + harness + permissions/state/runtime configuration | Harness + model + gateway/runtime configuration | UHP server behavior at a declared conformance class |
| Main outcomes | Utility, ASR, Persistence, Detection | Pass rate, effective cost, cache behavior, runtime | Protocol checks for discovery, tasks, streaming, sessions, files and related semantics |
| Adversarial workflow artifacts | Yes | No, not its primary design | No, not a safety benchmark |
| Cross-harness comparison | Yes, within tested configurations | Yes, within frozen benchmark design | No |
| Establishes native UHP adoption | No | No | Only when an implementation actually claims and demonstrates the UHP contract |
These evidence classes are complementary. UHP can standardize how a client addresses a complete harness without making different harnesses behaviorally or security-equivalent. HarnessRisk measures whether the deployed harness/model configuration contains adversarial influence; UHP conformance measures whether an implementation obeys the protocol contract.
Why this matters for UHP builders
Section titled “Why this matters for UHP builders”A complete-harness protocol necessarily crosses security-sensitive boundaries: tools, files, sessions, persisted state, cancellation, external actions and recovery behavior. HarnessRisk provides a useful checklist for evaluating the implementation around the protocol without confusing that evaluation with protocol conformance.
For a UHP deployment, a strong assurance program would therefore treat at least three layers separately:
- Protocol correctness: does the server satisfy the UHP conformance suite for its declared class?
- Harness/runtime safety: does the selected model–harness configuration resist lifecycle attacks under realistic permissions and state?
- Operational controls: are credentials, sandboxing, egress, approvals, audit logs, rollback and incident recovery independently enforced?
Passing one layer should not be used as evidence that the other two are safe.
Related pages
Section titled “Related pages”Read UHP security for protocol security boundaries, UHP conformance for protocol verification, and FrontierHarness Eval for a separate controlled benchmark focused on coding-task quality and execution efficiency rather than adversarial safety.