Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
AI Summary
This paper proposes an interaction-centric taxonomy for diagnosing agent failures. Instead of labeling only the system-level outcome, it maps failures to interactions between components such as the model, harness, user, tools, memory, environment, and grader, while identifying the side responsible for repair. The taxonomy contains 41 failure modes. Its intended use is operational: model-side failures suggest post-training, harness-side failures suggest scaffolding or tool-integration changes, and environment or grader failures suggest redesigning evaluation conditions.
Why it's worth reading
Agent evaluation is moving beyond binary success rates. This 41-mode, interaction-level framework is timely because it connects observed failures to concrete repair ownership across models, harnesses, environments, and graders.
Deep Read
1. What happened
Original fact: The paper introduces an interaction-centric taxonomy that localizes agent failures to interactions among system components and identifies the responsible side. The abstract states that the taxonomy contains 41 failure modes. Analysis: Its purpose is to solve the repair-assignment problem across models, harnesses, environments, and benchmarks.
2. Core technology
Original fact: Each failure is assigned to an edge between two components, together with a fault side indicating where the repair belongs. The component set includes models, harnesses, users, tools, memory, environments, and graders. Analysis: This turns an outcome label into a directed interaction-and-responsibility representation.
3. Key evidence and numbers
Original fact: The abstract reports 41 failure modes and argues that conventional evaluations overemphasize system-level outcomes, while existing taxonomies are often benchmark-specific. Unverified inference: The supplied abstract does not include the complete 41-mode list, annotation statistics, case counts, or measured repair gains, so coverage and reliability cannot yet be assessed.
4. Why it matters
Analysis: The same visible failure can originate in model behavior, tool orchestration, memory operations, environmental constraints, or grader logic. A success-rate-only view can lead teams to perform ineffective post-training or misdiagnose infrastructure issues as capability gaps. Interaction-level localization could make evaluation outputs more actionable for engineering.
5. Practical impact
Analysis: Agent teams could record the failed interaction, responsible side, and proposed repair in incident reports. Evaluation platforms could report failures separately for models, harnesses, tools, and environments. Benchmark developers could audit whether task definitions, environment feedback, or graders create false failures. The framework may also support regression testing and repair prioritization.
6. Limitations and uncertainty
Original fact: Only the abstract was supplied here; the paper's annotation protocol, cross-benchmark validation, inter-rater agreement, and real-world repair experiments were not verified. Analysis: Responsibility may be distributed across multiple components, especially in feedback loops and tightly coupled systems. Assigning a single fault side may also introduce subjectivity. Whether the taxonomy consistently improves repair decisions remains an empirical question.
7. Original sources
- arXiv:2607.28802
- Source listing: Hugging Face Papers (hf-papers)
- Published: 2026-07-29T20:00:00.000Z