Failure Taxonomy for Agent Systems

Every production agent system eventually breaks. Not when a prompt is typed wrong, but when a tool call returns something unexpected, the model starts repeating itself, or the cost spirals out of control. These failures are not random; they follow patterns. If you are building the harness—the observability, retry logic, evaluation isolation, and replay infrastructure—you need a shared vocabulary for what goes wrong. This post introduces a failure taxonomy for agent systems: contract violations, environment drift, model hallucination, and budget overflow. The site tracks related research papers separately, but here we focus on practical harness engineering.

Contract Violations

The most common failure source is a mismatch between what the agent expects and what the tool or API delivers. Tool contracts define inputs, outputs, side effects, and invariants. When the agent calls a tool with arguments outside its schema, or the tool returns data that violates its spec, you have a contract violation. In the harness, these should be caught before they propagate. Use typed tool definitions with runtime validation—JSON Schema is fine, but consider codec-based checks for richer types. Log every violation at the harness level, not inside the tool implementation. That way you can replay the exact call and inspect the mismatched field. Contract violations are the easiest to detect and the most important to track because they cascade into other failures.

Environment Drift

Agents operate in dynamic environments. A database returns stale data, an API rate limit changes, a file system path becomes invalid, or the time-of-day affects a business rule. This is environment drift. The harness cannot prevent it, but it can detect it. Instrument all external calls with latency and error distribution metrics. When a tool that used to succeed starts failing at consistent times or on certain inputs, flag it. Build replay workflows that let you rerun an agent script against a snapshot of the environment state, not the live system. Eval isolation is critical here: if your evaluation harness uses production data, a slow response or a schema change can invalidate an entire test run. Use deterministic replays and freeze environment fixtures where possible.

Model Hallucination

When the model invents tool arguments, misreminds previous outputs, or produces plausible-sounding but factually wrong responses, you have hallucination. In agent systems, hallucination often manifests as “tool misuse”: calling a search tool with a non-existent parameter, or ignoring the contract entirely. The harness should treat hallucination as a probabilistic failure, not a bug. You cannot eliminate it; you can only bound it. Use structured output constraints (grammars, controlled decoding) to reduce the space of possible invocations. Add confidence checks before executing irreversible actions—for example, ask the model to self-verify or use a separate smaller model to validate tool calls. Track hallucination frequency via eval suites that replay tool sequences and check whether the chosen arguments match the tool contract. And always have a retry budget that allows the agent to recover from a hallucinated call by catching the error and feeding it back to the model.

Budget Overflow

Agent systems are not free. Each tool call, each token generation, each retry consumes time and money. Budget overflow happens when the agent exceeds a predefined cost or latency threshold. This is a harness-level failure because the agent itself has no concept of budget. Implement per-session or per-step budgets that stop the agent when limits are hit. This is not just about money; latency budgets prevent runaway loops. Instrument costs as part of the observability pipeline—track cumulative token usage, API call count, and elapsed time per step. When a budget overflow occurs, log the full trace so you can decide whether the agent was on a productive path or just spinning. Build a “budget aware” harness that can escalate to a human or switch to a cheaper fallback model before the hard limit is reached.

How the Failures Interact

The four categories are not independent. Environment drift often triggers contract violations: an API returns a field with a new type, and the tool contract fails validation. Hallucination can cause budget overflow by leading the agent into repeated tool calls that all fail. A single contract violation, if not caught, can poison the model’s context and increase the probability of further hallucination. The harness should track failure causality, not just event counts. Use structured logging with a failure taxonomy tag (e.g., failure_type: contract_violation) and correlate it with parent events. When replaying a failure, annotate each step with its category. Over time you will see which failure modes co-occur and which are root causes versus symptoms.

Operational Checklist

Using this taxonomy does not make failures disappear. But it turns noise into signal. When your harness can classify a failure as contract violation vs. environment drift, you triage faster and fix the right root cause. Start with contract validation and budget enforcement; they give you the most immediate protection. Then layer on hallucination detection and drift monitoring. The taxonomy is a living document—refine it as you see new failure patterns. And remember, the site tracks related research papers separately, so check there for deeper dives into specific categories.

Back to research index