Model Selection for Coding Agents: A Harness Engineering Framework

Every team building coding agents eventually hits the same bottleneck: choosing a model provider is not a benchmark sprint but an ongoing operational constraint. Pass rate on a leaderboard tells you little about how a model behaves under concurrent tool calls, retry storms, or non-deterministic API latencies. This post offers a harness-centric framework for evaluating LLM providers across four axes that directly impact agent reliability and cost: pass rate, token cost, latency, and harness compatibility. We assume you already have an evaluation harness—if not, build one before selecting a model. The site tracks related research papers separately; here we focus on engineering practice.

Pass Rate as an Observability Signal

Pass rate is the fraction of end-to-end coding tasks that result in a correct, executable solution under controlled conditions. But raw pass rate is misleading without eval isolation. Run each model variant in a dedicated environment so that transient side effects—leaked environment variables, leftover files, or port conflicts—do not corrupt results across trials. Instrument every eval run with structured logs capturing model responses, tool call sequences, and final verdicts. This turns pass rate from a single number into a debuggable trace.

Use replay to isolate failures. When a task fails, replay the exact tool call sequence against a reference model or a fixed completion to determine whether the failure originated in the model’s reasoning, the tool execution, or the harness itself. Replay is also your primary tool for regression testing after model updates. Without it, you cannot distinguish a model regression from a harness bug.

Token Cost and Retry Budgets

Token cost is often quoted per million tokens, but the operational cost of a coding agent is dominated by retries. A single task may require multiple rounds of code generation, compilation, and test execution. Each round consumes input tokens for the prompt history and output tokens for the generated code. Model providers with low per-token prices but high failure rates can be more expensive than a pricier model that passes on the first attempt.

Define explicit retry budgets per task. For example, allow at most three generation attempts before falling back to a cheaper or faster model. Measure the average token spend per successful task, not per completion. Build dashboards that show this metric over time, segmented by task difficulty and model provider. When a provider changes pricing or behavior—which happens frequently—your harness should detect shifts in cost-per-pass before they impact your budget.

Latency and Concurrency

Latency is the wall-clock time from submission of a prompt to receipt of the complete response. For interactive coding agents, high latency breaks developer flow. But even for batch evaluation, latency affects your experiment velocity. Measure p50, p95, and p99 latency for each provider under realistic concurrency—single-threaded benchmarks hide queueing delays that appear when you run multiple agents in parallel.

Harnesses should model timeouts per step, not per task. A model that takes thirty seconds to generate a five-line function might still be acceptable if it runs concurrently with other agents, but the same latency serialized across a queue will throttle your throughput. Implement adaptive concurrency: start with a conservative request limit and increase it as you observe the provider’s tail latency behavior. Log every timeout and classify it as a provider failure or a harness misconfiguration.

Harness Compatibility: Tool Contracts and Traceability

Not all models expose function calling in the same way. Some providers require a separate tool definition schema, others embed tool calls as constrained text. Your harness must abstract these differences behind a unified tool contract: a JSON Schema definition for each tool, with strict types, required fields, and descriptions. The contract is the interface between the model and the tools. If a model cannot reliably produce tool calls that match the contract, it is incompatible regardless of its pass rate on generic benchmarks.

Validate tool call compliance during every step. When a model returns a malformed tool call—missing arguments, wrong types, or hallucinated tool names—the harness should capture the raw response, classify the failure type (syntax, missing field, type mismatch), and optionally attempt a structural repair. Track the frequency of each failure type per provider. Over time you will see patterns: some models consistently omit optional fields, others double-escape strings. These patterns inform your prompt engineering and fallback strategies.

Failure Taxonomy and Operational Decisions

A coding agent faces multiple failure modes: compilation errors, test assertions failing, infinite loops, API timeouts, and tool call malformations. Build a failure taxonomy that maps each observed failure to a root cause category. Examples:

Compilation failure – the generated code does not parse or type-check. Test failure – code compiles but fails one or more test cases. Tool error – the model attempted an invalid tool call or ignored a required argument. Provider error – the API returned a 5xx status, a timeout, or a corrupted response. Harness error – the evaluation environment misbehaved (e.g., disk full, network partition).

Each failure type may require a different retry strategy. Compilation failures benefit from showing the compiler error to the model. Test failures may need additional test outputs. Provider errors should trigger an exponential backoff and possibly a provider fallback. By instrumenting your harness with this taxonomy, you can compare providers on the distribution of failures, not just the overall pass rate. A model with a 70% pass rate but zero malformed tool calls may be more reliable in production than one with 75% pass rate but frequent tool call errors that require manual intervention.

Cost-Latency-Pass Rate Tradeoffs

No single provider dominates all three metrics. The cheapest model may have the worst pass rate; the fastest may be the most expensive. Your harness should support a tiered model strategy: use a fast, cheap model for simple tasks (e.g., one-line edits, straightforward boilerplate) and route complex tasks (multi-file refactoring, ambiguous specifications) to a more capable, slower model. The router can be as simple as a heuristic based on task description length or as sophisticated as a classifier trained on historical pass rates per task cluster.

Instrument the router decisions. Log why a task was assigned to which tier and track the resulting pass rate and cost per tier. Over time you will learn which task types benefit from the expensive model and which do not.