When a traditional backend service regresses, the debugging path is well-worn. You have a commit range, a known input, and a diff. The system is deterministic given fixed code and fixed inputs, so you can revert one variable at a time and collect real signal. Bisect the commits, narrow the range, read the diff. This works because the failure is a function of code alone.
Agent systems break that assumption in three places at once. A failure in a production agent typically cannot be resolved by bisecting commits because the output depends not just on code, but on model behavior, accumulated context state, and external tool responses. All three can change independently, and all three influence the final output. Understanding why each dimension creates its own problem is the starting point for building any useful regression practice.
Three Variables Classical Debugging Ignores
Model Behavior
LLMs are not deterministic functions in the engineering sense. Even at temperature zero, behavior is reproducible only within a fixed model version. When you upgrade from one checkpoint to the next, the token probability distributions shift in ways that are not fully documented and not always predictable in advance. A model that consistently parsed a date field as ISO-8601 may, after an upgrade, follow a different internal path and produce a subtly different representation. The output might be correct in 99% of cases and wrong in 1%, which is exactly the kind of failure that slips past manual review.
This creates a category of regression that has no corresponding commit. The codebase is unchanged. The tests all pass. But the model running in production has a different version than the one running in your test suite, and that difference is enough to produce a new failure mode. Version control does not track model checkpoints, so bisecting by commit gives you no information.
Accumulated Context
An agent's reasoning at step seven depends on what steps one through six produced. Each tool call injects material into the context window. If a tool at step three returns a different JSON structure than expected, every subsequent reasoning step works with different material. The failure may not appear until step seven, when the agent tries to reference a field that no longer exists at the path it expected.
You cannot reproduce this by replaying step seven in isolation. You need the full context that earlier steps created, which means you need a snapshot of what every prior tool call actually returned during the failing session. Without that snapshot, you can run step seven a hundred times with what looks like "the same" setup and never see the failure, because the context that led to it is gone.
Tool State
Agents query external systems that have state the agent did not create and cannot control. A calendar query that returned "no conflicts" during a morning test run might return "three conflicts" when the same test runs two hours later because a meeting was booked in the interim. A database query that returned five rows in development might return twelve in staging because someone populated the staging environment differently after the test was written.
The agent's output changes not because any code changed, but because the world it queried changed. Tool state sits entirely outside the version control perimeter. There is no commit hash that describes what your external calendar API returned at 2:14 PM on a specific Tuesday. This is why tool state is the hardest of the three variables to account for in a regression suite: it is inherently ephemeral unless you explicitly capture and freeze it.
A Case in Production
Consider a scheduling agent that processed meeting invites for a small team. A tool call to their calendar integration returned JSON where the time zone field key had changed from timezone to tz following an API update from the calendar vendor. The agent, not finding the timezone field it expected, silently defaulted to UTC. Every meeting it scheduled for the next several hours landed at the wrong local time for users in US/Pacific.
The session log existed. The failure was real and reproducible in the sense that the same wrong output appeared in the logs. But when an engineer tried to reproduce the issue locally, the calendar API had already been patched to return the correct field name again. Running the agent locally against the current API produced correct output. The engineer could describe what happened. They could not demonstrate that any proposed fix actually prevented it, because the conditions that triggered it no longer existed.
This is the structural gap. A log entry describes a failure. A reproducible test case lets you verify a fix. Those are not the same thing, and the distance between them is the state that was present during the failure but not captured anywhere.
Why Commit Bisection Breaks Down
Traditional bisection works because a service failure is a function of code. Revert to an earlier commit, run the failing input, check if it passes. If it does, the regression is somewhere in the later commits. The input is fixed, the code is the variable.
Agent failures are functions of code, model behavior, context state, and tool state. Reverting a commit changes only one of those four variables. The other three remain at whatever values they held during the original failure, or more likely, at different values because time has passed and the world has drifted. The conditions that produced the failure have dissolved.
This is why engineers who have debugged agent failures in production often describe the process as trying to locate something that has already stopped existing. The failure occurred in a specific configuration of all four variables. By the time investigation starts, at least two of those variables have changed. You are not debugging the failure. You are debugging a similar-but-different state that happens to share some surface characteristics with it.
The Temperature-Zero Objection
A counterpoint engineers often raise: if you run agents at temperature zero, their behavior is deterministic, which should substantially improve reproducibility. This is a real benefit and it does help. Temperature zero means the model produces the same output for the same input, within a single model version. That is a genuine improvement over sampling with nonzero temperature, and teams building agents for production use cases are right to default to it.
But temperature zero does not address the other two dimensions. Tool state is not controlled by temperature settings at all. The contents of the context window are determined by what each tool call returns, and if those returns vary between sessions, the inputs to each reasoning step vary regardless of sampling temperature. Temperature zero is necessary but not sufficient for reproducibility.
There is also the model version issue. Temperature-zero outputs are reproducible within a fixed model version. When you upgrade the model, the token distributions shift, and the output at temperature zero can change. A regression triggered by a model version bump is not temperature-controllable. You need the model version pinned in addition to the temperature fixed.
What a Regression Suite Actually Needs
A useful regression suite for agents needs to capture more than a description of a failure. Each case needs to bundle several things together: the full context accumulated up to the failure point, the tool responses frozen to what they were at failure time, the model version locked to what was running, and a specification of what correct output looks like for this specific case. Without all four, the suite describes failures but cannot verify fixes.
This is not a novel idea. Browser automation has used record-and-replay for years. Compiler testing uses golden files. VCR-style HTTP recording has been common in API testing since at least 2012. The principle is familiar. The challenge with agents is that each captured case interleaves model calls with external state reads across potentially many steps, which makes the mechanics of capture and replay more involved than recording a single HTTP request and response pair.
What This Approach Does Not Solve
Capturing and replaying known failures does not give coverage over failure modes you have not yet encountered. An agent that handles the captured stale-field case correctly might still fail when a tool returns an HTTP 429, or when the context window truncates mid-plan on an unusually long session, or when the model produces a valid but semantically wrong answer that your output specification does not catch. Replay gives you a regression floor. It does not eliminate undiscovered failure modes.
A replay-based test suite answers a specific question: given that this failure occurred, did the proposed change prevent it? That is a narrower claim than "this agent will not fail in production." Integration testing, exploratory testing, and production monitoring still matter. The replay case is specifically for ensuring that a failure you already saw does not silently recur after a change.
The practical consequence is that the discipline of capturing failures as they occur matters as much as the tooling for replaying them. A replay environment is only as useful as the failures you have actually captured. The bottleneck for most teams is not the replay infrastructure. It is the absence of a habit of converting production failures into bounded, replayable cases at the moment they occur, before the ephemeral state that defined them drifts away.