When an agent fails in production, the natural response is to write a fix. The harder question is: what does a trustworthy fix look like? Not in terms of code correctness, but in terms of verifiable evidence that the fix actually addresses the failure you observed rather than a plausible theory about what might have gone wrong.
A fix that passes a replay environment is structurally different from a fix that passes a code review. Code review checks whether the change is logically sound. Replay verification checks whether the specific failure no longer occurs given the exact inputs that caused it. Both are necessary. Neither substitutes for the other. This post is about the anatomy of the second kind of verification, and what separates a confident merge from a hopeful one.
The Anatomy of a Fix Proposal
A fix proposal for an LLM agent failure has several components that are worth distinguishing. First, there is the code change itself: the modification to the prompt, the agent logic, the tool call handler, or whatever code path the failure passed through. Second, there is the failure case it targets: the specific captured context that the fix is intended to address. Third, there is the replay result: what the agent produced when run with the fix applied against the captured context. Fourth, there is the scope statement: what the fix claims to do and what it explicitly does not claim.
Most fixes that get merged without incident have all four components, even if they are not labeled as such. The scope statement is the one most often left implicit, and it is the source of most post-merge surprises. A fix that addresses a specific class of prompt truncation failure may inadvertently change behavior for inputs that were not in the failure set. If the scope is not explicit, nobody checks for that.
Reading a Replay Result
A replay result is not binary. "The agent no longer fails on this input" is the minimum bar, but there are several additional things worth looking at before merging.
First, did the agent produce the same output for the same reasons? If you changed the prompt to add explicit formatting instructions, and the agent now produces a correctly formatted response, that is good. But if the agent is now producing the right format because the new prompt accidentally shifted the model toward a different interpretation of the task, you have masked the original failure without understanding it. The replay result shows you the output changed. It does not automatically show you why it changed.
To check this, look at the intermediate steps in the replay: the tool calls the agent made, the model's reasoning if it was surfaced, the parse steps that followed. If the intermediate steps look different from a successful baseline in ways that are not explained by the fix, that is a signal to investigate before merging.
Second, did the fix change behavior on inputs near the failure case that you did not intend to change? If the failure was triggered by a specific token pattern in the input, and your fix adds a normalization step for that pattern, what does the normalization do to inputs that have the pattern but do not fail? Replay the failure case, but also replay a few nearby cases from your baseline set to check that their outputs are unchanged.
What Makes a Diff Trustworthy
Trustworthiness in a diff is a function of three things: specificity of the failure case, clarity of the causal chain, and narrowness of the scope.
Specificity means you are working from a captured production failure, not a hypothesized failure. The diff was written to address something you observed in a specific run, with specific inputs and specific outputs. That is a much stronger starting point than "we think inputs like this might cause failures."
Clarity of the causal chain means you can state why the original inputs caused the failure and why the proposed change breaks that causal chain. This requires reading the captured context step by step. In many cases, the causal chain is clear: the model received a truncated context because the message list was not trimmed, and the fix adds trimming before the model call. In other cases, the causal chain is less clear, and the fix is more of an intervention than a surgical correction. Being honest about which situation you are in changes how carefully you should review the fix.
Narrowness of scope means the diff changes exactly what needs to change and nothing else. A single prompt modification that addresses one failure class is easier to verify than a refactor that restructures the agent's state management. Both might be valid changes. The latter requires broader replay coverage before merging.
A Walkthrough: Fixing a Stale Tool Call Response
Take a concrete case. An agent that summarizes research documents calls a retrieval tool to fetch related papers. The retrieval tool has a caching layer, and in a specific production run, the cache returned a stale JSON payload from six hours earlier. The agent summarized the stale papers and produced an outdated summary. The bug report pointed to the final output. The captured context revealed the stale cache hit.
The first proposed fix: add a cache TTL check in the retrieval tool wrapper. The diff adds a call to check the cache entry's age and falls through to the live API if the entry is older than one hour.
def fetch_related_papers(query: str) -> list:
cached = cache.get(query)
if cached and cache.age(query) < 3600:
return cached
result = live_api.search(query)
cache.set(query, result)
return result
Replaying the original failure context with this fix: the cache entry is now rejected (it is six hours old), the live API is called, and the agent receives current papers. The summary is correct. Fix passes the replay.
Before merging, the scope question: what happens to runs where the cache is fresh? The fix leaves those unchanged, the TTL check passes, the cached result is returned. No behavior change for the common case. The fix is narrow and the causal chain is clear. This is a trustworthy diff.
Compare to an alternative fix: "rewrite the retrieval layer to always call the live API and remove caching entirely." That also passes the replay. The summary is correct. But the scope is much broader: you have changed behavior for every run that previously benefited from the cache, and you have introduced a new cost (additional API calls on every run). The replay verified one failure case. It did not verify that the broader scope change does not introduce new problems.
The Counter-Argument: Verified Against One Case Is Still One Case
A reasonable objection is that replay verification against a single captured failure is weak evidence. The fix passed one case. You do not know if it passes all similar cases. This objection is correct, and a good fix proposal acknowledges it. The replay is not proof of correctness. It is proof that the specific failure is resolved. The confidence you take to merge should be proportional to how well the fix's causal reasoning generalizes beyond the single case.
A fix that addresses a root cause (cache age not being checked) generalizes well: every case where the cache is stale benefits from the fix, and the reasoning is general. A fix that patches a specific output format ("add a try/except around this JSON parse and return a default") resolves the one failure case but may leave similar JSON parse failures on adjacent code paths untouched.
Replay verification raises the floor: you know the specific case is resolved, and you can merge without hoping. It does not eliminate the ceiling concern: a fix that passes replay can still leave the root cause in place. The structured diff proposal is trustworthy when the fix addresses the root cause, not just the symptom.