The testing practices that work for traditional software break down for LLM agents in a specific and predictable way. Unit tests check individual functions against expected outputs. Integration tests check that components fit together. Evals check that the agent produces acceptable outputs across a sample of inputs. None of these test a specific production failure against the exact conditions that produced it.
This is the production testing gap. It is not a failure of the individual practices. Unit tests should test functions. Evals should test distributions of behavior. The gap is a third category: the ability to take a specific observed failure in production, reproduce it exactly, apply a fix, and verify that the fix resolves it. Most agent teams have no mechanism for this, and as a result, a class of regressions accumulates that neither unit tests nor evals are designed to catch.
Three Layers of Testing That Miss the Gap
Consider the typical testing stack for an agent team that is shipping to production. They have unit tests for the non-LLM components: the tool call handlers, the output parsers, the pre/post processing functions. These are correct and necessary. They do not touch the behavior of the agent as a whole.
They have evals. An eval set is a collection of input-output pairs, usually curated by hand, sometimes generated from production examples, that defines what acceptable agent behavior looks like across a range of scenarios. When you update the prompt or change the model, you run the evals and check that the acceptable-output rate does not drop. Evals are good at catching prompt changes that degrade performance in ways you anticipated when you wrote the eval set. They are not as good at catching degradation on inputs outside the eval set.
They have some form of production monitoring: error rates, response latency, user satisfaction signals. These tell you when something has gone wrong at aggregate scale. They are not designed to tell you what went wrong on a specific run.
The gap is the space between the individual function (unit test) and the aggregate distribution (eval) and the population average (monitoring). It is the specific run that failed in a specific way. If you cannot reproduce that run exactly, you cannot confirm that a proposed fix resolves it.
What Accumulates in the Gap
Two kinds of failures accumulate in the production testing gap. The first is the one-off failure that is attributed to "model nondeterminism" and closed without a fix. The agent produced a bad output once. You cannot reproduce it in dev. The eval set does not capture it. It goes in the backlog as "intermittent, low priority." This is often correct for truly random single-occurrence failures. The problem is that many failures that look random on the surface are actually deterministic given the exact production inputs. You cannot tell which kind it is if you cannot reproduce it.
The second is the regression that degrades slowly. A prompt change that makes things slightly worse on a class of inputs your eval set does not cover will not show up as a clear eval regression. It will show up as a slight increase in the failure rate on those input types over time. By the time it is visible in monitoring, the change that caused it may have been followed by ten other changes, and the causal link is hard to establish.
Without a way to reproduce specific failures, you are investigating regressions by reading diffs and guessing. That is not a method. It is a hope.
The gap is where these two failure types accumulate unresolved. Teams that ship LLM agents to production for the first time often have a period of false confidence: their evals pass, their monitoring looks clean, and things seem fine. Six weeks later, they have a dozen unresolved tickets in the backlog with descriptions like "agent gave wrong answer for unusual input" and no reproducible test case attached to any of them.
Why Traditional Reproduce Strategies Do Not Work
The standard debugging approach for a production failure is: reproduce it locally, identify the cause, write a test that fails before the fix and passes after, merge the fix. This works for deterministic code because the inputs are fully specified and the outputs are deterministic. For an LLM agent, the approach breaks at the first step.
You cannot reproduce the failure locally unless you have the exact tool call outputs that the production run received. The search result the agent saw in production is not the same as the search result you get when you run the search query now. The model's response to the same prompt may differ between the production model version and the version you have locally. The context window state at the time of failure reflects a message history that is not recorded in your application logs.
Without capturing these at the time of failure, you are reproducing a scenario that is similar to the failure, not the failure itself. That is useful for understanding the general class of problem, but it does not give you a specific test case that fails before the fix and passes after. The test you write is testing your hypothesis about what went wrong, not the actual failure.
The Mechanism That Closes the Gap
Closing the production testing gap requires capturing the full execution context of a failed run at the time it fails: the inputs, the message list at each model call, the tool call arguments and responses, and the model version in use. That captured context becomes the test case. When you apply a proposed fix and replay the captured context, you are running the exact inputs through the updated code. If the replay produces the correct output, you have evidence that the fix resolves the specific failure. If it does not, you know the fix is insufficient before you merge.
This is not a replacement for evals. It is a complement. Evals tell you how the agent behaves across a distribution of anticipated inputs. Replay tells you whether a specific observed failure is resolved. You need both. Evals without replay means you are confident about the general distribution but have no mechanism for specific failures. Replay without evals means you can fix specific failures but cannot detect broad regressions.
The Cost Side of This Approach
Capturing full execution contexts has a storage cost. Each captured run contains the full message list at each model call, which for agents with long conversations can run to several hundred kilobytes. You also need to think about what data you are capturing and how to handle it in compliance with your data policies. If production messages contain user data, your capture store has the same sensitivity requirements as your primary data.
These are real costs and worth thinking about before you instrument a production agent. The alternative is worth thinking about too: unresolved production failures that accumulate in the backlog, regressions that are invisible until they are large enough to show up in aggregate monitoring, and fixes that are merged on hope rather than verified against the actual failure case. For most teams that are actively shipping agents, the cost of the capture infrastructure is less than the cost of the gap.
This is not an argument that every failure needs a replay environment or that every fix needs replay verification. Some failures are clearly explained by reading the code and obviously addressed by the fix. The replay environment is most valuable for the class of failures where the cause is not obvious from the code alone, where the fix is not obviously correct, or where the same problem has recurred after a previous fix attempt. That is a significant enough subset of production failures to justify the infrastructure.