Passing local tests does not mean your agent will behave the same way in production. This is not a new observation for software engineers, but for agents that use language models to make decisions, the dev-to-prod gap has a different character than it does for deterministic code.
In a traditional service, if your unit tests pass and your integration tests pass, you have covered most of the failure surface. An agent that calls a language model is different: its behavior depends on inputs that you did not fully specify, tool call outputs that change over time, and a model whose outputs are not strictly reproducible. The conditions you test locally are almost never the conditions the agent encounters in production. That gap is not incidental. It is structural, and closing it requires treating production failures as first-class test inputs rather than noise to be ignored.
What Actually Changes Between Dev and Prod
The most obvious thing that changes is external state. In dev, you mock your tool calls. The search API always returns the same three results. The database query returns the same fixture. The upstream service responds in under 100ms. In production, the search API returns different results for the same query depending on the current index state. The database query touches real rows that have been modified since your fixture was created. The upstream service occasionally times out and returns a partial response.
For deterministic code, the fix is straightforward: use more realistic test data. For an agent, the problem runs deeper. The language model's decisions depend on the content of those tool call outputs, not just their structure. An agent that decides whether to escalate a support ticket based on a knowledge base search will behave differently when the knowledge base has been updated since your fixtures were captured. The model is not wrong. It is responding correctly to different inputs. You simply did not test those inputs.
The second change is context. When you write a test for an agent, you write a clean prompt with clear inputs and a well-scoped task. In production, prompts arrive with encoding artifacts, unexpected field orderings, truncated histories, and conversational noise accumulated over many turns. A context window that comfortably fits your test case may be close to the limit for a real production message. When that happens, the model receives a truncated version of the inputs you intended to provide, and it does its best with what it received.
Context window truncation is particularly hard to catch because it degrades gracefully from the model's perspective. The model does not know what was cut. It produces a response that looks coherent. Your monitoring will not flag it unless you are explicitly instrumenting context utilization at each step. This is one class of failure that is nearly invisible unless you are comparing what you sent against what the model actually received.
Model Version Drift
The third major source of divergence is model version drift. Most teams pin a specific model version in staging but accept automatic minor version updates in production, or simply let the production version fall out of sync over time. A minor version change in a model's instruction following can shift how the agent parses JSON output, how it interprets an ambiguous instruction, or which of two equally plausible actions it takes at a decision point. These changes do not show up as errors in your logs. They show up as different behavior, and the difference is often subtle enough that it takes several incidents before the pattern is visible.
This is a harder problem than the external state problem, because you may not even know a model version change happened. If your production system auto-updates, and your staging system is pinned, the two environments will silently diverge, and the tests you run in staging will not reflect what the model will do in production.
A Concrete Failure Scenario
Consider an agent that handles customer support routing. It reads an incoming ticket, calls a search tool to find relevant knowledge base articles, and classifies the ticket into one of five categories based on what the search returns. In dev, the search tool returns fixtures, and every test ticket routes correctly.
In production, the agent encounters a ticket with an unusual structure: the subject line contains HTML entities that were not decoded before the ticket arrived. The ticket body is long, and after a multi-turn conversation history, the context window is near its limit. The search tool returns three articles, two of which were added to the knowledge base after the last fixture update. The model reads the articles, notes that two of them describe a feature introduced after the test scenarios were written, and makes a classification decision based on information the test suite did not anticipate. The ticket routes to the wrong category.
None of this is a bug in the traditional sense. The agent is doing what it should do. The failure is that the production environment is different from the dev environment, and the difference is large enough to change the agent's decision at a key step.
How Replay Environments Change the Analysis
A replay environment captures the actual production context: the real tool call outputs, the real message history, the real model version, the real context window state at the moment of failure. When you replay a failure, you are not asking whether the agent works with inputs like these. You are asking whether the agent works with exactly these inputs, in exactly this state.
That distinction shifts the debugging question from "why might this category of input cause a problem" to "why did this specific input cause this specific output." The second question is much easier to answer, and the fix you derive from it is grounded in a specific observation rather than a guess about what might have gone wrong.
Replaying the routing failure above, you would see the HTML entity encoding in the subject, measure the actual context window utilization at the classification step, examine the two new knowledge base articles, and identify which article influenced the classification. The fix might be to normalize HTML entities before passing the subject to the agent, to trim conversation history earlier, or to add the two new articles to the fixture set and verify the routing still holds. Each of those is a specific, testable change. You do not have to guess.
The Reasonable Objection
An experienced engineer will raise this: "We run evals with diverse inputs. We have hundreds of examples in our eval set. Why is that not enough?"
The answer is that evals test the agent in an eval environment, not in a production environment. They tell you how the agent behaves across a distribution of inputs you have anticipated. They do not tell you how it behaves when it encounters the specific combination of context state, tool call outputs, and model version that existed at the time of a real failure. The eval set is a statement about expected behavior across a designed distribution. A replay is a recording of actual behavior in an observed state. They are complementary, and one does not substitute for the other.
Evals are good at catching regressions you can describe in advance. Replay environments are good at diagnosing failures you could not have described in advance, because you did not know the exact conditions until they occurred in production.
What Replay Does Not Solve
Capturing and replaying a production failure tells you what happened and whether a proposed fix resolves the captured case. It does not tell you whether the fix generalizes to similar cases you have not yet captured. A fix verified against one replay is a floor: you know the specific failure is gone. You do not know whether the same code path has other failure modes under different inputs. That question still requires evals and broader testing.
The replay environment is a precision tool. It is very good at one thing: confirming that a specific observed failure no longer occurs, so you can merge with confidence rather than hope. The dev-to-prod gap is structural for LLM agents. Treating production failures as test cases is how you start closing it.