A pull request that closes a production bug has a credibility problem. The author says the fix works. They ran it locally, it looked right. A reviewer reads the diff, it looks plausible. Neither of them has actually reproduced the original failure and verified that the change prevents it. They are reasoning from code, not from evidence. Most of the time this works out. Sometimes it does not, and the bug resurfaces in production within a few days.
The problem is not that engineers are careless. It is that manually reproducing an agent failure is genuinely hard, often requiring state that no longer exists, and that the review process currently accepts reasoning as a substitute for verification. That norm works adequately for deterministic services, where "I changed the condition that caused this" is close enough to proof. It works poorly for agent systems, where the failure was a function of model behavior, context state, and tool responses, and "I changed the condition" is at best a partial argument.
This is an argument for a different norm: a fix proposal is not credible until it has passed the failure it claims to fix. Not "the author believes it will pass" but "it has been run against the captured failure case and it passed." The PR should carry that result as a signal to the reviewer, not as a request for the reviewer to reason through whether the diff looks right.
The Credibility Gap in Agent Fix Reviews
When reviewing a fix for a classical service regression, the reviewer has a few grounding signals. The failing test exists in the repo and was presumably red before the fix. The diff shows what changed. The reviewer can reason about whether the change addresses the root cause. If the fix is plausible and the test is now green, that is decent evidence.
For agent regressions, that workflow breaks at the first step: there usually is no test that was red before the fix, because the original failure was a production incident whose conditions no longer exist. The reviewer is looking at a diff and a description of what happened. They are being asked to evaluate whether this diff would have prevented a failure in a system with nondeterministic components, using conditions that cannot be reconstructed from the diff alone. That is a much harder epistemological task, and reviewers typically compress it into "this looks right" without realizing how much ground they have covered through inference rather than evidence.
The result is a review process that functions as a logic check rather than a verification. Logic checks catch reasoning errors. They do not catch cases where the reasoning is correct but incomplete, or where the agent's behavior under the specific failure conditions differs from what the diff implies. For deterministic systems, the gap between logic and verification is narrow. For agent systems, it can be wide.
What a Replay Gate Adds to the Review Process
A replay gate is a step that runs the proposed change against the captured failure environment before the pull request is reviewed, and attaches the result to the PR. The reviewer's question shifts from "does this diff look like it would fix the failure?" to "this diff passed the failure case: is there anything about the approach that still concerns me?"
These are not the same question. The first requires the reviewer to mentally simulate the agent's behavior under the specific conditions of the failure, which is speculative. The second is a quality check on the approach with a concrete verification result already in hand. The reviewer can focus on whether the fix is over-fitted to the specific captured case, whether it might introduce regressions in other paths, whether the implementation is correct. That is a more tractable review task and a more productive conversation.
There is also a selection effect on proposals. When a fix must pass a replay gate before the PR opens, authors tend to develop their proposals differently. They know the gate is coming, so they are more likely to reason about the specific failure mechanism rather than proposing a change that looks plausible and hoping it works. In teams that adopt this workflow, the review conversation typically moves faster because the proposal arrives with its own justification already verified rather than asserted.
A Concrete Example
An agent responsible for summarizing contract renewal notices misread a date field when the upstream contract management system began encoding dates in a format the agent had not seen before. The original format was 2025-10-01. The new format, rolled out without announcement in a minor API update, was 01/10/2025. The agent parsed this as October 1 in US format, where the upstream system intended a European day-month-year format. For contracts in US time zones this was unnoticed. For one customer in a European jurisdiction it produced a renewal date eleven months early, which triggered an incorrect automated notification.
The production session was captured. The tool response that returned the ambiguously formatted date was frozen as part of the replay environment. When the engineer working the fix proposed a change to normalize date parsing before interpretation, they ran it against the replay environment. The replay passed: the normalized parsing correctly produced October 1 rather than January 10. The PR description linked to the replay result.
The reviewer, seeing that the specific failure case had been verified, could focus their attention on a different question: would this normalization logic handle other date formats from the same upstream system correctly, or was it over-fitted to the two formats seen so far? That conversation produced a more general implementation than the author had initially proposed, because the review was about correctness and completeness rather than about whether the fix would work at all.
What the Replay Gate Does Not Do
A replay gate verifies one thing: that the proposed change passes the specific captured failure case. It does not verify that the change is correct in general. A fix can be over-fitted to a single case, passing the replay while leaving related failure modes open. A fix can pass the replay on the pinned model version but behave differently on the current production version. A fix can introduce a regression on a different path that the replay case does not exercise.
These limitations do not undercut the value of the gate. They clarify what the gate answers. The reviewer's job after the gate passes is not to re-verify the failure case: the gate did that. The reviewer's job is to check for over-fit, check for unintended regressions, and evaluate the implementation quality. That is a narrower task and a more honest one than trying to evaluate both the gate question and the quality questions simultaneously from a diff alone.
There is also a coverage question. A replay gate over a single captured failure is weak coverage. A replay suite covering a dozen captured failure cases provides stronger evidence that a proposed change is not regressing. Building that suite takes time and requires capturing failures as they occur rather than after the fact. The gate workflow is only as useful as the case library behind it.
A Note on Proposal Confidence
One objection to requiring a replay gate before opening a PR is that it adds friction to the fix workflow. The author has to run the capture, build the replay environment, run the gate, attach the result. For a straightforward deterministic fix, this can feel like overhead that adds no information, because the author already knows the fix is correct.
This objection is strongest for fixes that are genuinely mechanical: correcting a typo in a field name, adjusting a retry count. For those cases, the gate mostly confirms what the author already knows, and the overhead is real. But it also builds the habit and the infrastructure that make the gate meaningful for the non-obvious cases. Teams that only run replay gates "when they think they need one" tend to underestimate when they need one, because the cases where verification is most important are often the cases where the fix looks most obviously correct.
The value of requiring the gate consistently, even when it feels redundant, is that it removes the judgment call about when to apply it. That judgment call is where slippage happens.