A bug report for an agent failure typically arrives as a combination of a user complaint, a session ID, and a log excerpt. The engineer who receives it knows something went wrong. They do not yet have a test case. Those are two different things, and the gap between them is where a lot of agent debugging time disappears.
A bug report tells you a failure occurred and provides some evidence about what the output was. A test case bundles everything needed to re-run the failing agent, defines what correct output looks like for that specific situation, and lets you verify whether a proposed fix prevents the failure without requiring any manual steps. The path from one to the other involves a series of extraction and bounding decisions that are easy to underestimate as a category of work.
This post walks through that path as Relai approaches it, step by step.
Step One: Extracting What the Report Contains
The first step is reading the report for what it actually contains, not what it implies. A report typically names a symptom: "the agent scheduled the wrong meeting time" or "the agent failed to complete the task and returned an error." This is the observation. It is not the test case.
From the report and the associated session data, the extraction step pulls three things: the session identifier that links to the production trace, the specific output that was wrong and why it was wrong, and the point in the session where the failure became deterministic. That last one requires reading the trace. In many agent failures, the wrong output is the downstream consequence of a wrong decision earlier in the session. A test case that targets the symptom without capturing the earlier decision point may pass even when the root cause is unfixed.
Consider an agent that failed to route a support ticket correctly. The symptom was the wrong routing destination. But reading the trace, the decision was made at step four of an eight-step session, based on a classification tool call that returned an outdated category mapping. Steps five through eight were all reasonable given the wrong classification. A test case that checks the routing output without freezing the classification tool response would need to somehow produce the wrong classification again to test the fix correctly. The right scope for the test case is the point where the failure became deterministic, which is step four, not step eight.
Step Two: Locating the State Boundaries
A test case needs a defined scope. It starts somewhere and ends somewhere. Starting too early means capturing state that is not relevant to the failure. Starting too late means missing the conditions that produced it. The boundary question is: what is the minimal set of state that fully determines the failure?
For agent sessions this typically means the full context at the start of the relevant subsequence, plus every tool response that was issued before the failure-determining step. You are not trying to capture the entire session history unless the failure genuinely depends on all of it. You are trying to find the minimal bounded window where, given those inputs and that tool state, the failure occurs reliably under the original model version.
A practical way to verify the boundary is to ask: if I run the agent with just these inputs and this frozen tool state, does the failure occur? If the answer is no, the boundary is wrong. The scope is missing something. If the answer is yes, the boundary is sufficient. Whether it is minimal requires checking whether you can remove any component and still reproduce the failure. A test case that is larger than necessary is not wrong, but it is harder to understand and more likely to require updates when unrelated parts of the system change.
Step Three: Freezing Tool State
The tool state freeze is the step that most ad-hoc reproduction attempts skip, and it is the most common reason a manually constructed test case produces inconsistent results.
The freeze records each tool call and its response from the original failing session. During replay, instead of issuing live calls to the external systems, the environment serves back the recorded responses in order. This means the agent sees exactly the data it saw during the failure, regardless of what those external systems contain at replay time.
The tricky part is completeness. An agent session may involve a dozen tool calls, and the temptation is to only freeze the "relevant" ones, the ones that seem directly connected to the failure. This is usually a mistake. Tool calls that appear irrelevant at first reading often affect the context that subsequent steps reason over. The agent's state after step three is a function of all tool responses up to that point, not only the ones that look meaningful in isolation. A complete freeze captures all responses in the session window, not a selected subset.
There is also the ordering question. Tool calls in some agent architectures can be issued in parallel or in conditional branches. The freeze needs to capture not just the content of each response but the pairing between each call and its response, so replay can serve the right response for each call rather than serving responses in sequence and hoping the call order matches.
Step Four: Authoring the Output Specification
The last component of the test case is the most human-intensive part: defining what correct output looks like for this specific case.
The naive approach is to record the correct output and use it as a golden file for exact match comparison. This causes spurious failures whenever the model produces equivalent output in a different form. An agent that summarizes a support ticket might produce "Customer reports login failure after password reset" one time and "User cannot log in following password change" the next. Both are correct. An exact match comparison would fail on the second output.
The output specification needs to express semantic intent rather than surface form. For the routing case, the specification might be: the ticket is assigned to the security queue, not the billing queue. For a scheduling case: the meeting time is in the correct time zone, and the date matches the requested date. For a summarization case: the summary contains the key issue, the urgency classification, and the affected component.
Writing specifications at this level requires understanding what the agent is supposed to accomplish, not just what it happened to produce in a correct session. This is genuinely harder than golden-file comparison, and it is work that only an engineer familiar with the domain can do correctly. Tooling can support it, but cannot replace it.
An output specification is an argument about correctness, not a transcript. Authoring it forces clarity about what the agent was supposed to accomplish, which often surfaces ambiguities that were not visible before the failure.
Step Five: Verifying the Case Is Reproducible
Before the test case is usable as a verification gate, you need to confirm it actually reproduces the failure. Run the agent with the bounded inputs, frozen tool state, and pinned model version. Verify that the output fails the output specification. If it does not fail, something is missing from the capture.
This step is often skipped under time pressure, and it is always a mistake to skip it. A test case that does not reproduce the failure will pass regardless of whether a proposed fix works. It provides false confidence and may prevent the real failure from being caught in the future. Verifying that the case is red before applying the fix is the same discipline as running a failing test before patching the code.
Once the case is confirmed red, it can serve as a replay gate for proposed fixes: run the agent with the proposed changes against this environment and check whether the output now passes the specification. A PR that passes the replay gate has demonstrated something concrete about the fix. One that has not run the gate is still reasoning from inference.
What This Process Does Not Cover
Converting a bug report to a test case is useful work, but it addresses failures that have already occurred. It does not prevent novel failure modes that have not been encountered yet. A test suite built entirely from past failures will have blind spots wherever the failure space has not yet been explored.
The process also requires that the failing session was captured before the ephemeral state dissolved. If an engineer notices a failure two days after it occurred, and the session trace is not retained, the conversion process cannot start from the original state. The test case can only be constructed from what was preserved. Teams that want to convert failures into test cases consistently need to retain session traces for a window long enough that engineers have time to act on reports.