Capture agent failures as replayable test cases.
Relai wraps every production bug in a replay environment, then proposes a fix it has to pass before your PR opens.
# agent run: invoice-extractor v0.4.2
# timestamp: 2024-11-07T14:22:31Z
tool_call(
name="extract_line_items",
args={"doc_id": "inv-9921"}
)
AssertionError: expected 4 items
got 0, table boundary misseddiff --git a/extractors/table.py
- boundary = find_boundary(page)
+ boundary = find_boundary(
+ page, expand=True
+ )
# replay verification running...
items_found: 4
assertion: PASSAgent failures are impossible to reproduce without infrastructure built for them.
Agent errors vanish the moment the run ends, leaving only a log line. There is no artifact to reproduce against, no environment to re-enter.
Patches merge without ever running against the original failure scenario. The bug may reappear on the next prompt change or context window shift.
LLM-powered agents defy traditional regression suites. Determinism cannot be assumed, and test coverage built for classical code simply does not transfer.
Three steps from failure to verified fix.
Capture
Relai intercepts and freezes the agent run at the exact point of failure, including all tool calls, context, and environment state. The frozen snapshot becomes a permanent, reproducible artifact.
Propose
Relai analyzes the captured failure context and generates a candidate fix in the form of a structured diff. The fix is applied directly to the frozen environment, isolated from production.
Verify
The fix is run against the replay. It must pass the original failure assertion before Relai opens a pull request. Nothing merges until the replay confirms the fix works.
Built for the failure modes that classical testing misses.
Replayable Environments
Every agent failure captured by Relai becomes a deterministic replay environment: a timestamped snapshot of the tool-call sequence, the context passed to the model, the outputs returned, and the assertion that failed. This is not a log. It is an interactive environment you can step into, modify, and run against, exactly as it was when the failure occurred. Relai serializes the environment with enough fidelity that a proposed fix applied to the snapshot produces consistent, repeatable results, giving you the confidence that a passing replay means a passing production run.
Relai connects to your repository and opens verified pull requests automatically. Failure captures from production route back to the same branch they came from.
Every proposed fix is expressed as a structured diff against your actual codebase, not a freeform suggestion. Reviewers see exactly what changes and why.
Retention windows scale by plan: 7 days on Developer, 30 on Team, 90 on Scale. Search, filter, and compare failure patterns across agent versions to identify regression trends.
Send failure events and fix proposals to Slack, PagerDuty, or any endpoint. Configure per-severity routing so critical failures reach the right person immediately.
Join our early-access program.
Get direct input into the product roadmap, priority support from the founding team, and founding-customer pricing locked in before general availability.
Apply for Early Access