A replay environment sitting on someone's laptop is only useful when that person remembers to run it. A replay environment wired into CI runs on every pull request, surfaces regressions before they reach main, and forces the fix credibility question before merge rather than after deploy. The gap between those two situations is not philosophical. It is concrete pipeline configuration, and this post walks through how to close it.
We will cover the integration mechanics for GitHub Actions, the questions you need to answer before you can run reliably in CI, and the practical failure modes that catch teams in the first few weeks after they wire this up. Nothing here is specific to any particular agent framework. The patterns apply wherever you have a captured failure case and an agent you can run from the command line.
What You Need Before You Wire In
Before any YAML is written, three questions need answers, because each one affects what the pipeline step actually does.
First: where do your replay environments live? CI needs to retrieve them. If they are in a directory in your repository, that is the simplest case. If they are in a remote store, your pipeline needs credentials to fetch them, and you need to decide whether to pull all environments on every run or only the ones relevant to the changed code. For most teams starting out, keeping replay environments in the repo under a tests/replays/ directory is the right call. It keeps the history in version control and avoids the credential complexity.
Second: what model version do your environments pin? If your pinned version requires a specific API endpoint, the CI environment needs access to it, and it needs the right API key. This is where environment secrets enter the picture. Do not commit API keys to the repo. Put them in your CI secrets store and reference them as environment variables. If your agent uses a local model with a fixed checkpoint, the CI image needs to include that checkpoint, which may significantly affect build times and image sizes.
Third: how long does a replay run take? A test that takes two minutes to run is a different pipeline citizen than one that takes twenty. Agent replay runs can be slow because they involve real model inference, even with frozen tool state. If your suite grows to a dozen cases and each takes two minutes, the combined runtime may require either parallelizing the runs or accepting a twelve-minute CI step. Plan for this before you have twelve cases, not after.
A Minimal GitHub Actions Setup
The basic pipeline structure has three steps: install the SDK, fetch any remote artifacts, run the suite. For a team with replay environments in the repository and a standard Python agent, it looks like this:
name: Agent regression tests
on:
pull_request:
branches:
- main
jobs:
replay:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install dependencies
run: pip install relai-sdk
- name: Run replay suite
env:
RELAI_API_KEY: ${{ secrets.RELAI_API_KEY }}
run: relai run --capture ./agent.py --suite tests/replays/This is the skeleton. The --suite flag points to the directory where replay environments are stored. Each environment in that directory is a self-contained artifact with frozen tool state, pinned model version, and an output specification. The relai run command iterates over each one, runs the agent against it, and exits with a non-zero code if any case fails the specification. GitHub Actions treats a non-zero exit as a failed check, which blocks merge.
Handling Model API Latency in CI
The most common practical problem teams hit after wiring in replay tests is inconsistent run times due to model API latency. Your tool state is frozen so tool calls are fast, but the model inference calls are live and subject to whatever the API is experiencing at that moment. A test that takes 45 seconds on Tuesday morning might take 4 minutes on Friday afternoon if the API is under load.
There are two ways to approach this. The first is to set a per-case timeout that is generous enough for normal variation but tight enough to catch genuine hangs. Most teams find that three times the median run time is a reasonable starting point. The second is to run model inference against a local endpoint in CI when possible, trading API dependency for infrastructure complexity. Which tradeoff is right depends on whether you have the infrastructure to host a local inference endpoint and whether your pinned model versions are available locally.
Be careful with flaky-test suppression logic. A replay case that fails intermittently due to API latency is surfacing a real problem: your agent may fail in production under similar load conditions. Suppressing flaky replay tests can mask genuine reliability issues. The right response to a flaky replay is to investigate why the agent's behavior under load produces inconsistent results relative to the output specification, not to mark the case as non-blocking.
Structuring the Suite for Fast Feedback
As your suite grows, run time becomes a friction point. A suite that takes fifteen minutes to complete will see engineers bypassing it or merging without waiting for results. Keeping the cycle under five minutes is worth engineering for explicitly.
One approach is tagging. Cases can be tagged by the component or path they exercise. Your pipeline can then run a targeted subset of cases based on which files changed. If the PR only touches the tool parsing logic, you may not need to run cases that exercise the scheduling output path. This requires maintaining tag metadata on each case and writing the pipeline logic to select cases by tag, which is additional complexity but worthwhile at scale.
A simpler approach that works at smaller suite sizes is ordering cases by historical run time and running the fastest ones first. Engineers see quick results for simple regressions early, and slower, heavier cases run while they are reading review comments. Most CI systems support this natively without any additional configuration.
Parallelization is the other lever. If your cases are independent, which they should be since each bundles its own frozen state, they can run in parallel across multiple runners. GitHub Actions matrix jobs work well for this. Be mindful of API rate limits when running many model inference calls in parallel against a shared API key.
The Coverage Question
A CI step that runs replay tests is only as useful as the cases in the suite. An empty suite passes every build. A suite with one case from six months ago may not exercise any of the code paths that have been active recently. Coverage in a replay suite is not the same as coverage in a unit test suite, because the cases are drawn from production failures rather than engineered to exercise specific paths.
The honest way to think about replay coverage is as a floor on regression risk. Your suite contains the failures you have already encountered and captured. It verifies that you do not reintroduce them. It says nothing about failures you have not yet encountered. Teams that conflate a passing replay suite with "the agent is correct" are overreading the signal.
What the CI integration actually gives you is accountability for known failures. If a replay case fails on a PR, the PR author knows before merge. If the suite passes, the author knows they have not regressed on the captured cases. That is a well-defined claim, and it is a genuinely useful one to be able to make before merging.
What CI Integration Does Not Solve
Running replay tests in CI does not eliminate the need for production monitoring. New failure modes will always emerge that are not in your case library. CI replay testing is defense against regression, not defense against novelty. Teams that shut down their production alerting on the grounds that CI catches everything will encounter failures that CI could not have anticipated.
There is also a culture dimension. A CI gate that engineers regularly bypass with --force-merge or by retrying until the flaky case passes is not functioning as intended. The gate's value depends on engineers treating it as meaningful signal. If your team's culture around CI checks does not include agent regression tests in the same category as unit tests, the integration adds bureaucracy without adding safety. Getting the team to treat a failing replay as a blocking signal rather than a nuisance requires making the cases accurate and fast enough that a failure actually means something went wrong, not that CI is being annoying.