> ## Documentation Index
> Fetch the complete documentation index at: https://docs.signalmd.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Test a fix
> Turn production calls into tests, run them against a version of your agent, and read what a pass and a fail mean.
A version comparison tells you whether a fix worked once enough callers have met it. A test tells you sooner: you run production calls against a version of your agent and read whether the fix held and whether anything that used to work broke.
## What a test is
Most tests start from a stored call; others start from your brief or a description. A test page shows what the test plays and checks:
| Section | What it holds |
| - | - |
| **Caller** | Who plays the caller: the **Goal**, and a **Persona** with the difficult things the original caller did (refused a question, asked for a person, went silent) |
| **Criteria** | What must all be true to pass: the caller's goal, the source issue written as its opposite ("the date booked is the date the caller said"), the ending, and the lines of your [brief](/concepts/the-brief) about this part of the agent |
| **Start and tool answers** | How the call began and the answers each tool gives, as recorded on the original call. A test does not touch your real systems, and it does not pass one day and fail the next with no change to the agent |
Personal data is masked before a test is stored or sent to Retell: each real name, number or address becomes one fake value, used consistently across the test. Criteria that point at the brief follow it: edit the line and every test that uses it changes too. A line since removed shows "Brief line changed" and stops the test passing until someone fixes it.
## What can be tested
Tests run as text: Retell plays a simulated caller against your agent's own prompt or flow, with no audio.
| Lane of the issue | Can be tested | Why |
| - | - | - |
| Understanding | Yes | The words are given; what the agent does with them is the test |
| Flow and state | Yes | Steps, transitions and stored values do not depend on audio |
| Integrations | Yes, except timing | What the agent sends a tool and does with the answer shows in text. Fixtures answer at once, so a slow tool does not |
| Hearing | No | A misheard word needs audio |
| Speaking | No | Timing, talking over the caller and long turns only exist on a call |
SignalMD refuses a test it cannot run and says why: the issue needs audio or timing (phone tests are not built yet), or it has no lane, such as callers hanging up in the first seconds. A run also needs the version's configuration stored in SignalMD, and an agent with MCP tools cannot be tested, because those tools would reach your real systems during a test.
## 1. Turn testing on
Testing is off until an admin turns it on. Open **Runs** under **Test** in the sidebar, select **Settings**, and in the **Testing** panel:
1. Select **Testing off** so it reads **Testing on**.
2. Paste a Retell key that can write under **Retell write key**. It is never shown again.
3. Set a **Monthly test budget (USD)**: the most your tests may spend at Retell this month.
4. Select **Save**.
Nobody can run a test without all three. The panel shows what has been spent this month.
## 2. Create tests
Open a call that is evidence for an issue, choose it in **Pick an issue** and select **Test issue**. SignalMD builds a test that checks this issue no longer happens on that call.
On the issue page, select **Generate tests**. SignalMD proposes replays of up to 3 of the issue's calls, one per kind of caller it happened to, then written variations that try to provoke the same problem another way.
On **Tests**, select **Generate** and pick **From calls** (real calls in a period), **From agent config** (the use cases in your brief) or **Describe cases**. Choose the use cases and **Personas** to cover, and optionally a **Focus** and **What counts as success**.
Generation only proposes. The preview offers **Minimum** (every use case and persona at least once), **Recommended**, **Full** and **Custom**, with what one run would cost. Select **Accept selected**, optionally with a group name to run the group together. **Personas** offers kinds of caller **Identified on calls**, a **Standard set** of eight, or **Your own**; a replay plays its own caller.
Replays of real calls join the release set when accepted. Written tests run and report but never gate a release, because no real caller stands behind them.
## 3. Run them
On **Tests**, select **Run…**, or select tests and **Run selected**. Choose which tests (**Selected**, **Filtered**, **Release** or **Group**) and the **Agent version**, then start the run. The dialog shows the **Reserved budget**, and **View run** opens the run.
A run targets a version whose configuration SignalMD has stored: the version current when you connected, and each new version SignalMD has seen since. To try a change before it goes live, the copilot can draft it and test it on a copy of your agent; see [Build](/copilot/build).
When SignalMD sees a new version, it runs the release set automatically if this month's budget covers every test and its possible reruns. If not, the run shows **Waiting for approval** until someone selects **Approve and run**. Add tests to the release set with **In release** on a test or **Add to release** on a selection.
## 4. Read the results
The run page, such as **Run against v12**, shows the **Pass rate** and counts of each result. **Change from the previous run** compares it with the last completed run on another version that covered the same tests: **Fixed**, **Newly broken**, **Still failing** and **Still passing**. Select a row under **Results by test case and persona** for each criterion's verdict, the transcript and Retell's explanation.
| Result | What it means |
| - | - |
| **Passed** | Every criterion was met and Retell's own grader agreed |
| **Failed** | A criterion was not met, with the turn that shows it. A failed test is run again once, and a second failure stays failed |
| **Flaky** | Failed, then passed when run again. Never counted as a pass |
| **Error** | Retell could not run it, the result could not be decided, or the run never reached the step or tool the issue was about. A fix that routes callers around the broken step cannot pass: not reached is not fixed |
| **Caller drift** | The simulated caller stopped acting like the source caller. Neither a pass nor a fail |
A pass means your agent handled that caller, on that path, with those tool responses, and nothing more.
After every run, [auto-improve](/guides/auto-improve-tests) checks whether the result was about your agent or about the test, and fixes the test when the test was at fault.
### When a test counts toward a release
The **Gates** column on **Tests** says whether a test's result counts: **Yes**, **Not yet** or **Never**. A replay gates only once it has been calibrated, which plays it three times:
* A test of an issue that does not reliably fail on the version its source call ran on is marked **Does not reproduce**: a test that passes on the broken version tests nothing.
* A test of a path that worked that does not reliably pass on the version in production is marked **Unstable**.
The app has no control for calibration yet. A signed-in person starts it through the API with [Calibrate tests](/api-reference/runs/calibrate-tests); API keys are refused. Until then, replays show **Not yet** and their results do not gate.
Guard tests run more than once on every release and pass only when every play passes. A test is a guard when its issue is marked **Fix deployed**, **Measuring**, **Resolved** or **Regressed**, or when it is marked as one.
## What a run costs
Each test in a run is one simulated call that Retell runs on your account, through your write key, and bills to you. Before a run starts, SignalMD shows its estimated cost and reserves it against the monthly budget, counting possible reruns. A run the budget cannot cover is refused or waits for approval. SignalMD's own checking of the results is not charged to your Retell budget. Simulated tests also draw on your plan's monthly credits; see [Pricing](/billing/pricing).
## Tests and version comparisons
Tests answer before callers meet the change, for three lanes. A [version comparison](/guides/compare-versions) answers after it ships, for every real caller and every lane. A test can pass while production does not move, if the calls you tested were not the ones that mattered, so read both before you mark an issue **Resolved**.
From a pipeline, the API can create tests, start runs and read results: see [Start a run](/api-reference/runs/start-a-run).
## Next steps
Read what production did after the change.
From an issue on the board to a shipped change.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.