> ## Documentation Index > Fetch the complete documentation index at: https://docs.signalmd.dev/llms.txt > Use this file to discover all available pages before exploring further. # Auto-improve tests > When a run of a test finishes, SignalMD checks whether the result was right and fixes the test when the test, not your agent, was at fault. A test is only useful if its result is about your agent. A test that gives a tool no answer, or a simulated caller that wanders off its script, can fail a good agent or pass a broken one. Auto-improve catches those runs and fixes the test, so a failure you read is a failure of the agent. It is on for every agent unless an admin turns it off, and the **Tests** page says **Auto-improve is on**. It works from the runs you already make: your own runs and scheduled ones. Accepting a test starts nothing. A run that needs no fix costs nothing extra. ## The two questions When a run of a test finishes, SignalMD asks one question about that run: | The run | The question | | - | - | | Did not pass | **Should it have passed?** | | Passed | **Should it have failed?** | The answer is **Yes**, **No** or **Unclear**, with the reason. SignalMD answers **Yes** only when it can point at something in the run that the test decided rather than your agent. For example: * The agent called a tool the test gave no answer for. * The simulated caller left its script, or did not do what the test asked of it. * The test did not hand in a value the agent needed when the call started. * The run was cut off before the part the test checks. * A pass never reached the step or tool a criterion is about. A failure where every tool had an answer, the caller kept to its script and every check was decided is answered **No**: the agent failed, not the test. ## What it changes, and what it never does When the answer is **Yes**, SignalMD fixes only the part of the test the reason points at: | Reason | What changes | | - | - | | A tool had no answer | That tool's answer, copied from one of your stored calls | | The caller left its script | The caller's goal, lines or behaviours. A caller that stopped before saying the original caller's lines gets those lines written into the test, word for word, and is asked to say every one | | A start value was missing | That value | It never changes what the test checks: the criteria and their wording, whether the test should pass or fail, its persona, or its source call. It never removes a line or behaviour from the original caller. If the criteria look wrong, SignalMD suggests a change and waits for you to approve or reject it. Each fix is saved as a new version of the test. SignalMD then checks the fixed test in text conversations until it settles. If it cannot settle the test, it stops and the test reads **Needs you**, with the reason. Nothing more is changed until you edit the test. ## Credits A run that needs no fix spends nothing on auto-improve. Each run auto-improve plays to check a fix spends credits as a test run does, and working out a fix spends credits as chat does. When your workspace has none left, nothing more is run and the test reads **Paused**. See [Pricing](/billing/pricing). ## Tests found at fault together improve together When one run of several tests finds more than one of them at fault, those tests improve as one batch: * SignalMD checks every test of the batch that is still improving together. * When several tests need the same fix, such as an answer for the same tool, SignalMD works it out once and applies it to each of them. Each test still gets its own new version, and the version says it came from the batch fix. * Each test keeps its own rules: its own question, its own fixes and its own reason when it needs you. * Nothing a batch plays is charged to your Retell account or your test budget. Each run spends credits like a test run. If a batch is ever paused, select **Resume** on it. The **Tests** page links to the newest batch above the list. A batch shows, at the top, how many tests are **Ready**, **Improving**, **Possible bot problem** and **Needs you**, and what it cost. Below, each test is one line with its state and its version. Select a test to open its versions. ## Where a test stands The **Tests** list shows each test's state in the **Auto-improve** column, and the test's own page shows it beside the title: | Label | What it means | | - | - | | **Improving** | Fixing the test, or checking that the fixed test passes | | **Ready for calls** | Passes on this version of the test | | **Reproduces the issue** | A test made from an issue fails on that issue on every run | | **Possible bot problem** | The test was not at fault in its text runs; your agent failed them. Not confirmed on Retell: run the test on Retell to confirm it before you treat it as a problem in your agent | | **Needs you** | A decision only you can make, with the reason: a suggested change to the criteria, a test auto-improve could not settle, or runs whose checks could not be decided | | **Paused** | A batch was paused, or your workspace has no credits left this month. Upgrade, then select **Resume** on the batch, or run the test again | | **Stopped** | You edited or archived the test while it was being improved | ## Versions Open a test and select **Versions**. At the top, two versions are compared, the newest against the one before it. Pick any two in **Compare**. The comparison covers the whole test: the caller's goal, lines and behaviours, the persona, how the call starts and its values, each tool's answer, the criteria and the expected result. Removed lines are marked red and added lines green. The parts that did not change are folded into one line that names them; select it to read them. When the newer version was made by **Auto-improve**, its note sits above the comparison: * the run and the question, such as "Run failed. Should it have passed? Yes"; * **Why**: what SignalMD found in the run, such as "The tool lookup\_patient was called and had no mock"; * **Fix**: what it did about it; * **Batch fix**, when the same fix was made for several tests of a batch. A version you or a teammate made shows **A person**, and is compared the same way. Below the comparison, every version is one line, newest first, with who made it and the parts it changed. Select one to compare it with the version before it. Below the versions, **Runs answered** lists every run with its question, answer and reason. The **Runs** tab shows the same answer on each run's row. The **Auto-improve** section at the top shows that auto-improve is on, where the test stands and what it cost. ## Turn it off An admin changes it in **Runs** under **Test**, then **Settings**, in the **Testing** panel: **Auto-improve tests** is on by default. Off, every run is still answered so you can read it, and no test is changed. Select **Save** after changing it. The runs auto-improve plays to check a fix are text conversations. They cost nothing on your Retell account and do not count against your monthly test budget. A run you start yourself with **Run**, on Retell or as a phone call, works as before and is answered the same way. Runs that auto-improve plays to check a fix are kept apart from your other runs. They never count toward a test's pass rate or its calibration. ## Related * [Test a fix](/guides/test-a-fix): write tests from production calls and run them. * [Compare versions](/guides/compare-versions): see whether a change helped real callers. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.