Skip to main content
When you publish a change in Retell, SignalMD compares production calls on the new version with calls on the version before. This guide shows where to read that comparison and what to do with each result. What each metric and verdict means is in Versions.

​
1. Let calls arrive on the new version

SignalMD picks up a new version within about 10 minutes, or at once when you publish from Build. The new version starts with no calls and fills as calls arrive on their own. To bring in calls from the days since the change at once, use Import more calls on Agent config with a window that covers them. Calls already imported are skipped, so you can import again as often as you like.

​
2. Open the comparison

Open Agent config. History lists every version with its First call, Calls and Got it done, plus Handed to a person when any call ended in a transfer. A version whose configuration Retell never gave back is marked “no config”. Below it, each change has its own section titled with the two versions, such as v7 → v8:
  • Changed lists what differs in the configuration, with the old and new text for prompts. When one side was not captured, it reads “Unknown: the configuration of v6 was not captured.” and the section is tagged Calls only.
  • Effect shows how many calls each version has, then a row per metric with both values and a verdict.
  • Issues that moved, when there are any, lists up to 5 issues whose share of calls changed by a real amount.
  • The tag next to the title is the conclusion, and the sentence under the table says what moved and which row decided it.
From an issue, Compare versions under By version opens the same page.

​
3. Read it in order

1

Check the calls

The Effect heading shows calls on each version, such as “(210 → 220 calls)”. Each row has its own count: Got it done counts only calls where the caller spoke and the call has been read. Hover a value to see its count. A row without enough calls says Not enough calls rather than guessing.
2

Read the conclusion tag

Got it done leads. The guardrails and the issues that moved decide only when it did not change. Handed to a person never decides.
3

Read each row

A real change shows as a signed difference, green or red. Everything else shows a grey label (see below). Rows SignalMD has not finished reading on these calls are left out and named after “Not shown:” under the table.
4

Read the summary

It names the row that carried the conclusion. A conclusion carried by Reply time alone is a faster agent, not proof that more callers got what they came for.
5

Check the target issue

If the change was aimed at an issue, open it and read By version. The comparison is about the whole agent; By version is about the one problem.

​
4. Act on the result

​
Row verdicts

​
Conclusions

Enough calls to get a verdict is not the same as enough to see a small effect. The fewer calls a version has, the larger a change has to be to show as real, so a modest improvement on a quiet agent can read Within noise for a while (see Why a small change can read as unchanged). If you expect a small effect, plan to wait, and use tests for an earlier answer.

​
Common traps

A comparison is before and after, not a controlled test: every caller after the change meets the new version. If a campaign, a new phone number or a change upstream sent a different kind of caller, the numbers move whatever your change did. On Overview, select Edit and group the Got it done chart by Use case to see whether the mix shifted at the version marker. Editing changes the board for everyone who can see this agent.
A holding line in the prompt shortens the silence the board measures, so Reply time improves. If the cause was a slow lookup, the caller still waits for the data. Check that the number that moved belongs to the lane of the cause, and that the target issue’s By version share fell.
If the new version went live just before a spike or a quiet period, the comparison mixes your change with the calendar. Wait until the new version has a normal week of calls, then compare again.
Changed may list a prompt edit, a tool change and a new voice. The comparison cannot say which one moved a number. Ship one change per version when you can.
With several rows checked at once, one will now and then move by chance alone. A single guardrail moving, with nothing else, is weak evidence. Read its calls before acting.
A resolved issue regresses only when it comes back clearly more often than before it was resolved, not on a single call. One call that shows it again leaves it resolved, with “Seen again on” and the date under the title. Open the issue and check By version for the versions of the calls that show it: if they are on the new version, the fix did not hold; if they are on an older one, the change may not be live for every caller yet.

​
Get the same comparison over the API

The SignalMD API serves version comparisons to your own tools, such as a release script that checks a change after it ships. You need an API key; see Authentication.
Terminal
Compare two versions takes any two versions, not only neighbours. It returns the configuration changes between them (changes is null when either configuration was not captured, which is not the same as nothing changed) and the comparison of production calls in comparison. Its result holds the conclusion with the app’s wording in conclusionLabel, the calls on each side, a row per metric, and the narrative the app prints under the table. isRandomized is always false: a version comparison is never a controlled experiment. To list comparisons, newest first, use List version comparisons, and fetch one with Get a version comparison. Get the agent ID from List agents. From an AI assistant, the MCP server offers the same through its compare_versions tool.

​
Next steps

Test a fix

Get an answer before callers meet the change.

Versions

What each metric and verdict means.