Testing an AI agent with test sets

A test set sends the same customer messages to your AI agent on every run and compares the answers with the run before, so you see what a change did.

A test set sends the same customer messages to your AI agent on every run and compares the answers with the run before, so you see what a change did. You can add a check per message, and the results show which messages passed.

Create a test set

1. Start a test of the AI agent with Test AI agent, or with Test dialogue, Test branch or Test procedure, and open the preview. See Getting started with AI Agent Preview.
2. Click Test sets in the toolbar, below Copy link to conversation log.
3. Click New test set and give it a name, for example the topic it covers.
4. Add customer messages with Add message, or paste several at once with Add several, one message per line.
5. Add checks where you want them. Answer contains looks for a text in the answer, its button labels and card titles, not case-sensitive. Handover checks whether the AI agent hands the conversation to an agent: Expected, Not expected or Don't check.
6. Click Save.

Test sets belong to one AI agent. Edit changes a set later; Delete, clicked twice, removes the set and its runs.

Run it

1. Click Run next to the set. The extension starts a new test conversation, as Restart conversation does.
2. It sends the messages one by one, each after the AI agent has finished answering the one before, and waits up to 60 seconds per message. The toolbar button shows how far it is, and you can close the panel in the meantime.
3. When the run ends, a message says how many passed. Stop ends a run early.

While a set runs, you cannot type or click in the preview: that would mix in with the run. A test set only runs when you click Run. It cannot run while you show a conversation from the conversation logs, or without the test widget.

Read the results

Results shows the last run. Per message you see the status, the checks, the answer with its buttons and sources, and the time to the first answer; hover over that time to see the time to the last message too.

• Passed: every check passed. Answered: the message had no checks and got an answer.
• Failed: a check failed. No answer in time, No answer and Not sent say what went wrong.
• Changed: the answer differs from the previous run, with the removed words struck through and the added words highlighted. Same answer as the previous run. when nothing changed.

The menu Run at the top opens an earlier run; the last 10 runs of each set are kept. The link to the conversation logs opens the run's test conversation there. Run again starts a new run.

Good to know

• A run is a test conversation like any other. It shows in the conversation logs of the AI agent, as described in Reviewing conversation transcripts.
• The comparison looks at the words of the answer, not at its meaning. When the wording of an answer may differ between runs, put the facts that matter in Answer contains.
• Test sets and their runs are stored in this browser only, per AI agent, for the 50 AI agents you edited most recently. Colleagues do not see them. What else the extension stores is in What the extension can access and store.

See also

• Testing tools in the preview
• Reading the conversation log
• The AI Agent Inspector panel in Chrome DevTools
• Testing an AI agent before you publish it