Debugging: re-run the test sets after a change

Lesson 10 of 11 · Debugging. After a change, you run the test sets from lesson 4 again and compare each answer with the previous run.

Lesson 10 of 11 · Debugging. After a change, you run the test sets from lesson 4 again and compare each answer with the previous run.

The scenario

The article behind an odd answer has been fixed, as in lesson 7, and a colleague changed how your AI agent looks up an order. Before you tell your team it is done, you want to know that the customer's question now gets the right answer, and that nothing else changed by accident.

Goal: confirm the fix and catch side effects, with a run you can compare to the one before.

What you need

• The test sets from lesson 4, in the same Chrome where you made them.

Steps

1. Open a test of your AI agent in the preview and click Test sets in the toolbar. The panel is described in Testing an AI agent with test sets.
2. Click Edit next to the set for the topic you changed. Add the customer's question from the ticket, with the correct fact under Answer contains, if you have not done so in lesson 7, and click Save.
3. Open the results of the set and click Run again. The run starts a new test conversation, as before.

4. Compare. An answer that differs from the previous run is marked Changed, with the removed and added words highlighted. An answer that did not change says Same answer as the previous run., followed by the old outcome when that changed, such as Was: Failed.

   

5. Check every Changed mark, not only the failed messages. A changed answer to a question you did not touch is the side effect you are looking for.
6. To compare with an older run, pick it in the Run list. The last 10 runs of each set are kept.
7. For a message that fails, click Open in conversation logs and trace it as in lesson 7, with Viewing a logged conversation in the preview and Using the inspector.
8. Run every set after each change, before you tell the team it is done.

Why it matters

• A change in one article or procedure can change answers on other topics. Running all sets shows it, message by message, in the same list as before, so you find it before your customers do.
• If your AI agent writes its replies with generative AI, the wording can differ between runs. A Changed mark then shows the new wording, and Answer contains tells you whether the fact is still there.
• A handover that appears or disappears also counts as a change: the result says The handover changed.