Planning: turn your acceptance criteria into test sets

Lesson 4 of 11 · Planning. You turn your acceptance criteria into test sets with checks, and run them in the preview from lesson 1.

Lesson 4 of 11 · Planning. You turn your acceptance criteria into test sets with checks, and run them in the preview from lesson 1.

The scenario

Before your AI agent goes live, your team or your customer wants proof that it handles the questions you agreed on. You have a list: what each answer must contain, and when the AI agent must hand the customer to a human agent. You don't want to type every question again after each change.

Goal: one test set per topic, with checks, that passes before go-live.

What you need

• Your acceptance criteria: customer messages, a word or phrase each answer must contain, and whether a handover to an agent is expected.

Steps

1. Write the criteria down first, one row per customer message. Use your own questions and facts. For example:

   Customer message	Answer contains	Handover
   Where is my order?	order number	Don't check
   Can I return an item?	30 days	Not expected
   Do you sell gift cards?	gift card	Not expected
   I want to talk to a person		Expected

2. Start a test with Test AI agent, choose Show as Web Widget, and click Test sets in the toolbar. The panel is described in Testing an AI agent with test sets; Zendesk's own test options are in Testing an AI agent before you publish it.
3. Click New test set and fill in the Name, for example Orders and returns.
4. Under Customer messages, click Add several, paste the messages one per line and click Add messages.

5. For each message, fill in Answer contains and choose Handover: Expected, Not expected or Don't check. Then click Save.

   

6. Click Run. The extension starts a new test conversation and sends the messages one by one, each after your AI agent has finished answering. While the set runs, the preview is read-only and the toolbar shows which message it is at. Stop ends the run.

7. Read the results. Each message shows Passed, Failed, Answered when it has no checks, or No answer in time, with your AI agent's answer, its buttons and sources, and the response time. Open in conversation logs opens the conversation of the run. To read it in the preview, see Viewing a logged conversation in the preview.

   

8. Make a set for each topic in your criteria, for example Payments, or Opening hours. When every set passes, your AI agent meets the criteria you agreed.

Why it matters

• The results list every message with its outcome, so your team or customer signs off on one list instead of a few screenshots of single tests.
• Test sets are stored in this browser, per AI agent. A colleague on another computer does not see them. Share a run through its conversation log link, or with a screenshot of the results. What the extension keeps is in What the extension can access and store.
• The last 10 runs of each set are kept. In lesson 10 you compare a new run with this one.
• Write the check as a fact, such as 30 days, rather than a full sentence. A rephrased answer then still passes, as long as the fact is there.