Creating manual rating categories

Writing a category reviewers can apply consistently. The description is marked optional, and it is the part that decides whether it works.

Writing a category reviewers can apply consistently. The description is marked optional, and it is the part that decides whether it works.

Create one

1. In Quality assurance, click your profile icon, then Settings. In the sidebar under Account, click Scorecards.
2. Click Create at the top and select Manual category.
3. Enter a Category name, which is the question the reviewer answers, and a Description for reviewers. The description is marked optional and it is the part that decides whether the category works.
4. Click Create category, then add it to the scorecards that should use it. Until you do that, nobody can score anything with it.
5. Have two reviewers score the same conversation with it, and rewrite the description if they disagree.

Creating the category asks for those two things and nothing else. The scale, the weight and whether it is critical belong to the scorecard the category sits on, which is also why the same category can carry a different weight in two workspaces.

Ask one thing

"Was the reply accurate and complete?" is two questions. A reviewer facing a reply that was accurate and incomplete has no honest score to give.

Split it, or accept that the category will be applied inconsistently.

Write it as a question about the conversation

Not about the agent. "Did the reply address the customer's actual question" is checkable. "Does this agent understand our products" is a judgement about a person from one exchange.

The guidance is what makes it work

Two or three lines per score point, with an example from a real conversation. Without it, each reviewer builds their own private standard and the scores stop meaning one thing.

This is the part people skip and the part that determines whether the category survives.

Keep the scale simple

Three points is usually enough: below, meets, exceeds. A five or ten point scale gives an illusion of precision and produces clustering in the middle.

Test with two reviewers

Same conversation, scored independently. If they disagree, rewrite the guidance rather than assuming they will converge with practice.

Rewriting is safe, so there is no reason to hesitate: editing a category's name and description does not affect its reporting, and changes apply to future conversations only. So improving the wording does not rewrite history or invalidate scores already given.

Retire a category by deactivating it

Deactivated categories remain available in reporting and filtering, and dashboards keep showing accurate historical scores and trends.

Deleting does not. It is permanent and erases all category data from reporting, so a year of scores on that category disappears from every dashboard that used it. Deactivate unless you actively want the history gone.

Neither is labelled the way you would search for it. The options menu beside a category offers Mark as active, Mark as inactive and Delete.

Allow not applicable

Some categories do not apply to some conversations. Without that option, reviewers score something arbitrary, and the average is polluted.

See also

• What rating categories are
• Viewing and managing rating categories
• Root causes for review scores