Why an AI agent does not understand your knowledge base

An AI agent giving vague answers is usually reading a knowledge base written for people who already know the answer. Rule out the configuration first.

An AI agent giving vague answers is usually reading a knowledge base written for people who already know the answer. Before you rewrite anything, rule out the three configuration reasons that produce the same symptom.

Symptoms

• Answers that are broadly on topic and never specific.
• An article you know is correct is never used.
• The agent answers in one language and cites articles in another.
• A generative reply fails with a technical error rather than an answer.

First: is there anything to read?

At least one knowledge source has to be connected. With none connected, a generative replies block or a procedure asked for a knowledge reply produces a technical error, not a poor answer. That is a distinct symptom and it points straight at the configuration.

Check the other end of the same question while you are there. Zendesk says too many sources can in some cases reduce accuracy and increase latency, and recommends keeping the number within a reasonable limit without naming one. Vague-but-on-topic answers from an agent connected to everything you own are worth reading in that light: the fix may be removing a source rather than rewriting an article.

Second: is a search rule excluding it?

Search rules decide which sources, or which parts of them, the agent may use. A rule filtering on Article, Locale or Labels can quietly put your best article out of scope while everything looks connected.

Check what the rule that applies to that conversation covers before concluding the article is at fault.

Third: is it a locale problem?

Where knowledge sources are scoped per locale, a connection points at one language's article set, identified by the locale in the URL such as /hc/en-us. A Dutch article then is not in what the agent can see, however good it is.

This is also the usual explanation for an agent that answers in one language while citing an article in another.

Then: the four things that break AI answers

Once the configuration is ruled out, the causes are in the content, and they are all writing problems.

1. Articles covering several subjects. An article about installing, configuring and troubleshooting a feature is three answers in one. Asked about any of them, an AI returns a blend, which is right about nothing in particular.
2. The answer buried after the context. Articles that open with background and reach the answer in paragraph four. AI weights the beginning, so it summarises your preamble.
3. Contradictions. Two articles saying different things, usually because one is old. A person spots the newer one; an AI has no way to know which to believe and may quote either.
4. Knowledge that is not written down. The most common of all. An AI can only answer from what exists, and where the real answer lives in an agent's head, no amount of configuration produces it.

The good news

Everything that makes a knowledge base work for AI also makes it work for people: one subject per article, the answer first, no contradictions, current content. This is not a separate project. It is the writing you should have been doing, with a new reason to do it.

Where to start

Look at the questions your AI agent handles badly. Each one is a missing article, an article covering too much, a contradiction, or a piece of configuration keeping it out of scope. All four are fixable in an afternoon each.

See also

• How to write an article that AI can use
• One subject per article: why that matters for AI
• My AI agent solves very little: where to start