Plan requirement
| Subscription | Suite Team or higher, Knowledge Professional or higher |
| Access | Admin |
The crawler indexes public pages you point it at, which makes it the route into federated search that needs no developers. It handles pages that need JavaScript to render, switching to a browser mode when it detects them, so a modern site is not in itself a problem.
Set one up
- In Knowledge admin, open Settings, then Search settings.
- Under Crawlers, click Manage, then Add crawler, then Continue.
- On the Source and URLs tab, give it a Connection name and enter the URLs to start from.
- Set the Syncing frequency.
- Decide on Sync only specified URLs, described below.
- Run it, then search for something you know is on the site.
The limits worth knowing before you plan
| Crawlers per account | 50 |
| URLs per crawler | 500 |
| Depth | Four levels, or the whole site, whichever comes first |
| Syncing frequency | Weekly by default. Also Daily, Monthly, or Never for a one-off |
A large site can therefore need several crawlers pointed at different sections rather than one aimed at the root.
Sync only specified URLs decides the whole shape
Leave it off and the crawler indexes the pages you listed and every page they link to, which is how marketing pages, the careers page and old campaign microsites end up in help centre search. Turn it on and it takes exactly the addresses you entered and nothing else.
Start narrow. It is easier to widen a crawler than to explain why a job advert is answering a support query.
Scope it further under Advanced settings
There you will find Look for more URLs while syncing, plus Limit sync to specific patterns and Exclude specific patterns from syncing. Patterns are the practical middle ground between one page and a whole site, and checking the result matters more than getting them right first time.
The language comes from the page, not from a setting
There is nothing to choose. The crawler reads the lang attribute on the <html> tag first, then the Content-Language header, then a <meta> tag, and only if none of those is there does it guess from the text itself. The crawler stores a record under the base language code and not a regional variant: a page marked lang="en-gb" is indexed as en. It checks that code against a standard list of supported languages, not against the locales you happen to have enabled.
When nothing matches a locale your help centre supports, the crawl logs a Locale not detected error and it lands in the error report sent to whoever owns the crawler. Content arriving under the wrong language is therefore fixed on the crawled site's markup, not here.
It only sees public pages
Anything behind a login is invisible to it. That is a limitation and a safeguard at once: it cannot index something private by accident. If your firewall is strict, allow the user agent Zendesk/External-Content so the crawl is not blocked.
One more exclusion is worth checking up front: the crawler does not work with sites served using gzip compression encoding, and content from those sites will not appear at all.
Two things on your site that it ignores
A changefreq value in your sitemap does not affect when the crawler runs, and a crawl-delay directive in robots.txt is not respected. The schedule is the setting in Zendesk and nothing else, which is worth telling whoever maintains the site before they try to steer it from their end.
If a run cannot finish, because the site is down for instance, what was indexed last time stays searchable and Zendesk tries again later. A failed crawl does not empty your results.
Check it after site changes
A crawler is tied to the structure of the site it reads, so a redesign that changes addresses leaves it indexing nothing or indexing the wrong pages. If a crawl fails outright the crawler's owner gets an email with troubleshooting tips, but a crawl that succeeds against a changed site tells you nothing, which is why a redesign is worth pairing with a search you know should return a result.
Comments
0 comments
Article is closed for comments.