Plan requirement
| Subscription | Suite Team or higher, Knowledge Professional or higher |
| Access | Admin |
Filters decide which pages of a site the crawler takes and which it leaves. Set them before the first run, not after.
Set them
- In Knowledge, click Settings in the sidebar, then Search settings. Under Crawlers, click Manage, then Add Crawler.
- Give it the Start URLs: a domain, a sitemap path such as
example.com/sitemap.xml, or individual pages. Up to five, with + Add another. - Add the paths to include and the paths to exclude, one URL per line.
- Save and run the crawler.
- Search for something you know is on an excluded page, and confirm it is absent.
You can run several crawlers, on the same site or different ones, which is often tidier than bending one crawler around every exception. A sitemap as the start URL is worth preferring where the site has one: it tells the crawler what the site owner considers a page, rather than leaving it to follow whatever links exist.
Include narrowly rather than exclude broadly
Pointing the crawler at a documentation path and letting everything else alone is a rule that keeps working when the site grows. Crawling everything and excluding the parts you thought of is a rule that quietly fails the next time somebody adds a section.
What usually needs excluding
- Blog and news, where dated posts contradict current documentation.
- Marketing pages, which answer nothing a customer with a problem is asking.
- Archives and old versions, which are the most damaging of the three because they look authoritative.
Check what came back
After the first run, look at what was indexed rather than assuming the filters did what you meant. A wrong pattern usually shows up as either far more pages than expected or almost none.
Revisit it when the site changes
Filters are written against a structure. A site redesign that moves paths turns a correct filter into one that silently indexes the wrong half of the site.
Comments
0 comments
Article is closed for comments.