Plan requirement
| Subscription | Suite Growth or higher, Guide Professional or higher |
| Access | Admin |
Filters decide which pages of a site the crawler takes and which it leaves. Set them before the first run, not after.
Set them
- Open the crawler's configuration in Knowledge.
- Add the paths to include, and the paths to exclude.
- Save and run the crawler.
- Search for something you know is on an excluded page, and confirm it is absent.
Include narrowly rather than exclude broadly
Pointing the crawler at a documentation path and letting everything else alone is a rule that keeps working when the site grows. Crawling everything and excluding the parts you thought of is a rule that quietly fails the next time somebody adds a section.
What usually needs excluding
- Blog and news, where dated posts contradict current documentation.
- Marketing pages, which answer nothing a customer with a problem is asking.
- Archives and old versions, which are the most damaging of the three because they look authoritative.
Check what came back
After the first run, look at what was indexed rather than assuming the filters did what you meant. A wrong pattern usually shows up as either far more pages than expected or almost none.
Revisit it when the site changes
Filters are written against a structure. A site redesign that moves paths turns a correct filter into one that silently indexes the wrong half of the site.
Comments
0 comments
Article is closed for comments.