Plan requirement
| Subscription | Suite Team or higher, Knowledge Professional or higher |
| Access | Admin |
The crawl completed and nothing appeared in search. Four documented causes account for nearly all of it, and it is worth saying at the outset that JavaScript is not one of them: the crawler switches to a browser mode for pages that need rendering.
Why did the crawler find nothing?
- The crawl completes but indexes zero pages.
- Far fewer pages are indexed than the site has.
- Content you can see in a browser does not appear in help centre search.
Start with the email
When a crawl fails, the crawler's owner receives an email with troubleshooting tips. Read that before anything else, because a crawl that reports success and one that failed silently need different investigations.
One: the scope excludes everything
The crawl reports success here, because the crawler did exactly what it was told.
Check Sync only specified URLs first. With it on, the crawler takes the listed addresses and nothing else, so a list of section landing pages yields those pages alone. Then check Limit sync to specific patterns and Exclude specific patterns from syncing under Advanced settings. Open the site, copy a real address, and hold it against your patterns rather than reading them for plausibility.
Two: the crawler cannot reach the pages
Anything behind a login is invisible to it, and if the pages are not public, crawling is not the mechanism to use at all.
Where the pages are public but a firewall sits in front, allow the user agent Zendesk/External-Content. That is the specific thing to ask for, and it is quicker than describing the symptom to whoever runs the site.
Three: the site is served with gzip compression
The crawler does not work with sites that use gzip file compression encoding, and content from those sites will not appear in search results. This one produces a clean, complete, empty crawl, so it is worth ruling out early even though it looks like nothing to do with your settings.
Four: the content is out of range
A crawler reaches four levels deep, or the whole site if that comes first, and takes up to 500 URLs. Documentation sitting five clicks from your starting address is not missing, it is out of range.
The fix is another crawler pointed further in, rather than a wider one at the root. You can have 50 per account.
Still nothing
Note which of the four you have ruled out and how, then contact pluscloud support with the site address, your URL patterns and the crawl email. That is enough to reproduce it without going back and forth.
Comments
0 comments
Article is closed for comments.