How to Configure Crawling for Your Site
Different websites need different crawling approaches. This guide gives you ready-to-use recipes for the most common site types. Pick the one that matches your site, apply the settings, and you’re done.
Not sure what type of site you have? Leave the wizard on AI-assisted configuration mode and it picks settings for you. To tune things by hand, switch to Manual mode, choose Sitemap Discovery, and open Advanced Options. Only change things if your content comes back empty or noisy.
Tip: try a Site Preset first. In Manual mode, Advanced Options opens with a Site Preset picker (Standard Website, SPA / JavaScript App, Confluence / Wiki, Static Site / Blog). Picking one fills in all settings for that site type, and you can fine-tune from there. The recipes below match these presets.
For a complete list of every setting and what it does, see Crawling Settings Reference.
Quick decision table
Section titled “Quick decision table”| Your website type | Discovery Mode | Preset | Key settings | Speed |
|---|---|---|---|---|
| Static docs (Docusaurus, MkDocs, Hugo, Jekyll) | Sitemap Discovery | Static Site / Blog | Simple Mode ON | Fastest |
| WordPress / CMS blog | Sitemap Discovery | Standard Website | Reader Mode ON | Fast |
| React / Vue / Angular SPA | Link Crawling | SPA / JavaScript App | Reader Mode ON, Advanced Link Detection ON | Slower |
| Confluence / Wiki | Link Crawling | Confluence / Wiki | Reader Mode ON, Content Area Selector, Concurrency 2-3 | Slowest |
| Any site without a sitemap | Link Crawling | Adjust per site type above | Adjust per site type above | Varies |
| Large site (1,000+ pages) | Sitemap Discovery (if sitemap exists) | Standard Website | Tune Max Pages and Concurrency | Varies |
Recipe: Static documentation site
Section titled “Recipe: Static documentation site”Examples: Docusaurus, MkDocs, ReadTheDocs, Hugo, Jekyll
| Setting | Value |
|---|---|
| Discovery Mode | Sitemap Discovery |
| Site Preset | Static Site / Blog |
| Simple Mode | ON |
| Reader Mode | OFF |
| Concurrency | 6-8 |
Why this works: These tools generate clean HTML with sitemaps. No browser rendering or content cleaning needed. This is the fastest possible configuration.
Recipe: WordPress or CMS blog
Section titled “Recipe: WordPress or CMS blog”Examples: WordPress, Ghost, Webflow, Squarespace
| Setting | Value |
|---|---|
| Discovery Mode | Sitemap Discovery |
| Site Preset | Standard Website |
| Simple Mode | OFF |
| Reader Mode | ON |
| Concurrency | 5-8 |
Why this works: CMS sites usually have sitemaps but include sidebars, related posts, ads, and navigation that pollute your knowledge base. Reader Mode strips all of that out and keeps just the article content.
Recipe: React / Vue / Angular SPA
Section titled “Recipe: React / Vue / Angular SPA”Examples: Custom docs portals, Storybook, SPA-based knowledge bases
| Setting | Value |
|---|---|
| Discovery Mode | Link Crawling |
| Site Preset | SPA / JavaScript App |
| Reader Mode | ON |
| Advanced Link Detection | ON |
| Wait Time for Page Content (seconds) | 2-3 seconds |
| Max Depth | 10-15 |
| Concurrency | 5 |
Why this works: Modern web apps (SPAs) build pages dynamically with JavaScript and often hide navigation links from standard crawlers. Advanced Link Detection finds these hidden links, and the browser renders the full page content. Many chatbot tools can’t handle these sites at all, ChatbotIQ detects them automatically and adapts.
Recipe: Confluence or wiki
Section titled “Recipe: Confluence or wiki”Examples: Confluence Cloud, Confluence Server, MediaWiki
| Setting | Value |
|---|---|
| Discovery Mode | Link Crawling |
| Site Preset | Confluence / Wiki |
| Reader Mode | ON |
| Content Area Selector | #main-content (ask your developer if unsure) |
| Wait Time for Page Content (seconds) | 3 seconds |
| Concurrency | 2-3 |
| Max Depth | 15-20 |
Why this works: Confluence loads content in the background after the page appears, so it needs extra wait time. Low concurrency prevents Confluence from blocking the crawler (it’s aggressive about rate limiting). The Content Area Selector targets the article area and skips Confluence’s menus and sidebar.
Important: High concurrency on Confluence will trigger 429 rate-limit errors. Keep it at 2-3.
Recipe: Site without a sitemap
Section titled “Recipe: Site without a sitemap”Examples: Legacy sites, hand-coded HTML, miscellaneous web apps
| Setting | Value |
|---|---|
| Discovery Mode | Link Crawling |
| Simple Mode | ON if static HTML, OFF if JavaScript |
| Reader Mode | ON |
| Max Pages | Start with 100, increase after review |
| Max Depth | 5-10 |
| Concurrency | 5-8 |
Why this works: Without a sitemap, Link Crawling is your only option, it discovers pages by following links. Start with a smaller page limit to verify the crawler is finding the right pages before scaling up.
Recipe: Large site (1,000+ pages)
Section titled “Recipe: Large site (1,000+ pages)”Examples: Enterprise documentation, large knowledge bases
| Setting | Value |
|---|---|
| Discovery Mode | Sitemap Discovery (if sitemap exists) |
| Site Preset | Standard Website |
| Reader Mode | ON (if needed) |
| Max Pages | Your plan’s page limit |
| Concurrency | 6-8 |
Why this works: Sitemap Discovery handles large sites efficiently because discovery is instant via the sitemap. High concurrency speeds up the crawl phase. If the site has no sitemap, use Link Crawling but expect longer discovery times.
Speed vs. coverage tradeoffs
Section titled “Speed vs. coverage tradeoffs”Fastest to slowest
Section titled “Fastest to slowest”| Configuration | Speed | Best for |
|---|---|---|
| Sitemap Discovery + Simple Mode | ~100+ pages/min | Static HTML with sitemap |
| Sitemap Discovery + Browser (default) | ~20-30 pages/min | CMS/WordPress with sitemap |
| Link Crawling + Browser | ~10-20 pages/min | SPAs, no-sitemap sites |
| Link Crawling + Reader Mode | ~5-15 pages/min | JS-heavy sites needing clean extraction |
| Link Crawling + Reader Mode + Low Concurrency | ~2-5 pages/min | Confluence, rate-limited sites |
Best all-rounder
Section titled “Best all-rounder”Sitemap Discovery + Reader Mode ON covers ~70% of websites. It works for any site with a sitemap and server-rendered content.
If you need maximum coverage, Link Crawling + Reader Mode + Advanced Link Detection handles ~95% of sites, just slower.
When to trade speed for coverage
Section titled “When to trade speed for coverage”- Start fast: Use Sitemap Discovery with defaults. If pages come back with good content, you’re done.
- Escalate if needed: If content is missing or noisy, enable Reader Mode. If pages are missing entirely, switch to Link Crawling.
- Only go slow when necessary: Low concurrency and a high Wait Time for Page Content are only needed for specific sites like Confluence.
If your crawl isn’t working as expected, see the crawling troubleshooting section for common issues and fixes.
Related
Section titled “Related”- Crawling Settings Reference - every setting with type, default, and description
- How Web Crawling Works - understand Sitemap Discovery vs. Link Crawling
- Add a Website Source - the basics of adding a source
- Keep Content Up to Date - re-crawling and refresh strategies