Skip to content
ChatbotIQ

How to Add a Website Source

This guide walks you through adding a website to your knowledge base so your bot can answer questions about its content.

Looking for other source types? See Source Types for PDFs, Q&A pairs, spreadsheets, and more.

Add a new source

Section titled “Add a new source”
  1. Go to Knowledge Base in the sidebar.
  2. Click ”+ Add Source”.
  3. Choose Website as the source type.
  4. Enter the base URL of the site you want to crawl (e.g., https://docs.example.com).
  5. ChatbotIQ automatically discovers the pages on your site. This can take a few seconds to a few minutes depending on the site size.
  6. Review the discovered pages. You can select or deselect individual pages before crawling.
  7. Click Start Crawl to begin downloading and indexing the content.

First time? Start with a smaller site (under 100 pages) to see how the crawler handles your content. You can always add more sources later.

Understand source statuses

Section titled “Understand source statuses”

Each source in your knowledge base shows its current state:

StatusWhat it means
DiscoveringChatbotIQ is scanning the site for pages (reading the sitemap or following links).
CrawlingPages are being downloaded. You’ll see progress like “12/47 pages crawled”.
IndexingDownloaded content is being converted into embeddings so your bot can search it.
ReadyEverything is indexed. Your bot can now use this source to answer questions.
FailedSomething went wrong. You can retry failed pages without re-crawling everything.

Manage your sources

Section titled “Manage your sources”

From the Knowledge Base page, you can:

  • Refresh — re-crawl an existing source to pick up content changes. ChatbotIQ shows you which pages changed, which stayed the same, and which failed.
  • Cancel — stop a running crawl or discovery at any time.
  • Retry failed pages — if some pages failed during a crawl, retry just those pages without re-crawling the entire site.
  • View report — after a crawl completes, see a detailed breakdown: every page with its status, the number of content chunks created, and page-by-page details.
  • Delete — remove a source and all its indexed content. Your page count decreases automatically.

Tips for better results

Section titled “Tips for better results”
  • Enable Reader Mode if your pages include navigation bars, sidebars, or footer content that shouldn’t be in your knowledge base. See Configure Crawling for Your Site.
  • Check the crawl report after your first crawl. Look for pages with very little content — they might need different crawl settings.
  • Use the Playground debug panel to test how well your bot retrieves content from this source.
Section titled “Related”