How to Keep Content Up to Date

Your website changes over time, and your bot should reflect those changes. Here’s how to keep your knowledge base current.

Refresh a source

Go to Knowledge Base in the sidebar.
Find the source you want to update.
Click Refresh.

ChatbotIQ re-crawls all the pages using the same settings as the original crawl. When it finishes, you’ll see a crawl report.

Read the crawl report

After a refresh, the report shows:

Status	Meaning
Changed	Content on this page was updated since the last crawl. The new version is now indexed.
Unchanged	Content is the same as before. No re-indexing needed.
Failed	The page couldn’t be accessed. See troubleshooting below.
New	A page that wasn’t in the previous crawl (discovered via sitemap update or new links).

Tip: Check the “Changed” pages to make sure the new content looks correct. ChatbotIQ has degradation protection — if a page’s content shrunk significantly, the update is blocked to prevent data loss.

Retry failed pages

If some pages failed during a crawl:

Open the crawl report.
Click Retry Failed to re-crawl only the failed pages.

Common reasons for failure:

The page was temporarily unavailable (server error).
The URL was removed or redirected.
Rate limiting by the source website (lower concurrency and try again).

When to refresh

After updating your website — if you published new content or updated existing pages, refresh immediately.
On a regular schedule — weekly or bi-weekly is good for most sites.
After knowledge gap analysis — if you added content to fill gaps, refresh to index it.

Delete outdated content

If pages were removed from your website:

Go to the source in Knowledge Base.
View the indexed pages.
Delete individual pages that no longer exist, or delete the entire source if it’s no longer relevant.

Your page count decreases automatically.

Add a Website Source — source management basics
Configure Crawling for Your Site — crawling settings
Find and Fix Knowledge Gaps — discover what’s missing