URL sources let your agent learn from your live website. Point it at a page (or your whole site) and Chatzuri crawls and indexes the content.
How to add a URL
- Open Sources → URLs
- Paste the URL into the input box
- Choose Single page or Crawl site
- Click Add
Single page vs full crawl
- Single page — fetches just that one URL. Use when you want one specific document (a pricing page, an FAQ).
- Crawl site — follows links from the starting page and indexes every page on the same domain. Use when you want broad coverage of a knowledge base or blog.
Tip
Crawls can pick up navigation, footers, and cookie banners. After a crawl, scan the indexed pages and remove any that are pure boilerplate.
Refreshing URL content
Webpages change. To pull the latest version of a page, find it in the URL list and click Re-crawl. The old indexed copy is replaced with the new one.
For sites that change often (pricing, blog), re-crawl on a schedule — set a recurring Agent Task to do this automatically.
What pages should I add?
- Pricing and plans
- Help / FAQ pages
- Product feature pages
- Comparison pages
- Documentation
What to avoid:
- Login-walled pages — the crawler can't see them
- Heavy JavaScript apps that need a browser to render
- Marketing pages that are mostly imagery and animation
Restricting the crawl
For "Crawl site", you can set a maximum number of pages and an allowlist of URL patterns. This keeps crawls focused and predictable.
Heads up
Always check whether the site you're crawling allows it. Crawl your own properties freely; for third-party sites, check their terms of service.