Adding a Website

This article shows you how to add a website to a knowledge base as a data source, whether that is a single page or a whole section of a site, so that an AI agent can answer from its content. For what a knowledge base is, see Knowledge Base; if the project has none yet, start with Creating a Knowledge Base. To upload files instead, see Adding Documents.

Adding the website

The example in this article adds the support section of a company site, https://www.example.com/support, and nothing else from that site.

1. Open Knowledge Bases from the project menu and select the knowledge base. It opens on the Data Source section.
2. Select + New Data Source. On Add New Data Source, Websites is already selected.
3. Fill in the form as described below, then select Save.

1. Main URL: The address to start from. It begins with http:// or https:// and must be a public website. An address that cannot be reached at all is refused. The Main URL cannot be changed after you save. For the example: https://www.example.com/support.

2. + Add tag: Tags the website. The tags apply to every page crawled from it. The tags must first be defined on the knowledge base; see Knowledge Base Tags.

3. Crawl just this page / Multiple pages: Crawl just this page (the default) indexes only the exact URL and does not follow its links. Multiple pages follows links to related pages and adds the crawl settings described in the next section. For the example, choose Multiple pages.

4. Crawler Type: How pages are fetched. Default: Adaptive switching between browser and raw HTTP.

5. Content Extraction Method: What is kept from each crawled page. Default: Readable Text.

Crawling more than one page

Choosing Multiple pages adds five settings to the form.

1. Include URLs: The pages to crawl, one pattern per line, in glob style such as https://www.sample.com/contact-us/** or /about-us. For the example: https://www.example.com/support/**.

2. Exclude URLs: The pages to leave out, in the same format. The Main URL is always crawled, whatever the two lists say.

3. Maximum Pages to Crawl: The most pages the crawl collects. Default: 5000.

4. Maximum Crawl Depth: How many links away from the Main URL the crawler follows. Default: 10.

5. Allow Crawling External Websites: Whether links to other domains are followed. Default: off.

The finished example looks like this:

Following the crawl

After you save, the website appears in the Websites table of the Data Source section. While it is being crawled, its Train Status reads Processing followed by the current stage, and hovering it shows how many pages have been crawled so far. The page refreshes itself while anything is processing. The crawl ends at Indexed or Failed. The full list of statuses, and what makes a knowledge base Ready, is in Adding Documents.

⚠️ Warning: A single Failed source keeps the whole knowledge base Not ready. Until it is Ready, a flow's Knowledge Retrieval or RAG Agent node cannot select it, and the project cannot be published. Re-run the failed website with Refresh Index, or delete it.

Once the website is Indexed, the magnifier icon on its row opens Indexed URLs, the list of pages the crawl actually picked up. Check it after the first crawl of a multi-page site: a page missing from this list is not in the knowledge base.

Refreshing a website

A website is never re-crawled automatically. When the site changes, the knowledge base keeps the pages as they were at the last crawl. To bring it up to date:

1. Select the pencil (Edit) icon on the website's row.
2. Change any settings you need, or none. Every setting except the Main URL can be edited.
3. Select Save and confirm. The website is re-crawled and re-indexed with the saved settings.

Refresh Index does not do this. It re-runs only sources that failed or whose knowledge base settings changed after they were indexed, so it does not crawl an Indexed website again.

To crawl a different address, add it as a new website and delete the old one with the trash icon on its row.

Limits