Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI

Artificial Intelligence #

When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest. Amazon Bedrock Knowledge Bases is the fully managed RAG capability of Amazon Bedrock. With Amazon Bedrock Managed Knowledge Base, you connect your data sources, and Amazon Bedrock manages ingestion, storage, indexing, and retrieval for you. You don’t provision or scale a vector store, and embedding and reranking use service-managed models by default.

For web content, start with the Amazon Bedrock Web Crawler data source connector for managed knowledge bases. You give it the seed URLs or sitemaps that you’re authorized to crawl, and it crawls within the scope and rate limits that you set. Scheduled syncs keep your knowledge base current as pages change, and the connector can wait for content that loads with JavaScript. There’s no crawler for you to build, host, or operate, so this connector is the starting point for most public website content. Some sources need more control over what reaches the knowledge base. You might want to index only the main content region of each page, remove navigation and other boilerplate, or attach your own metadata attributes to every document. For that class of source material, you can extend the reach of your knowledge base with a preparation step built with Crawl4AI, an open source Python crawler. The step writes Markdown and a metadata sidecar for each page to Amazon Simple Storage Service (Amazon S3), and the knowledge base ingests both through its Amazon S3 data source.

In this post, we show you how to choose the appropriate web ingestion path and implement custom preparation for content that needs additional processing. You then start ingestion into a managed knowledge base and validate retrieval quality. The companion aws-samples repository provides the complete workflow as a sample implementation.

This is batch ingestion of known web properties, not real-time web access at inference time such as Web Grounding with Amazon Nova.

Solution overview #

Start with the Amazon Bedrock Web Crawler when it can reach and index the pages you need. It respects robots.txt, and after the first full crawl, each sync captures only added, updated, and deleted content.

Use the Crawl4AI preparation path when public source material needs one or more of the following before ingestion:

• Structured Markdown preparation : UseDefaultMarkdownGenerator with a pruning content filter to reduce navigation and other boilerplate while preserving headings, lists, and tables. Because pruning is heuristic, review representative output from your sites.
• Bounded, targeted extraction : Extract only the page region that an optional CSS selector matches. Constrain breadth-first traversal by seed origin, page/path/host scope, depth and page caps, and URL patterns.
• Custom metadata : Attach source URLs, crawl timestamps, content type, and your own attributes to each document, so that you can filter retrieval results on them.

Because managed and customer-managed knowledge bases both support Amazon S3 data sources, the Crawl4AI path works with either type. This post uses a managed knowledge base and notes where a customer-managed knowledge base differs.

The rest of this post follows the companion sample. Treat its architecture and defaults as a starting point, not as requirements of Amazon Bedrock Knowledge Bases.

By default, the sample runs the AWS Fargate task in a public subnet, where it reaches your configured websites through an internet gateway. Crawl4AI’s role ends once it has prepared the source documents and metadata sidecars. From ingestion onward, Amazon Bedrock Knowledge Bases handles parsing, chunking, embedding, indexing, and retrieval. The following diagram shows the Crawl4AI path from the schedule through to retrieval.

The numbered steps in the diagram map to the following workflow:

1. Amazon EventBridge starts the Amazon Elastic Container Service (Amazon ECS) task on AWS Fargate on the configured schedule.
2. The task crawls the public websites you configured and prepares Markdown with a metadata sidecar for each page.
3. The task uploads the prepared files to the Amazon S3 bucket that backs the knowledge base data source.
4. The task starts an Amazon Bedrock Knowledge Bases ingestion job.
5. Amazon Bedrock Knowledge Bases reads the new and changed objects, parses them, and applies the chunking strategy you selected.
6. It creates embeddings and indexes them in the knowledge base’s vector store, which Amazon Bedrock manages for a managed knowledge base.
7. Your generative AI application queries the knowledge base for relevant chunks or for a generated response with citations.

The task reads its non-secret pipeline configuration from AWS Systems Manager Parameter Store at startup.

The workflow processes public pages only and doesn’t implement authenticated sessions. For sites that require sign-in, evaluate the Amazon Bedrock Web Crawler authentication options first, which include basic, form-based, and SAML authentication. Extending the sample with authentication requires session handling, credentials in AWS Secrets Manager, and a security review.

How the workflow works #

The AWS Cloud Development Kit (AWS CDK) app packages the implementation in a container image during deployment. When the schedule fires, the container runs scripts/container_entrypoint.py and executes crawl, upload, and ingestion in sequence. The following three steps explain the core logic.

Step 1: Crawl and extract content

The crawl step performs bounded crawling and writes a Markdown file with a metadata sidecar for each successfully prepared page. The following excerpt is simplified. The module src/crawl4ai_bedrock_kb/crawl.py contains the complete seed pacing, URL filtering and scoping, page limits, and redirect checks:

Amazon Bedrock Knowledge Bases supports several document formats for Amazon S3 data sources, including plain text, HTML, Markdown, PDF, and DOCX. The workflow writes Markdown because it preserves headings, lists, and tables without carrying all raw HTML into ingestion.

Step 2: Upload to Amazon S3

The upload step sends each prepared Markdown document and its matching metadata sidecar to the configured S3 bucket, under the workflow’s s3Prefix. For page.md, the sidecar must be named page.md.metadata.json, the complete source filename with .metadata.json appended. The workflow stores each sidecar in the same S3 location as its document. Amazon Bedrock limits each metadata sidecar to 10 KB. If a sidecar would exceed that limit, the run stops with an error before it writes either file.

Each attribute in the sidecar uses the typed format that both managed and customer-managed knowledge bases accept for Amazon S3 data sources. The following abbreviated sidecar, formatted for readability, shows three of the attributes that the workflow writes:

The pipeline uploads only the paths produced by the current crawl, so stale local files are not sent.

Step 3: Let Amazon Bedrock Knowledge Bases chunk, embed, and index

After content is in S3, an ingestion job hands each document to the data source’s ingestion pipeline. By default, a managed knowledge base parses each file with its managed parser, which automatically selects a parsing approach for your content. You select the chunking strategy when you connect the data source. You can’t change it afterward, so evaluate the options before you create a production data source. A managed knowledge base offers the following chunking strategies:

• Default chunking uses fixed-size chunks of 300 tokens with 20 percent overlap.
• Fixed-size chunking splits content by a maximum token count and an overlap percentage that you set.
• No chunking treats each file as a single chunk, for content that you have already split.

Chunk boundaries and retrieval results depend on the content and the selected strategy, so compare the options with representative pages and evaluation questions. To change strategies later, create a new data source. For more information, see Customize ingestion for a data source. A customer-managed knowledge base also offers semantic and hierarchical chunking and additional parsing options. For details, see How content chunking works for knowledge bases.

The workflow then starts an ingestion job so the knowledge base processes the new documents. The following simplified excerpt from src/crawl4ai_bedrock_kb/ingest.py shows the call:

The same StartIngestionJob and GetIngestionJob calls work for both knowledge base types, and you can also monitor progress on the console.

Deploy the solution #

In this section, you deploy the workflow into your AWS account with AWS CDK. Only the knowledge base prerequisites differ between the two knowledge base types. When you’re finished, follow the steps in the Clean up section to remove the resources.

Prerequisites

Before you start, make sure that you have:

• An AWS account.

• One of the following knowledge bases:

  • A managed knowledge base , which this post uses, in an AWS Region wheremanaged knowledge bases are available . For instructions, seeCreate a managed knowledge base and theAmazon S3 connector . On the data source, leaveMetadata files prefix empty, because the workflow writes each metadata sidecar alongside its document.
  • An existing customer-managed knowledge base. To add an Amazon S3 data source, seeConnect to Amazon S3 for your knowledge base . The workflow writes a metadata sidecar for every document, so confirm that your vector store can store and filter those attributes, as described inMetadata and filtering . For example, an Amazon OpenSearch Serverless vector index must use thefaiss engine. An Amazon Aurora vector index needs a custom metadata column or a column for each metadata attribute. With Amazon S3 Vectors, a capability of Amazon S3, you canattach up to 1 KB of custom metadata to each vector, so keep attributes that you add short.

• An Amazon S3 data source in that knowledge base whose status is AVAILABLE . It can be a new data source or one you already use. It must use the S3 bucket that you configure for this workflow. If the data source filters content by prefix or file pattern, the filters must allow the files that the workflow writes under itss3Prefix (defaultweb-content/ ).

• The knowledge base, the S3 bucket, and the stack you deploy all in one Region.

• Authorization to crawl the public websites you configure.

• A deployment workstation with Git, Python 3.10 or later, Node.js , the AWS Command Line Interface (AWS CLI) version 2, the AWS CDK CLI v2, and a running Docker-compatible container engine. Thedeployment guide lists the minimum AWS CDK CLI version.

• AWS credentials configured on the deployment workstation with permission to bootstrap the target environment and deploy the CDK stack.

• A bootstrapped CDK environment in the target account and Region. The deployment steps include the bootstrap command.

You need the workstation only to build and deploy. After deployment, the schedule runs the task in AWS.

Deploy the scheduled workflow

Clone the companion repository and install the deployment dependencies:

Keep Docker running. Set AWS_DEFAULT_REGION to your knowledge base’s Region first, because cdk deploy uses the Region from your environment or AWS profile, not the one in the bootstrap command. Bootstrap the target account and Region once, if needed. Then deploy the stack with your knowledge base, data source, S3 bucket, and authorized seed URLs:

Replace every example value, including ACCOUNT_ID, REGION, and the four required context keys, with your own. The example values are syntactically valid, so an unedited command deploys a stack that points at a bucket you don’t own. You can also set these keys in cdk.json.

The maxDepth and maxP