# Build a live RAG pipeline with Apify, n8n, and Qdrant
> Source:
> Published: 2026-09-16 09:12:00+00:00
Your crawler runs, the chunks reach the vector store, and the chatbot answers. Then, the pricing page changes. A product page is deleted. It keeps answering from the chunks it stored last week, confidently and wrongly, and nothing reports an error.
The crawl ran again. Nothing removed the old chunks or replaced the changed ones.
This guide builds an n8n RAG pipeline that does both. [Website Content Crawler](https://apify.com/apify/website-content-crawler) crawls the site into an Apify dataset. It's an Actor, the Apify term for a ready-made cloud program that you configure and run rather than build. A webhook starts n8n when the crawl finishes, and a second Actor chunks the pages, embeds them, and writes them into Qdrant.
Only the pages that changed are re-embedded, and pages deleted from the site are removed from the index. The workflow runs on self-hosted n8n.
This build takes about an hour once the accounts exist, and longer if you also need a tunnel in front of n8n. You need to know how to set environment variables and run n8n yourself.
## What goes wrong on the second run
On the second run, 3 things happen. The crawler returns the same pages, and the vector store node inserts a second copy of each. A page that changed since the first crawl now has 2 versions stored, and search returns both. Nothing removes a page deleted from the site, so its chunks remain.
No log line marks any of it. Answer quality drops instead, which is much harder to notice. [How retrieval-augmented generation works](https://blog.apify.com/what-is-retrieval-augmented-generation/) is well documented. Keeping that index correct over time isn't.
The n8n community forum has an open thread asking for [record management in vector stores](https://community.n8n.io/t/vector-store-record-management-in-n8n/66833). The answer from a community member in December 2024 was direct: there's no built-in record management feature for vector databases in n8n. That still holds in n8n 2.37.10, and Step 3 has the per-node breakdown. The workaround people use is to delete the whole collection before every load.
That workaround works, and on a small site it's enough. On a large one it's expensive, and the collection is only partly loaded while the rebuild runs.
## What you will build
The alternative to deleting the whole collection every night is to compare each page against what is already stored and write only the difference. The workflow has 7 nodes:
``` php
Apify Trigger -> Vectors before sync -> Sync dataset into Qdrant
-> Let the sync settle -> Vectors after sync -> Read sync log -> Freshness report
```
Here’s the n8n editor view:
Only 3 of the 7 nodes do the work. The other 4 exist to prove it happened: the counts on either side of the sync, plus `Read sync log` and `Freshness report`.
You never select **Run** on this workflow. You schedule the crawl instead, on the Apify side. When the crawl finishes, the webhook starts n8n and the 7 nodes run in order. The report shows what changed on each run.
The dataset in the middle is a saved copy. You can inspect what the crawl collected and compare it against the previous night. Replaying it into the database costs no new crawl.
## Before you start
You need 4 accounts or installs, and 3 of them are free at this scale:
- An Apify account. A free plan includes a monthly platform allowance and needs no credit card. Current limits are on the [Apify pricing page](https://apify.com/pricing) .
- n8n, self-hosted. The Community edition is free to run on your own hardware under the n8n fair-code license. Check the [n8n pricing page](https://n8n.io/pricing/) for current terms. A self-hosted instance also has to be reachable from the public internet, which is what lets Apify deliver the webhook in Step 2. On a laptop that means putting a tunnel in front of n8n, so arrange it now.
- A Qdrant Cloud cluster. The free plan includes 1 GB RAM and 4 GB disk, which is far more than this corpus needs. Current limits are on the [Qdrant pricing page](https://qdrant.tech/pricing/) .
- An OpenAI API key for embeddings. This is the paid item, and embedding this corpus cost about $0.003 at list price on the first run.
### Set the environment variables
`QDRANT_API_KEY` and `OPENAI_API_KEY` come from the n8n environment. The workflow passes `OPENAI_API_KEY` inside a JSON body to an Actor. It uses `QDRANT_API_KEY` both there and as an `api-key` header on both count nodes.
Set both where your instance reads its environment. That's a `-e` flag on `docker run`, the `environment:` block in Docker Compose, or a shell export for an npm install.
Then set `N8N_BLOCK_ENV_ACCESS_IN_NODE=false` explicitly. On n8n 2.x that switch blocks expressions from reading the environment unless it's set to the string `false`. Leave it unset and every `$env` read returns `access to env vars denied`.
All 3 go in the same place, and n8n reads the environment at startup, so restart it after any change.
The switch is instance-wide and applies to everything or to nothing. It doesn't grant this workflow access to those 2 variables. It grants every expression in every workflow on that instance access to the whole process environment.
A production install usually holds more than 2 API keys. It commonly holds the database password, and `N8N_ENCRYPTION_KEY` where the operator sets that explicitly rather than letting n8n generate one. That key decrypts every credential that the instance stores. Set the switch only where every workflow author is already trusted with everything in that environment.
On a shared instance, put the Qdrant key in a **Header Auth** credential on both count nodes instead. That removes 2 of the 4 `$env` reads, and a credential needs no environment access.
### Install the Apify community node
A new n8n install doesn't include the Apify community node.
Add it under **Settings > Community Nodes** with the package name `@apify/n8n-nodes-apify`. Do not run `npm install` in the nodes folder yourself.
n8n strips the package's peer dependencies and passes `--ignore-scripts` before installing. A plain `npm install` pulls `n8n-workflow` instead, then fails trying to compile `isolated-vm` against the image's Node version.
The Apify Trigger node needs a credential, and so do the 2 HTTP Request nodes that call the Apify API. Copy a token from the [API & Integrations](https://console.apify.com/settings/integrations) page in Apify Console, and in n8n create a credential of type **Apify account** with it. Creating the credential and attaching it to a node are separate steps, and it's common to do the first and forget the second.
Miss either step and nothing shows an error where you're looking. The workflow publishes, and the trigger then fails to start. The failure appears in a retry loop visible only in the n8n log, first as `Unrecognized node type: @apify/n8n-nodes-apify.apifyTrigger`. Once the node is present but no credential is attached, it fails as `No valid credentials found for apifyApi`.
### Import the workflow
n8n takes [`live-rag-pipeline.n8n.json`](https://gist.github.com/triposat/f493457342214fb41b71fa80a3f74c16) directly through **Import workflow from URL**, using the raw link to that file.
It needs 2 edits now:
1. Replace the placeholder cluster URL in 3 places. This is a plain URL rather than a credential, so n8n never prompts you for it. Your cluster's address is on its overview page in Qdrant Cloud, labeled **Endpoint** . Copy it with the`:6333` port at the end. The 3 places are:
- The `url` field on`Vectors before sync`
- The `url` field on`Vectors after sync`
- The `qdrantUrl` line inside the body of`Sync dataset into Qdrant`
2. The
3. Attach your Apify credential to the 3 nodes that use it.
## Step 1: Crawl the site into a dataset
Web scraping for RAG has to produce clean text rather than whole pages. Website Content Crawler (`apify/website-content-crawler`) crawls a site and strips navigation, footers, and modals before the text reaches you, which is what a text splitter needs. It writes a cleaned `text` field on each item that it stores, and that's the field that this pipeline embeds.
You run a single crawl by hand here. There are 3 settings that control what it costs and how much of the site it reaches, and each one needs a decision from you.
### Set the crawler type and memory explicitly
Apify input schemas carry 2 separate values for a field, and the [input schema specification](https://docs.apify.com/platform/actors/development/actor-definition/input-schema/specification/v1) defines both. A prefill "is only used in the user interface." A default applies whenever no value is given "via any means (API, CLI, scheduler, or user interface)."
For `crawlerType` those 2 values differ, and the half that costs you money is the one you never see. The schema default has stayed at `playwright:firefox` through every build change. Any run that names no crawler opens a headless browser on every page, whether the caller is an n8n node, a schedule, or the API.
The prefill is the unstable half. It changed twice in 2 days across builds 0.3.96 and 0.3.97, in both directions.
So name `crawlerType` explicitly in your input, and the question stops mattering.
Leaving it unnamed has a measurable cost. These 3 runs crawled the same 4 pages of Apify documentation at 4,096 MB on September 4, 2026:
| Run | `crawlerType` | Handler used | Runtime | Cost |
|---|---|---|---|---|
| **A** | not set | browser on 4 of 4 pages | 95.9s | $0.02415 |
| **B** | `cheerio` | HTTP on 4 of 4 pages | 16.1s | $0.00579 |
| **C** | `playwright:adaptive` | browser on 4 of 4 pages | 110.6s | $0.02787 |
Compared with run B, the unset default cost 4.17 times as much and adaptive cost 4.81 times as much. Adaptive runs rendering-type detection and then still opens a browser.
Memory is the second setting that you can change. Website Content Crawler declares `memoryMbytes: 8192` in its default run options, so a caller that passes no memory value runs at 8 GB. Apify bills compute units as memory multiplied by runtime, which makes that a direct multiplier on the crawl bill. Set it explicitly.
### Check the start URL before you schedule anything
In early September 2026 `https://docs.apify.com/integrations` redirected to `/platform/integrations`. Seeding the redirecting URL and running the same configuration 8 times returned 10, 28, 41, 44, 45, 49, 53, and 86 pages. Only 1 of those 8 runs reached the full site. Seeding the post-redirect URL instead returned 86 pages on 3 runs, with identical URL sets.
Apify has since removed that redirect. The check still applies, because any seed URL can start redirecting at any time:
```
curl -sI | grep -i '^location:'
```
If that prints a `location` header, seed the target instead. A crawl that silently returns a third of the site means you can only answer a third of the questions. Nothing in the run log says so.
### Run the crawl once
This input goes to the Actor itself, not into n8n. Save it in an [Actor task](https://docs.apify.com/platform/actors/running/tasks) in Apify Console. That's also where you set memory, because memory is a run option rather than an input field.
The input is:
```
{
"startUrls": [{ "url": "https://docs.apify.com/platform/integrations" }],
"crawlerType": "cheerio",
"maxCrawlPages": 500,
"maxCrawlDepth": 2,
"aggressivePrune": true
}
```
The example seeds the Apify documentation site, and every crawl measured in this guide ran against it. Point the crawl at a site you can edit if you want to test freshness in Step 6.
`maxCrawlPages` and `maxCrawlDepth` are limits rather than targets. At 500 pages and 2 levels they're already more than this documentation site needs. Raise them for a larger site, and keep them low enough that a misconfigured crawl stops early.
`aggressivePrune` is on because it removes lines that repeat across pages. On this crawl it left the page count unchanged and reduced total text by 2.1%. Line instances repeated across 20 or more pages dropped from 173 to 0.
Set the run's memory to 1,024 MB in the same place. On the example site that input returned 86 pages for $0.00809.
Run it once now. The trigger in Step 2 lists only Actors that your account has run, so this crawl puts Website Content Crawler in that list. That run leaves its pages in a dataset, which is what n8n reads next.
## Step 2: Trigger n8n when the crawl finishes
A webhook fires the moment the crawl finishes. A schedule trigger in n8n runs at fixed times, so it fires whether or not the crawl has finished.
The imported workflow already holds the **Apify Trigger** node, set to **On new Apify Event**. **Resource to Watch** is on Actor and **Event Type** on Succeeded.
Open that node and leave **Actor Source** on Recently Used Actors. Re-select Website Content Crawler from the **Actor** dropdown, even though a value is already stored.
That dropdown writes a resource locator rather than a plain string, and the node reads the object it writes. If the field holds only an ID, the node registers no webhook and publishes without an error, so it never fires.
Other nodes reference the trigger by name inside expressions, so if you rename it, change those expressions too.
### Make n8n reachable from the internet
The URL is validated at the moment the webhook is created, so it has to be an address Apify can reach before you publish. A local URL returns this:
```
Invalid value provided in webhook: Webhook requestUrl must be a valid URL.
Received "http://localhost:5678/webhook/..."
```
On a local install, put a tunnel in front of n8n first. ngrok, Cloudflare Tunnel, and Tailscale Funnel all work. Then set `WEBHOOK_URL` to that public address, in the same place as the other variables.
n8n then sends the tunnel address rather than `localhost` when it registers the webhook. Publishing before the tunnel is running creates nothing.
### Publish the workflow
On n8n 2.x the control is labeled **Publish**, and a live workflow shows as Published. The same control was called Active on 1.x, and the CLI still accepts the older `update:workflow --active` alongside the current `publish:workflow`.
Publishing creates a webhook in your Apify account through `POST /v2/webhooks`. Unpublishing deletes it. So an unpublished workflow is the first thing to check when a trigger does nothing.
### Confirm the webhook exists
A published workflow can still have no webhook, and n8n reports success either way. Read the webhook list directly:
```
curl -s -H "Authorization: Bearer $APIFY_TOKEN" \
"https://api.apify.com/v2/webhooks" \
| grep -o '"requestUrl": *"[^"]*"'
```
Export `APIFY_TOKEN` first, rather than pasting the token inline. Send it as a header rather than in the query string, because a token in a URL is written to server-side access logs.
Treat what this prints as a secret, too. Unless you put authentication in front of it, the webhook URL is the only thing protecting your pipeline from the public internet. Do not post it in forum threads or screenshots.
That list should hold a webhook pointing at your n8n instance. An empty list means the trigger registered nothing, which is the resource locator problem. Run another crawl and confirm an execution appears in n8n, not only that Apify reports a delivery. Expect that execution to stop at the sync node, because the integration Actor needs a one-time permission approval that Step 4 explains.
The trigger passes the whole Actor run object into n8n. The field that the rest of the pipeline needs is `resource.defaultDatasetId`, and n8n now receives it on every finished crawl.
## Step 3: Check what the vector store nodes can do
The first thing to try is an n8n vector store node in **Insert Documents** mode. Let the workflow run twice that way and the vector count doubles. That's not a misconfiguration.
The Qdrant vector store binding comes from LangChain rather than n8n. It assigns a random UUID to every chunk that arrives without an explicit ID, and n8n passes no IDs in insert mode. Every run, therefore, writes new points for text that already exists.
**Update Documents** is the next thing most people try. In the bundled `@n8n/ai-utilities` package, version 0.30.4, that mode has no text splitter and requires each item to resolve to exactly one document.
Those 2 limits make it a one-vector-per-document upsert. It suits short records with a stable ID. It isn't a way to maintain chunked web pages.
### Insert, update, and delete across the 13 nodes
These counts come from the `@n8n/n8n-nodes-langchain` package at version 2.37.5, bundled in n8n 2.37.10:
| Capability | Nodes |
|---|---|
| Insert | 13 of 13 root vector store nodes |
| Update by ID | 5 of 13: Azure AI Search, MongoDB Atlas, Pinecone, Redis, and Supabase |
| Delete | 0 of 13 |
The default operation set in the shared node factory is `['load', 'insert', 'retrieve', 'retrieve-as-tool']`. Update is opt-in, and the code comment for that mode says most providers omit it.
There's one more thing to check before you build the write path from these nodes. On the `n8nio/n8n:latest` image at 2.37.10, which includes Node v26.5.1, the Qdrant Vector Store node failed with `fetch failed`. The same container reached the database over plain HTTP with no error. The bundled `@qdrant/js-client-rest` 1.16.2 raises `invalid onError method` on that Node version.
So the n8n vector store nodes are built around insert, and keeping the vector database up to date belongs somewhere else.
## Step 4: Hand the write path to an integration Actor
Both halves of this pipeline are ready-made Actors from [Apify Store](https://apify.com/store), and the marketplace holds thousands more. Apify publishes integration Actors for Chroma, Milvus, OpenSearch, PGVector, Pinecone, Qdrant, and Weaviate. The code is public in [actor-vector-database-integrations](https://github.com/apify/actor-vector-database-integrations).
[Qdrant Integration](https://apify.com/apify/qdrant-integration) (`apify/qdrant-integration`) takes a dataset ID and syncs it into a collection. It identifies each page by a primary field, hashes the content into a checksum, and compares that checksum against what is already stored. The Actor only refreshes `last_seen_at` on unchanged chunks, so it doesn't re-embed them. That only works while the chunking and embedding settings stay the same between runs.
### Approve the permissions once
These Actors read your datasets and write to your storage, so the first run needs a one-time permission approval. The approval is account-wide rather than scoped to a single dataset or collection, so read it before you accept.
The API holds the run until you approve:
```
full-permission-actor-not-approved
This Actor requires full access to your account.
You must approve its permissions before running it.
```
Approve it once on the Actor page in Apify Console. The error is the same whether you start the Actor from n8n, the API, or a schedule.
### Inside the HTTP Request node
In the workflow, this is an **HTTP Request** node named `Sync dataset into Qdrant`, with your Apify credential attached. It POSTs to:
```
https://api.apify.com/v2/acts/tqzqfIiXvKWCpbRiv/runs?timeout=900&memory=1024&waitForFinish=60
```
`memory=1024` names the Actor's own default explicitly, so a later build change can't raise what the sync costs. `timeout=900` stops a stuck sync after 15 minutes rather than at the Actor's 1-hour default.
`waitForFinish=60` holds the response open until the run finishes, up to the API maximum of 60 seconds. The counts later in the workflow are then read after the sync rather than during it.
A sync longer than 60 seconds returns while still running, and those counts aren't final.
The body is one n8n expression, not plain JSON. The **JSON** body field is in expression mode and its value starts with `=`. Inside `JSON.stringify` the values are JavaScript, so `$env.QDRANT_API_KEY` is a reference rather than a quoted string:
```
={{ JSON.stringify({
qdrantUrl: 'https://YOUR-CLUSTER.qdrant.io:6333',
qdrantApiKey: $env.QDRANT_API_KEY,
qdrantCollectionName: 'docs',
qdrantAutoCreateCollection: true,
datasetId: $('Apify Trigger').first().json.resource.defaultDatasetId,
datasetFields: ['text'],
metadataDatasetFields: { url: 'url', title: 'metadata.title' },
embeddingsProvider: 'OpenAI',
embeddingsApiKey: $env.OPENAI_API_KEY,
embeddingsConfig: { model: 'text-embedding-3-small' },
dataUpdatesStrategy: 'deltaUpdates',
dataUpdatesPrimaryDatasetFields: ['url'],
deleteExpiredObjects: true,
expiredObjectDeletionPeriodDays: 30,
performChunking: true,
chunkSize: 1000,
chunkOverlap: 200
}) }}
```
Drop the `=` or quote the expressions and the Actor receives the literal text `{{ $env.QDRANT_API_KEY }}` as your API key.
### Check the fields that change the result
`dataUpdatesStrategy` is the one line that changes what the pipeline does. Set to `deltaUpdates`, it compares checksums and re-embeds what changed. Set to `add`, it appends everything on every run.
The Actor requires `qdrantUrl`, `qdrantCollectionName`, `embeddingsProvider`, `embeddingsApiKey`, and `datasetFields`. `qdrantApiKey` isn't on that list, but any Qdrant Cloud cluster requires authentication on writes, so treat it as required too. The dataset reference points at the trigger by node name rather than at `$json`, because at that stage `$json` is whatever the previous node emitted.
`embeddingsConfig` names the embedding model that you want. It's optional, and the underlying embeddings class uses `text-embedding-ada-002` when no model is named. That model costs $0.10 per 1M tokens against $0.02 for `text-embedding-3-small` at [published OpenAI prices](https://platform.openai.com/docs/pricing). Both produce 1,536 dimensions, so a run on the wrong model completes normally at 5 times the price.
`performChunking`, `chunkSize`, and `chunkOverlap` control how each page is split before embedding. The values here suit documentation prose, where a single answer usually sits inside one or two paragraphs. Changing them later invalidates every stored checksum, so choose values before the first scheduled run.
`dataUpdatesPrimaryDatasetFields` names the field that identifies a page, and the checksum comparison uses that field as its key. It's what separates an update from a duplicate. Here that field is `url`, so the same page reachable at 2 addresses would count as 2 pages.
`metadataDatasetFields` copies fields onto the stored chunk, so a retrieved result can be traced to its source URL.
### Know how your keys are stored
The Actor declares `qdrantApiKey` and `embeddingsApiKey` as secret fields, so Apify stores them encrypted. A run's saved input shows `ENCRYPTED_VALUE:` in place of each key. The rest of the input, including your cluster URL and collection name, is stored as written. Scope the OpenAI key to embeddings and the Qdrant key to this collection anyway.
## Step 5: Prove the second run
The proof comes from 4 nodes around the sync. `Vectors before sync` sits ahead of it, and `Vectors after sync`, `Read sync log`, and `Freshness report` sit after. Both counts are HTTP GETs against `/collections/docs` on your cluster, each carrying an `api-key` header set to `={{ $env.QDRANT_API_KEY }}`.
Only the first continues on error. The collection doesn't exist before the first run, so `Vectors before sync` has to tolerate a 404. The sync node creates the collection on that first pass. `Vectors after sync` must not continue on error, because the report reads a missing count as 0.
Make them consistent and a temporary read failure during the sync produces a report that shows an empty collection. Stopping there is the safer failure.
Between them, `Let the sync settle` is a Wait node set to 30 seconds. It gives the Qdrant point count time to update after the Actor reports that it finished. The file sets its unit explicitly, because the Wait node defaults to hours and a number with no unit leaves the run waiting overnight.
`waitForFinish` stops at 60 seconds, and the first pass over 86 pages took 35.4s. On a larger corpus the request returns while the sync is still running. `Freshness report` therefore checks the run status first, and if the sync hasn't finished it reports that the counts aren't final.
`Read sync log` fetches the Actor run log, and `Freshness report` parses one line from it:
``` js
const raw = $('Read sync log').first().json;
const text = typeof raw === 'string' ? raw : (raw?.data ?? JSON.stringify(raw));
const m = String(text).match(
/to add: (\d+), to update last_seen_at: (\d+), to delete: (\d+)/);
```
The report's conclusion comes from those 3 captures rather than from the point count. A run that re-embeds 5 changed chunks and expires 5 old ones leaves the count unchanged.
### Compare insert against delta updates
Both strategies ran on 2 crawls of the same 86 pages, into Qdrant 1.19.1:
| Strategy | After crawl 1 | After crawl 2 |
|---|---|---|
| **`add`** | 602 vectors | 1,204 vectors |
| **`deltaUpdates`** | 602 vectors | 602 vectors |
On the second delta pass, the Actor reported what it did:
```
Objects: to add: 0, to update last_seen_at: 602, to delete: 0
```
Nothing reached the embedding model on that run. The run finished in 17.1s for $0.00053, against 35.4s and $0.00119 for the first pass over the same 86 pages.
Those runs changed nothing. The next test started from an empty collection, then edited a single page of the same 86 and left the other 85 untouched. That page's text grew from 6,229 to 8,146 characters:
```
86 pages, first load to add: 602 to update last_seen_at: 0 to delete: 0 602 points
same 86, one page edited to add: 11 to update last_seen_at: 594 to delete: 8 605 points
```
The edited page held 8 chunks and now holds 11. The old 8 were removed in the same pass rather than after the 30-day expiry window. The collection never held 2 versions of that page. The other 85 pages, 594 chunks between them, were refreshed without reaching the embedding model.
### Set the expiry window from your crawl history
`deleteExpiredObjects` defaults to true, and `expiredObjectDeletionPeriodDays` defaults to 30. Deletion is time-based and scoped to the collection, not to the current crawl. Anything whose `last_seen_at` is older than the cutoff is removed. A page deleted from the site stops appearing in crawls, so its chunks expire and the sweep removes them.
Pair that with the crawl variation from Step 1 and a failure mode appears. This test used the 86-page crawl as night 1 and a partial 28-page crawl as night 2. The expiry sweep then ran with a window short enough for the night 1 pages to have expired:
```
night 1: full 86-page crawl pages=86 chunks=602 vectors_in_db=602
night 2: partial 28-page crawl pages=28 chunks=194 vectors_in_db=604
after expiry sweep vectors_in_db=194
```
The 2 extra vectors on night 2 are chunks whose checksum matched nothing already stored. Of the 194 that remained, 192 were originals. The sweep removed 410 of the first 602, 68.1%, for pages that still exist on the site.
This doesn't usually happen on the 30-day default, because every crawl has to miss the page for a month before it expires. Shortening the window to match a nightly crawl removes that protection. Fix the crawl so it returns the same pages every run. Then set the window longer than the longest series of incomplete crawls that you're willing to accept.
## Step 6: Query the index
Reading is the job that suits the n8n vector store node here, because it needs neither update nor delete. Add an **AI Agent** node and attach the vector store in **Retrieve Documents (As Tool for AI Agent)** mode. Connect a **Chat Trigger** to test it. These are nodes that you add yourself: the workflow file handles ingestion only.
If you're on the image described in Step 3, the Qdrant node fails here for the same transport reason that it fails on writes. Query the collection over plain HTTP, as the count nodes do.
Testing freshness needs a page you can edit. Change a sentence, re-run, and ask a question that only the new sentence answers. A question you can answer from either version proves nothing.
If you can't edit the source, the `Freshness report` is the check.
For where retrieval sits in a larger system, see [how web data flows into agent retrieval](https://blog.apify.com/ai-agent-infrastructure/).
## Step 7: Schedule it
Schedule the crawl on the Apify side with [Actor schedules](https://docs.apify.com/platform/actors/running/schedules), which take a cron expression and are timezone aware. The webhook then starts n8n whenever a run finishes, so you're not maintaining 2 schedules that can stop matching. Add a second **Apify Trigger** on the Failed event so a failed crawl reaches you rather than quietly skipping a night.
Then measure your corpus's change interval instead of guessing at a schedule. Run the crawl twice, a few days apart, and compare the page text per URL:
| Window | Pages unchanged | Changed | Added |
|---|---|---|---|
| **September 4 to September 7** | 86 of 86 | 0 | 0 |
| **September 7 to September 8** | 86 of 86 | 0 | 1 |
No page was edited in 4 days. A single new page appeared, which raised the crawl from 86 pages to 87. A nightly schedule on this corpus fetches all 86 pages again every night to learn that.
Match the schedule to the interval that you measured rather than to a habit. Weekly on a corpus that changes weekly costs a seventh as much as nightly, as long as nothing urgent appears between crawls.
One further setting would reduce the crawl bill. Website Content Crawler passes `customHttpHeaders` to the target site, so an `If-Modified-Since` header turns an unchanged crawl into a 304. In one before-and-after pair, that reduced the crawl from $0.01089 to $0.00151.
A conditional crawl of an unchanged site returns no items. To the expiry deletion in Step 5, a run of no items looks exactly like a failed crawl.
On a corpus stable enough to be worth crawling conditionally, 30 such nights in sequence put every point past the 30-day `last_seen_at` cutoff. Do not run conditional crawling and expiry deletion together until you have tested that combination against your own collection.
## What the n8n RAG pipeline costs to run
Figures come from measured runs on a free Apify plan in September 2026, plus published list prices for the other components.
| Component | Cost |
|---|---|
| Crawl, 86 pages, HTTP crawler at 1,024 MB | $0.00809 per run |
| Conditional request, untested against expiry deletion | $0.00151 per run |
| Browser crawler, measured on 4 pages at 4,096 MB | 4.17 times the HTTP crawler on the same 4 pages |
| Embeddings, first run, 602 chunks | about $0.003 at list price |
| Embeddings, later runs with delta updates | $0 when nothing changed |
| Qdrant Cloud free cluster | $0 |
| n8n Community, self-hosted | $0 |
Delta updates reduce the embedding bill and not the crawl bill, because by default every run fetches every page. Once embeddings drop toward zero, the crawl is nearly the whole cost, and it buys nothing on a corpus that hasn't changed.
On a corpus this size, the entire delta saving is the $0.003 embeddings row. Delta updates leave the collection whole, where a full rebuild empties it first. A larger corpus raises both the bill and the time spent reloading.
## Run your own n8n RAG pipeline twice
Measure your own change interval before you choose a schedule. Fix any crawl that returns a different page count each run before you change the expiry window.
The fastest check is to point [Website Content Crawler](https://apify.com/apify/website-content-crawler) at your own documentation and let the pipeline run twice. The free Apify plan is enough for that. Either the count is unchanged and nothing was re-embedded, or `Freshness report` names what changed.
## FAQ
### Do repeated inserts create duplicates?
Yes. Any code that writes without an explicit ID produces a new random UUID for each chunk, so the same text is stored twice. Running 86 pages twice through the integration Actor on its `add` strategy produced 602 vectors and then 1,204. On `deltaUpdates`, the same 2 runs left the count at 602.
### Can n8n delete vectors from a vector database?
No vector store node in n8n exposes a delete operation, checked against version 2.37.5 of the LangChain nodes package. Update by ID works on 5 of the 13 root vector store nodes. To remove records, use the database client directly, empty a namespace on insert, or let an integration Actor manage expiry.
### Do I need to re-embed everything when one page changes?
No. The integration Actor hashes each page into a checksum and compares it against what is stored. Editing a single page of 86 produced 11 chunks added, 8 removed, and 594 refreshed. Only the edited page reached the embedding model, and its old chunks were removed in the same pass.
### How often should I re-crawl for RAG?
Measure it rather than guessing: crawl twice a few days apart and compare the text per URL. One documentation corpus here had no edited pages across 4 days. It gained a single new page in that time, so 3 of those 4 nights would have found nothing.
### Why did my crawl return fewer pages than the site has?
Check whether the start URL redirects. Seeding a redirecting URL returned between 10 and 86 pages across 8 identical runs here, and only 1 of those runs reached the full site. Seed the post-redirect target instead. Crawling it returned 86 pages on all 3 runs here, with identical URL sets.