{"slug": "build-a-live-rag-pipeline-with-apify-n8n-and-qdrant", "title": "Build a live RAG pipeline with Apify, n8n, and Qdrant", "summary": "Apify published a guide for building a live retrieval-augmented generation (RAG) pipeline using Apify's Website Content Crawler, self-hosted n8n, and Qdrant that re-embeds only changed pages and removes deleted pages from the vector index. The guide notes that n8n 2.37.10 still lacks built-in vector store record management, a gap a community member confirmed in December 2024, so the common workaround is deleting the entire collection before every load. The 7-node workflow runs on a webhook triggered when an Apify crawl finishes, and the build takes about an hour once accounts exist.", "body_md": "Your crawler runs, the chunks reach the vector store, and the chatbot answers. Then, the pricing page changes. A product page is deleted. It keeps answering from the chunks it stored last week, confidently and wrongly, and nothing reports an error.\n\nThe crawl ran again. Nothing removed the old chunks or replaced the changed ones.\n\nThis guide builds an n8n RAG pipeline that does both. [Website Content Crawler](https://apify.com/apify/website-content-crawler) crawls the site into an Apify dataset. It's an Actor, the Apify term for a ready-made cloud program that you configure and run rather than build. A webhook starts n8n when the crawl finishes, and a second Actor chunks the pages, embeds them, and writes them into Qdrant.\n\nOnly the pages that changed are re-embedded, and pages deleted from the site are removed from the index. The workflow runs on self-hosted n8n.\n\nThis build takes about an hour once the accounts exist, and longer if you also need a tunnel in front of n8n. You need to know how to set environment variables and run n8n yourself.\n\n## What goes wrong on the second run\n\nOn the second run, 3 things happen. The crawler returns the same pages, and the vector store node inserts a second copy of each. A page that changed since the first crawl now has 2 versions stored, and search returns both. Nothing removes a page deleted from the site, so its chunks remain.\n\nNo log line marks any of it. Answer quality drops instead, which is much harder to notice. [How retrieval-augmented generation works](https://blog.apify.com/what-is-retrieval-augmented-generation/) is well documented. Keeping that index correct over time isn't.\n\nThe n8n community forum has an open thread asking for [record management in vector stores](https://community.n8n.io/t/vector-store-record-management-in-n8n/66833). The answer from a community member in December 2024 was direct: there's no built-in record management feature for vector databases in n8n. That still holds in n8n 2.37.10, and Step 3 has the per-node breakdown. The workaround people use is to delete the whole collection before every load.\n\nThat workaround works, and on a small site it's enough. On a large one it's expensive, and the collection is only partly loaded while the rebuild runs.\n\n## What you will build\n\nThe alternative to deleting the whole collection every night is to compare each page against what is already stored and write only the difference. The workflow has 7 nodes:\n\n``` php\nApify Trigger -> Vectors before sync -> Sync dataset into Qdrant\n  -> Let the sync settle -> Vectors after sync -> Read sync log -> Freshness report\n```\n\nHere’s the n8n editor view:\n\nOnly 3 of the 7 nodes do the work. The other 4 exist to prove it happened: the counts on either side of the sync, plus `Read sync log` and `Freshness report`.\n\nYou never select **Run** on this workflow. You schedule the crawl instead, on the Apify side. When the crawl finishes, the webhook starts n8n and the 7 nodes run in order. The report shows what changed on each run.\n\nThe dataset in the middle is a saved copy. You can inspect what the crawl collected and compare it against the previous night. Replaying it into the database costs no new crawl.\n\n## Before you start\n\nYou need 4 accounts or installs, and 3 of them are free at this scale:\n\n- An Apify account. A free plan includes a monthly platform allowance and needs no credit card. Current limits are on the [Apify pricing page](https://apify.com/pricing) .\n- n8n, self-hosted. The Community edition is free to run on your own hardware under the n8n fair-code license. Check the [n8n pricing page](https://n8n.io/pricing/) for current terms. A self-hosted instance also has to be reachable from the public internet, which is what lets Apify deliver the webhook in Step 2. On a laptop that means putting a tunnel in front of n8n, so arrange it now.\n- A Qdrant Cloud cluster. The free plan includes 1 GB RAM and 4 GB disk, which is far more than this corpus needs. Current limits are on the [Qdrant pricing page](https://qdrant.tech/pricing/) .\n- An OpenAI API key for embeddings. This is the paid item, and embedding this corpus cost about $0.003 at list price on the first run.\n\n### Set the environment variables\n\n`QDRANT_API_KEY` and `OPENAI_API_KEY` come from the n8n environment. The workflow passes `OPENAI_API_KEY` inside a JSON body to an Actor. It uses `QDRANT_API_KEY` both there and as an `api-key` header on both count nodes.\n\nSet both where your instance reads its environment. That's a `-e` flag on `docker run`, the `environment:` block in Docker Compose, or a shell export for an npm install.\n\nThen set `N8N_BLOCK_ENV_ACCESS_IN_NODE=false` explicitly. On n8n 2.x that switch blocks expressions from reading the environment unless it's set to the string `false`. Leave it unset and every `$env` read returns `access to env vars denied`.\n\nAll 3 go in the same place, and n8n reads the environment at startup, so restart it after any change.\n\nThe switch is instance-wide and applies to everything or to nothing. It doesn't grant this workflow access to those 2 variables. It grants every expression in every workflow on that instance access to the whole process environment.\n\nA production install usually holds more than 2 API keys. It commonly holds the database password, and `N8N_ENCRYPTION_KEY` where the operator sets that explicitly rather than letting n8n generate one. That key decrypts every credential that the instance stores. Set the switch only where every workflow author is already trusted with everything in that environment.\n\nOn a shared instance, put the Qdrant key in a **Header Auth** credential on both count nodes instead. That removes 2 of the 4 `$env` reads, and a credential needs no environment access.\n\n### Install the Apify community node\n\nA new n8n install doesn't include the Apify community node.\n\nAdd it under **Settings > Community Nodes** with the package name `@apify/n8n-nodes-apify`. Do not run `npm install` in the nodes folder yourself.\n\nn8n strips the package's peer dependencies and passes `--ignore-scripts` before installing. A plain `npm install` pulls `n8n-workflow` instead, then fails trying to compile `isolated-vm` against the image's Node version.\n\nThe Apify Trigger node needs a credential, and so do the 2 HTTP Request nodes that call the Apify API. Copy a token from the [API & Integrations](https://console.apify.com/settings/integrations) page in Apify Console, and in n8n create a credential of type **Apify account** with it. Creating the credential and attaching it to a node are separate steps, and it's common to do the first and forget the second.\n\nMiss either step and nothing shows an error where you're looking. The workflow publishes, and the trigger then fails to start. The failure appears in a retry loop visible only in the n8n log, first as `Unrecognized node type: @apify/n8n-nodes-apify.apifyTrigger`. Once the node is present but no credential is attached, it fails as `No valid credentials found for apifyApi`.\n\n### Import the workflow\n\nn8n takes [`live-rag-pipeline.n8n.json`](https://gist.github.com/triposat/f493457342214fb41b71fa80a3f74c16) directly through **Import workflow from URL**, using the raw link to that file.\n\nIt needs 2 edits now:\n\n1. Replace the placeholder cluster URL in 3 places. This is a plain URL rather than a credential, so n8n never prompts you for it. Your cluster's address is on its overview page in Qdrant Cloud, labeled **Endpoint** . Copy it with the`:6333` port at the end. The 3 places are:\n  - The `url` field on`Vectors before sync`\n  - The `url` field on`Vectors after sync`\n  - The `qdrantUrl` line inside the body of`Sync dataset into Qdrant`\n2. The \n3. Attach your Apify credential to the 3 nodes that use it.\n\n## Step 1: Crawl the site into a dataset\n\nWeb scraping for RAG has to produce clean text rather than whole pages. Website Content Crawler (`apify/website-content-crawler`) crawls a site and strips navigation, footers, and modals before the text reaches you, which is what a text splitter needs. It writes a cleaned `text` field on each item that it stores, and that's the field that this pipeline embeds.\n\nYou run a single crawl by hand here. There are 3 settings that control what it costs and how much of the site it reaches, and each one needs a decision from you.\n\n### Set the crawler type and memory explicitly\n\nApify input schemas carry 2 separate values for a field, and the [input schema specification](https://docs.apify.com/platform/actors/development/actor-definition/input-schema/specification/v1) defines both. A prefill \"is only used in the user interface.\" A default applies whenever no value is given \"via any means (API, CLI, scheduler, or user interface).\"\n\nFor `crawlerType` those 2 values differ, and the half that costs you money is the one you never see. The schema default has stayed at `playwright:firefox` through every build change. Any run that names no crawler opens a headless browser on every page, whether the caller is an n8n node, a schedule, or the API.\n\nThe prefill is the unstable half. It changed twice in 2 days across builds 0.3.96 and 0.3.97, in both directions.\n\nSo name `crawlerType` explicitly in your input, and the question stops mattering.\n\nLeaving it unnamed has a measurable cost. These 3 runs crawled the same 4 pages of Apify documentation at 4,096 MB on September 4, 2026:\n\n| Run | `crawlerType` | Handler used | Runtime | Cost | \n|---|---|---|---|---|\n| **A** | not set | browser on 4 of 4 pages | 95.9s | $0.02415 | \n| **B** | `cheerio` | HTTP on 4 of 4 pages | 16.1s | $0.00579 | \n| **C** | `playwright:adaptive` | browser on 4 of 4 pages | 110.6s | $0.02787 | \n\nCompared with run B, the unset default cost 4.17 times as much and adaptive cost 4.81 times as much. Adaptive runs rendering-type detection and then still opens a browser.\n\nMemory is the second setting that you can change. Website Content Crawler declares `memoryMbytes: 8192` in its default run options, so a caller that passes no memory value runs at 8 GB. Apify bills compute units as memory multiplied by runtime, which makes that a direct multiplier on the crawl bill. Set it explicitly.\n\n### Check the start URL before you schedule anything\n\nIn early September 2026 `https://docs.apify.com/integrations` redirected to `/platform/integrations`. Seeding the redirecting URL and running the same configuration 8 times returned 10, 28, 41, 44, 45, 49, 53, and 86 pages. Only 1 of those 8 runs reached the full site. Seeding the post-redirect URL instead returned 86 pages on 3 runs, with identical URL sets.\n\nApify has since removed that redirect. The check still applies, because any seed URL can start redirecting at any time:\n\n```\ncurl -sI <https://your-site.example/docs> | grep -i '^location:'\n```\n\nIf that prints a `location` header, seed the target instead. A crawl that silently returns a third of the site means you can only answer a third of the questions. Nothing in the run log says so.\n\n### Run the crawl once\n\nThis input goes to the Actor itself, not into n8n. Save it in an [Actor task](https://docs.apify.com/platform/actors/running/tasks) in Apify Console. That's also where you set memory, because memory is a run option rather than an input field.\n\nThe input is:\n\n```\n{\n  \"startUrls\": [{ \"url\": \"https://docs.apify.com/platform/integrations\" }],\n  \"crawlerType\": \"cheerio\",\n  \"maxCrawlPages\": 500,\n  \"maxCrawlDepth\": 2,\n  \"aggressivePrune\": true\n}\n```\n\nThe example seeds the Apify documentation site, and every crawl measured in this guide ran against it. Point the crawl at a site you can edit if you want to test freshness in Step 6.\n\n`maxCrawlPages` and `maxCrawlDepth` are limits rather than targets. At 500 pages and 2 levels they're already more than this documentation site needs. Raise them for a larger site, and keep them low enough that a misconfigured crawl stops early.\n\n`aggressivePrune` is on because it removes lines that repeat across pages. On this crawl it left the page count unchanged and reduced total text by 2.1%. Line instances repeated across 20 or more pages dropped from 173 to 0.\n\nSet the run's memory to 1,024 MB in the same place. On the example site that input returned 86 pages for $0.00809.\n\nRun it once now. The trigger in Step 2 lists only Actors that your account has run, so this crawl puts Website Content Crawler in that list. That run leaves its pages in a dataset, which is what n8n reads next.\n\n## Step 2: Trigger n8n when the crawl finishes\n\nA webhook fires the moment the crawl finishes. A schedule trigger in n8n runs at fixed times, so it fires whether or not the crawl has finished.\n\nThe imported workflow already holds the **Apify Trigger** node, set to **On new Apify Event**. **Resource to Watch** is on Actor and **Event Type** on Succeeded.\n\nOpen that node and leave **Actor Source** on Recently Used Actors. Re-select Website Content Crawler from the **Actor** dropdown, even though a value is already stored.\n\nThat dropdown writes a resource locator rather than a plain string, and the node reads the object it writes. If the field holds only an ID, the node registers no webhook and publishes without an error, so it never fires.\n\nOther nodes reference the trigger by name inside expressions, so if you rename it, change those expressions too.\n\n### Make n8n reachable from the internet\n\nThe URL is validated at the moment the webhook is created, so it has to be an address Apify can reach before you publish. A local URL returns this:\n\n```\nInvalid value provided in webhook: Webhook requestUrl must be a valid URL.\nReceived \"http://localhost:5678/webhook/...\"\n```\n\nOn a local install, put a tunnel in front of n8n first. ngrok, Cloudflare Tunnel, and Tailscale Funnel all work. Then set `WEBHOOK_URL` to that public address, in the same place as the other variables.\n\nn8n then sends the tunnel address rather than `localhost` when it registers the webhook. Publishing before the tunnel is running creates nothing.\n\n### Publish the workflow\n\nOn n8n 2.x the control is labeled **Publish**, and a live workflow shows as Published. The same control was called Active on 1.x, and the CLI still accepts the older `update:workflow --active` alongside the current `publish:workflow`.\n\nPublishing creates a webhook in your Apify account through `POST /v2/webhooks`. Unpublishing deletes it. So an unpublished workflow is the first thing to check when a trigger does nothing.\n\n### Confirm the webhook exists\n\nA published workflow can still have no webhook, and n8n reports success either way. Read the webhook list directly:\n\n```\ncurl -s -H \"Authorization: Bearer $APIFY_TOKEN\" \\\n  \"https://api.apify.com/v2/webhooks\" \\\n  | grep -o '\"requestUrl\": *\"[^\"]*\"'\n```\n\nExport `APIFY_TOKEN` first, rather than pasting the token inline. Send it as a header rather than in the query string, because a token in a URL is written to server-side access logs.\n\nTreat what this prints as a secret, too. Unless you put authentication in front of it, the webhook URL is the only thing protecting your pipeline from the public internet. Do not post it in forum threads or screenshots.\n\nThat list should hold a webhook pointing at your n8n instance. An empty list means the trigger registered nothing, which is the resource locator problem. Run another crawl and confirm an execution appears in n8n, not only that Apify reports a delivery. Expect that execution to stop at the sync node, because the integration Actor needs a one-time permission approval that Step 4 explains.\n\nThe trigger passes the whole Actor run object into n8n. The field that the rest of the pipeline needs is `resource.defaultDatasetId`, and n8n now receives it on every finished crawl.\n\n## Step 3: Check what the vector store nodes can do\n\nThe first thing to try is an n8n vector store node in **Insert Documents** mode. Let the workflow run twice that way and the vector count doubles. That's not a misconfiguration.\n\nThe Qdrant vector store binding comes from LangChain rather than n8n. It assigns a random UUID to every chunk that arrives without an explicit ID, and n8n passes no IDs in insert mode. Every run, therefore, writes new points for text that already exists.\n\n**Update Documents** is the next thing most people try. In the bundled `@n8n/ai-utilities` package, version 0.30.4, that mode has no text splitter and requires each item to resolve to exactly one document.\n\nThose 2 limits make it a one-vector-per-document upsert. It suits short records with a stable ID. It isn't a way to maintain chunked web pages.\n\n### Insert, update, and delete across the 13 nodes\n\nThese counts come from the `@n8n/n8n-nodes-langchain` package at version 2.37.5, bundled in n8n 2.37.10:\n\n| Capability | Nodes | \n|---|---|\n| Insert | 13 of 13 root vector store nodes | \n| Update by ID | 5 of 13: Azure AI Search, MongoDB Atlas, Pinecone, Redis, and Supabase | \n| Delete | 0 of 13 | \n\nThe default operation set in the shared node factory is `['load', 'insert', 'retrieve', 'retrieve-as-tool']`. Update is opt-in, and the code comment for that mode says most providers omit it.\n\nThere's one more thing to check before you build the write path from these nodes. On the `n8nio/n8n:latest` image at 2.37.10, which includes Node v26.5.1, the Qdrant Vector Store node failed with `fetch failed`. The same container reached the database over plain HTTP with no error. The bundled `@qdrant/js-client-rest` 1.16.2 raises `invalid onError method` on that Node version.\n\nSo the n8n vector store nodes are built around insert, and keeping the vector database up to date belongs somewhere else.\n\n## Step 4: Hand the write path to an integration Actor\n\nBoth halves of this pipeline are ready-made Actors from [Apify Store](https://apify.com/store), and the marketplace holds thousands more. Apify publishes integration Actors for Chroma, Milvus, OpenSearch, PGVector, Pinecone, Qdrant, and Weaviate. The code is public in [actor-vector-database-integrations](https://github.com/apify/actor-vector-database-integrations).\n\n[Qdrant Integration](https://apify.com/apify/qdrant-integration) (`apify/qdrant-integration`) takes a dataset ID and syncs it into a collection. It identifies each page by a primary field, hashes the content into a checksum, and compares that checksum against what is already stored. The Actor only refreshes `last_seen_at` on unchanged chunks, so it doesn't re-embed them. That only works while the chunking and embedding settings stay the same between runs.\n\n### Approve the permissions once\n\nThese Actors read your datasets and write to your storage, so the first run needs a one-time permission approval. The approval is account-wide rather than scoped to a single dataset or collection, so read it before you accept.\n\nThe API holds the run until you approve:\n\n```\nfull-permission-actor-not-approved\nThis Actor requires full access to your account.\nYou must approve its permissions before running it.\n```\n\nApprove it once on the Actor page in Apify Console. The error is the same whether you start the Actor from n8n, the API, or a schedule.\n\n### Inside the HTTP Request node\n\nIn the workflow, this is an **HTTP Request** node named `Sync dataset into Qdrant`, with your Apify credential attached. It POSTs to:\n\n```\nhttps://api.apify.com/v2/acts/tqzqfIiXvKWCpbRiv/runs?timeout=900&memory=1024&waitForFinish=60\n```\n\n`memory=1024` names the Actor's own default explicitly, so a later build change can't raise what the sync costs. `timeout=900` stops a stuck sync after 15 minutes rather than at the Actor's 1-hour default.\n\n`waitForFinish=60` holds the response open until the run finishes, up to the API maximum of 60 seconds. The counts later in the workflow are then read after the sync rather than during it.\n\nA sync longer than 60 seconds returns while still running, and those counts aren't final.\n\nThe body is one n8n expression, not plain JSON. The **JSON** body field is in expression mode and its value starts with `=`. Inside `JSON.stringify` the values are JavaScript, so `$env.QDRANT_API_KEY` is a reference rather than a quoted string:\n\n```\n={{ JSON.stringify({\n  qdrantUrl: 'https://YOUR-CLUSTER.qdrant.io:6333',\n  qdrantApiKey: $env.QDRANT_API_KEY,\n  qdrantCollectionName: 'docs',\n  qdrantAutoCreateCollection: true,\n  datasetId: $('Apify Trigger').first().json.resource.defaultDatasetId,\n  datasetFields: ['text'],\n  metadataDatasetFields: { url: 'url', title: 'metadata.title' },\n  embeddingsProvider: 'OpenAI',\n  embeddingsApiKey: $env.OPENAI_API_KEY,\n  embeddingsConfig: { model: 'text-embedding-3-small' },\n  dataUpdatesStrategy: 'deltaUpdates',\n  dataUpdatesPrimaryDatasetFields: ['url'],\n  deleteExpiredObjects: true,\n  expiredObjectDeletionPeriodDays: 30,\n  performChunking: true,\n  chunkSize: 1000,\n  chunkOverlap: 200\n}) }}\n```\n\nDrop the `=` or quote the expressions and the Actor receives the literal text `{{ $env.QDRANT_API_KEY }}` as your API key.\n\n### Check the fields that change the result\n\n`dataUpdatesStrategy` is the one line that changes what the pipeline does. Set to `deltaUpdates`, it compares checksums and re-embeds what changed. Set to `add`, it appends everything on every run.\n\nThe Actor requires `qdrantUrl`, `qdrantCollectionName`, `embeddingsProvider`, `embeddingsApiKey`, and `datasetFields`. `qdrantApiKey` isn't on that list, but any Qdrant Cloud cluster requires authentication on writes, so treat it as required too. The dataset reference points at the trigger by node name rather than at `$json`, because at that stage `$json` is whatever the previous node emitted.\n\n`embeddingsConfig` names the embedding model that you want. It's optional, and the underlying embeddings class uses `text-embedding-ada-002` when no model is named. That model costs $0.10 per 1M tokens against $0.02 for `text-embedding-3-small` at [published OpenAI prices](https://platform.openai.com/docs/pricing). Both produce 1,536 dimensions, so a run on the wrong model completes normally at 5 times the price.\n\n`performChunking`, `chunkSize`, and `chunkOverlap` control how each page is split before embedding. The values here suit documentation prose, where a single answer usually sits inside one or two paragraphs. Changing them later invalidates every stored checksum, so choose values before the first scheduled run.\n\n`dataUpdatesPrimaryDatasetFields` names the field that identifies a page, and the checksum comparison uses that field as its key. It's what separates an update from a duplicate. Here that field is `url`, so the same page reachable at 2 addresses would count as 2 pages.\n\n`metadataDatasetFields` copies fields onto the stored chunk, so a retrieved result can be traced to its source URL.\n\n### Know how your keys are stored\n\nThe Actor declares `qdrantApiKey` and `embeddingsApiKey` as secret fields, so Apify stores them encrypted. A run's saved input shows `ENCRYPTED_VALUE:` in place of each key. The rest of the input, including your cluster URL and collection name, is stored as written. Scope the OpenAI key to embeddings and the Qdrant key to this collection anyway.\n\n## Step 5: Prove the second run\n\nThe proof comes from 4 nodes around the sync. `Vectors before sync` sits ahead of it, and `Vectors after sync`, `Read sync log`, and `Freshness report` sit after. Both counts are HTTP GETs against `/collections/docs` on your cluster, each carrying an `api-key` header set to `={{ $env.QDRANT_API_KEY }}`.\n\nOnly the first continues on error. The collection doesn't exist before the first run, so `Vectors before sync` has to tolerate a 404. The sync node creates the collection on that first pass. `Vectors after sync` must not continue on error, because the report reads a missing count as 0.\n\nMake them consistent and a temporary read failure during the sync produces a report that shows an empty collection. Stopping there is the safer failure.\n\nBetween them, `Let the sync settle` is a Wait node set to 30 seconds. It gives the Qdrant point count time to update after the Actor reports that it finished. The file sets its unit explicitly, because the Wait node defaults to hours and a number with no unit leaves the run waiting overnight.\n\n`waitForFinish` stops at 60 seconds, and the first pass over 86 pages took 35.4s. On a larger corpus the request returns while the sync is still running. `Freshness report` therefore checks the run status first, and if the sync hasn't finished it reports that the counts aren't final.\n\n`Read sync log` fetches the Actor run log, and `Freshness report` parses one line from it:\n\n``` js\nconst raw  = $('Read sync log').first().json;\nconst text = typeof raw === 'string' ? raw : (raw?.data ?? JSON.stringify(raw));\nconst m = String(text).match(\n  /to add: (\\d+), to update last_seen_at: (\\d+), to delete: (\\d+)/);\n```\n\nThe report's conclusion comes from those 3 captures rather than from the point count. A run that re-embeds 5 changed chunks and expires 5 old ones leaves the count unchanged.\n\n### Compare insert against delta updates\n\nBoth strategies ran on 2 crawls of the same 86 pages, into Qdrant 1.19.1:\n\n| Strategy | After crawl 1 | After crawl 2 | \n|---|---|---|\n| **`add`** | 602 vectors | 1,204 vectors | \n| **`deltaUpdates`** | 602 vectors | 602 vectors | \n\nOn the second delta pass, the Actor reported what it did:\n\n```\nObjects: to add: 0, to update last_seen_at: 602, to delete: 0\n```\n\nNothing reached the embedding model on that run. The run finished in 17.1s for $0.00053, against 35.4s and $0.00119 for the first pass over the same 86 pages.\n\nThose runs changed nothing. The next test started from an empty collection, then edited a single page of the same 86 and left the other 85 untouched. That page's text grew from 6,229 to 8,146 characters:\n\n```\n86 pages, first load        to add: 602  to update last_seen_at: 0    to delete: 0     602 points\nsame 86, one page edited    to add: 11   to update last_seen_at: 594  to delete: 8     605 points\n```\n\nThe edited page held 8 chunks and now holds 11. The old 8 were removed in the same pass rather than after the 30-day expiry window. The collection never held 2 versions of that page. The other 85 pages, 594 chunks between them, were refreshed without reaching the embedding model.\n\n### Set the expiry window from your crawl history\n\n`deleteExpiredObjects` defaults to true, and `expiredObjectDeletionPeriodDays` defaults to 30. Deletion is time-based and scoped to the collection, not to the current crawl. Anything whose `last_seen_at` is older than the cutoff is removed. A page deleted from the site stops appearing in crawls, so its chunks expire and the sweep removes them.\n\nPair that with the crawl variation from Step 1 and a failure mode appears. This test used the 86-page crawl as night 1 and a partial 28-page crawl as night 2. The expiry sweep then ran with a window short enough for the night 1 pages to have expired:\n\n```\nnight 1: full 86-page crawl        pages=86  chunks=602  vectors_in_db=602\nnight 2: partial 28-page crawl     pages=28  chunks=194  vectors_in_db=604\nafter expiry sweep                                       vectors_in_db=194\n```\n\nThe 2 extra vectors on night 2 are chunks whose checksum matched nothing already stored. Of the 194 that remained, 192 were originals. The sweep removed 410 of the first 602, 68.1%, for pages that still exist on the site.\n\nThis doesn't usually happen on the 30-day default, because every crawl has to miss the page for a month before it expires. Shortening the window to match a nightly crawl removes that protection. Fix the crawl so it returns the same pages every run. Then set the window longer than the longest series of incomplete crawls that you're willing to accept.\n\n## Step 6: Query the index\n\nReading is the job that suits the n8n vector store node here, because it needs neither update nor delete. Add an **AI Agent** node and attach the vector store in **Retrieve Documents (As Tool for AI Agent)** mode. Connect a **Chat Trigger** to test it. These are nodes that you add yourself: the workflow file handles ingestion only.\n\nIf you're on the image described in Step 3, the Qdrant node fails here for the same transport reason that it fails on writes. Query the collection over plain HTTP, as the count nodes do.\n\nTesting freshness needs a page you can edit. Change a sentence, re-run, and ask a question that only the new sentence answers. A question you can answer from either version proves nothing.\n\nIf you can't edit the source, the `Freshness report` is the check.\n\nFor where retrieval sits in a larger system, see [how web data flows into agent retrieval](https://blog.apify.com/ai-agent-infrastructure/).\n\n## Step 7: Schedule it\n\nSchedule the crawl on the Apify side with [Actor schedules](https://docs.apify.com/platform/actors/running/schedules), which take a cron expression and are timezone aware. The webhook then starts n8n whenever a run finishes, so you're not maintaining 2 schedules that can stop matching. Add a second **Apify Trigger** on the Failed event so a failed crawl reaches you rather than quietly skipping a night.\n\nThen measure your corpus's change interval instead of guessing at a schedule. Run the crawl twice, a few days apart, and compare the page text per URL:\n\n| Window | Pages unchanged | Changed | Added | \n|---|---|---|---|\n| **September 4 to September 7** | 86 of 86 | 0 | 0 | \n| **September 7 to September 8** | 86 of 86 | 0 | 1 | \n\nNo page was edited in 4 days. A single new page appeared, which raised the crawl from 86 pages to 87. A nightly schedule on this corpus fetches all 86 pages again every night to learn that.\n\nMatch the schedule to the interval that you measured rather than to a habit. Weekly on a corpus that changes weekly costs a seventh as much as nightly, as long as nothing urgent appears between crawls.\n\nOne further setting would reduce the crawl bill. Website Content Crawler passes `customHttpHeaders` to the target site, so an `If-Modified-Since` header turns an unchanged crawl into a 304. In one before-and-after pair, that reduced the crawl from $0.01089 to $0.00151.\n\nA conditional crawl of an unchanged site returns no items. To the expiry deletion in Step 5, a run of no items looks exactly like a failed crawl.\n\nOn a corpus stable enough to be worth crawling conditionally, 30 such nights in sequence put every point past the 30-day `last_seen_at` cutoff. Do not run conditional crawling and expiry deletion together until you have tested that combination against your own collection.\n\n## What the n8n RAG pipeline costs to run\n\nFigures come from measured runs on a free Apify plan in September 2026, plus published list prices for the other components.\n\n| Component | Cost | \n|---|---|\n| Crawl, 86 pages, HTTP crawler at 1,024 MB | $0.00809 per run | \n| Conditional request, untested against expiry deletion | $0.00151 per run | \n| Browser crawler, measured on 4 pages at 4,096 MB | 4.17 times the HTTP crawler on the same 4 pages | \n| Embeddings, first run, 602 chunks | about $0.003 at list price | \n| Embeddings, later runs with delta updates | $0 when nothing changed | \n| Qdrant Cloud free cluster | $0 | \n| n8n Community, self-hosted | $0 | \n\nDelta updates reduce the embedding bill and not the crawl bill, because by default every run fetches every page. Once embeddings drop toward zero, the crawl is nearly the whole cost, and it buys nothing on a corpus that hasn't changed.\n\nOn a corpus this size, the entire delta saving is the $0.003 embeddings row. Delta updates leave the collection whole, where a full rebuild empties it first. A larger corpus raises both the bill and the time spent reloading.\n\n## Run your own n8n RAG pipeline twice\n\nMeasure your own change interval before you choose a schedule. Fix any crawl that returns a different page count each run before you change the expiry window.\n\nThe fastest check is to point [Website Content Crawler](https://apify.com/apify/website-content-crawler) at your own documentation and let the pipeline run twice. The free Apify plan is enough for that. Either the count is unchanged and nothing was re-embedded, or `Freshness report` names what changed.\n\n## FAQ\n\n### Do repeated inserts create duplicates?\n\nYes. Any code that writes without an explicit ID produces a new random UUID for each chunk, so the same text is stored twice. Running 86 pages twice through the integration Actor on its `add` strategy produced 602 vectors and then 1,204. On `deltaUpdates`, the same 2 runs left the count at 602.\n\n### Can n8n delete vectors from a vector database?\n\nNo vector store node in n8n exposes a delete operation, checked against version 2.37.5 of the LangChain nodes package. Update by ID works on 5 of the 13 root vector store nodes. To remove records, use the database client directly, empty a namespace on insert, or let an integration Actor manage expiry.\n\n### Do I need to re-embed everything when one page changes?\n\nNo. The integration Actor hashes each page into a checksum and compares it against what is stored. Editing a single page of 86 produced 11 chunks added, 8 removed, and 594 refreshed. Only the edited page reached the embedding model, and its old chunks were removed in the same pass.\n\n### How often should I re-crawl for RAG?\n\nMeasure it rather than guessing: crawl twice a few days apart and compare the text per URL. One documentation corpus here had no edited pages across 4 days. It gained a single new page in that time, so 3 of those 4 nights would have found nothing.\n\n### Why did my crawl return fewer pages than the site has?\n\nCheck whether the start URL redirects. Seeding a redirecting URL returned between 10 and 86 pages across 8 identical runs here, and only 1 of those runs reached the full site. Seed the post-redirect target instead. Crawling it returned 86 pages on all 3 runs here, with identical URL sets.", "url": "https://wpnews.pro/news/build-a-live-rag-pipeline-with-apify-n8n-and-qdrant", "canonical_source": "https://blog.apify.com/live-n8n-rag-pipeline/", "published_at": "2026-09-16 09:12:00+00:00", "updated_at": "2026-09-16 09:43:13.447956+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "developer-tools", "ai-products"], "entities": ["Apify", "n8n", "Qdrant", "Website Content Crawler", "Apify Trigger", "Qdrant Cloud"], "alternates": {"html": "https://wpnews.pro/news/build-a-live-rag-pipeline-with-apify-n8n-and-qdrant", "markdown": "https://wpnews.pro/news/build-a-live-rag-pipeline-with-apify-n8n-and-qdrant.md", "text": "https://wpnews.pro/news/build-a-live-rag-pipeline-with-apify-n8n-and-qdrant.txt", "jsonld": "https://wpnews.pro/news/build-a-live-rag-pipeline-with-apify-n8n-and-qdrant.jsonld"}}