Build a live RAG pipeline with Apify, n8n, and Qdrant

Apify published a guide for building a live retrieval-augmented generation (RAG) pipeline using Apify's Website Content Crawler, self-hosted n8n, and Qdrant that re-embeds only changed pages and removes deleted pages from the vector index. The guide notes that n8n 2.37.10 still lacks built-in vector store record management, a gap a community member confirmed in December 2024, so the common workaround is deleting the entire collection before every load. The 7-node workflow runs on a webhook triggered when an Apify crawl finishes, and the build takes about an hour once accounts exist.

Your crawler runs, the chunks reach the vector store, and the chatbot answers. Then, the pricing page changes. A product page is deleted. It keeps answering from the chunks it stored last week, confidently and wrongly, and nothing reports an error. The crawl ran again. Nothing removed the old chunks or replaced the changed ones. This guide builds an n8n RAG pipeline that does both. Website Content Crawler https://apify.com/apify/website-content-crawler crawls the site into an Apify dataset. It's an Actor, the Apify term for a ready-made cloud program that you configure and run rather than build. A webhook starts n8n when the crawl finishes, and a second Actor chunks the pages, embeds them, and writes them into Qdrant. Only the pages that changed are re-embedded, and pages deleted from the site are removed from the index. The workflow runs on self-hosted n8n. This build takes about an hour once the accounts exist, and longer if you also need a tunnel in front of n8n. You need to know how to set environment variables and run n8n yourself. What goes wrong on the second run On the second run, 3 things happen. The crawler returns the same pages, and the vector store node inserts a second copy of each. A page that changed since the first crawl now has 2 versions stored, and search returns both. Nothing removes a page deleted from the site, so its chunks remain. No log line marks any of it. Answer quality drops instead, which is much harder to notice. How retrieval-augmented generation works https://blog.apify.com/what-is-retrieval-augmented-generation/ is well documented. Keeping that index correct over time isn't. The n8n community forum has an open thread asking for record management in vector stores https://community.n8n.io/t/vector-store-record-management-in-n8n/66833 . The answer from a community member in December 2024 was direct: there's no built-in record management feature for vector databases in n8n. That still holds in n8n 2.37.10, and Step 3 has the per-node breakdown. The workaround people use is to delete the whole collection before every load. That workaround works, and on a small site it's enough. On a large one it's expensive, and the collection is only partly loaded while the rebuild runs. What you will build The alternative to deleting the whole collection every night is to compare each page against what is already stored and write only the difference. The workflow has 7 nodes: php Apify Trigger - Vectors before sync - Sync dataset into Qdrant - Let the sync settle - Vectors after sync - Read sync log - Freshness report Here’s the n8n editor view: Only 3 of the 7 nodes do the work. The other 4 exist to prove it happened: the counts on either side of the sync, plus Read sync log and Freshness report . You never select Run on this workflow. You schedule the crawl instead, on the Apify side. When the crawl finishes, the webhook starts n8n and the 7 nodes run in order. The report shows what changed on each run. The dataset in the middle is a saved copy. You can inspect what the crawl collected and compare it against the previous night. Replaying it into the database costs no new crawl. Before you start You need 4 accounts or installs, and 3 of them are free at this scale: - An Apify account. A free plan includes a monthly platform allowance and needs no credit card. Current limits are on the Apify pricing page https://apify.com/pricing . - n8n, self-hosted. The Community edition is free to run on your own hardware under the n8n fair-code license. Check the n8n pricing page https://n8n.io/pricing/ for current terms. A self-hosted instance also has to be reachable from the public internet, which is what lets Apify deliver the webhook in Step 2. On a laptop that means putting a tunnel in front of n8n, so arrange it now. - A Qdrant Cloud cluster. The free plan includes 1 GB RAM and 4 GB disk, which is far more than this corpus needs. Current limits are on the Qdrant pricing page https://qdrant.tech/pricing/ . - An OpenAI API key for embeddings. This is the paid item, and embedding this corpus cost about $0.003 at list price on the first run. Set the environment variables QDRANT API KEY and OPENAI API KEY come from the n8n environment. The workflow passes OPENAI API KEY inside a JSON body to an Actor. It uses QDRANT API KEY both there and as an api-key header on both count nodes. Set both where your instance reads its environment. That's a -e flag on docker run , the environment: block in Docker Compose, or a shell export for an npm install. Then set N8N BLOCK ENV ACCESS IN NODE=false explicitly. On n8n 2.x that switch blocks expressions from reading the environment unless it's set to the string false . Leave it unset and every $env read returns access to env vars denied . All 3 go in the same place, and n8n reads the environment at startup, so restart it after any change. The switch is instance-wide and applies to everything or to nothing. It doesn't grant this workflow access to those 2 variables. It grants every expression in every workflow on that instance access to the whole process environment. A production install usually holds more than 2 API keys. It commonly holds the database password, and N8N ENCRYPTION KEY where the operator sets that explicitly rather than letting n8n generate one. That key decrypts every credential that the instance stores. Set the switch only where every workflow author is already trusted with everything in that environment. On a shared instance, put the Qdrant key in a Header Auth credential on both count nodes instead. That removes 2 of the 4 $env reads, and a credential needs no environment access. Install the Apify community node A new n8n install doesn't include the Apify community node. Add it under Settings Community Nodes with the package name @apify/n8n-nodes-apify . Do not run npm install in the nodes folder yourself. n8n strips the package's peer dependencies and passes --ignore-scripts before installing. A plain npm install pulls n8n-workflow instead, then fails trying to compile isolated-vm against the image's Node version. The Apify Trigger node needs a credential, and so do the 2 HTTP Request nodes that call the Apify API. Copy a token from the API & Integrations https://console.apify.com/settings/integrations page in Apify Console, and in n8n create a credential of type Apify account with it. Creating the credential and attaching it to a node are separate steps, and it's common to do the first and forget the second. Miss either step and nothing shows an error where you're looking. The workflow publishes, and the trigger then fails to start. The failure appears in a retry loop visible only in the n8n log, first as Unrecognized node type: @apify/n8n-nodes-apify.apifyTrigger . Once the node is present but no credential is attached, it fails as No valid credentials found for apifyApi . Import the workflow n8n takes live-rag-pipeline.n8n.json https://gist.github.com/triposat/f493457342214fb41b71fa80a3f74c16 directly through Import workflow from URL , using the raw link to that file. It needs 2 edits now: 1. Replace the placeholder cluster URL in 3 places. This is a plain URL rather than a credential, so n8n never prompts you for it. Your cluster's address is on its overview page in Qdrant Cloud, labeled Endpoint . Copy it with the :6333 port at the end. The 3 places are: - The url field on Vectors before sync - The url field on Vectors after sync - The qdrantUrl line inside the body of Sync dataset into Qdrant 2. The 3. Attach your Apify credential to the 3 nodes that use it. Step 1: Crawl the site into a dataset Web scraping for RAG has to produce clean text rather than whole pages. Website Content Crawler apify/website-content-crawler crawls a site and strips navigation, footers, and modals before the text reaches you, which is what a text splitter needs. It writes a cleaned text field on each item that it stores, and that's the field that this pipeline embeds. You run a single crawl by hand here. There are 3 settings that control what it costs and how much of the site it reaches, and each one needs a decision from you. Set the crawler type and memory explicitly Apify input schemas carry 2 separate values for a field, and the input schema specification https://docs.apify.com/platform/actors/development/actor-definition/input-schema/specification/v1 defines both. A prefill "is only used in the user interface." A default applies whenever no value is given "via any means API, CLI, scheduler, or user interface ." For crawlerType those 2 values differ, and the half that costs you money is the one you never see. The schema default has stayed at playwright:firefox through every build change. Any run that names no crawler opens a headless browser on every page, whether the caller is an n8n node, a schedule, or the API. The prefill is the unstable half. It changed twice in 2 days across builds 0.3.96 and 0.3.97, in both directions. So name crawlerType explicitly in your input, and the question stops mattering. Leaving it unnamed has a measurable cost. These 3 runs crawled the same 4 pages of Apify documentation at 4,096 MB on September 4, 2026: | Run | crawlerType | Handler used | Runtime | Cost | |---|---|---|---|---| | A | not set | browser on 4 of 4 pages | 95.9s | $0.02415 | | B | cheerio | HTTP on 4 of 4 pages | 16.1s | $0.00579 | | C | playwright:adaptive | browser on 4 of 4 pages | 110.6s | $0.02787 | Compared with run B, the unset default cost 4.17 times as much and adaptive cost 4.81 times as much. Adaptive runs rendering-type detection and then still opens a browser. Memory is the second setting that you can change. Website Content Crawler declares memoryMbytes: 8192 in its default run options, so a caller that passes no memory value runs at 8 GB. Apify bills compute units as memory multiplied by runtime, which makes that a direct multiplier on the crawl bill. Set it explicitly. Check the start URL before you schedule anything In early September 2026 https://docs.apify.com/integrations redirected to /platform/integrations . Seeding the redirecting URL and running the same configuration 8 times returned 10, 28, 41, 44, 45, 49, 53, and 86 pages. Only 1 of those 8 runs reached the full site. Seeding the post-redirect URL instead returned 86 pages on 3 runs, with identical URL sets. Apify has since removed that redirect. The check still applies, because any seed URL can start redirecting at any time: curl -sI <https://your-site.example/docs | grep -i '^location:' If that prints a location header, seed the target instead. A crawl that silently returns a third of the site means you can only answer a third of the questions. Nothing in the run log says so. Run the crawl once This input goes to the Actor itself, not into n8n. Save it in an Actor task https://docs.apify.com/platform/actors/running/tasks in Apify Console. That's also where you set memory, because memory is a run option rather than an input field. The input is: { "startUrls": { "url": "https://docs.apify.com/platform/integrations" } , "crawlerType": "cheerio", "maxCrawlPages": 500, "maxCrawlDepth": 2, "aggressivePrune": true } The example seeds the Apify documentation site, and every crawl measured in this guide ran against it. Point the crawl at a site you can edit if you want to test freshness in Step 6. maxCrawlPages and maxCrawlDepth are limits rather than targets. At 500 pages and 2 levels they're already more than this documentation site needs. Raise them for a larger site, and keep them low enough that a misconfigured crawl stops early. aggressivePrune is on because it removes lines that repeat across pages. On this crawl it left the page count unchanged and reduced total text by 2.1%. Line instances repeated across 20 or more pages dropped from 173 to 0. Set the run's memory to 1,024 MB in the same place. On the example site that input returned 86 pages for $0.00809. Run it once now. The trigger in Step 2 lists only Actors that your account has run, so this crawl puts Website Content Crawler in that list. That run leaves its pages in a dataset, which is what n8n reads next. Step 2: Trigger n8n when the crawl finishes A webhook fires the moment the crawl finishes. A schedule trigger in n8n runs at fixed times, so it fires whether or not the crawl has finished. The imported workflow already holds the Apify Trigger node, set to On new Apify Event . Resource to Watch is on Actor and Event Type on Succeeded. Open that node and leave Actor Source on Recently Used Actors. Re-select Website Content Crawler from the Actor dropdown, even though a value is already stored. That dropdown writes a resource locator rather than a plain string, and the node reads the object it writes. If the field holds only an ID, the node registers no webhook and publishes without an error, so it never fires. Other nodes reference the trigger by name inside expressions, so if you rename it, change those expressions too. Make n8n reachable from the internet The URL is validated at the moment the webhook is created, so it has to be an address Apify can reach before you publish. A local URL returns this: Invalid value provided in webhook: Webhook requestUrl must be a valid URL. Received "http://localhost:5678/webhook/..." On a local install, put a tunnel in front of n8n first. ngrok, Cloudflare Tunnel, and Tailscale Funnel all work. Then set WEBHOOK URL to that public address, in the same place as the other variables. n8n then sends the tunnel address rather than localhost when it registers the webhook. Publishing before the tunnel is running creates nothing. Publish the workflow On n8n 2.x the control is labeled Publish , and a live workflow shows as Published. The same control was called Active on 1.x, and the CLI still accepts the older update:workflow --active alongside the current publish:workflow . Publishing creates a webhook in your Apify account through POST /v2/webhooks . Unpublishing deletes it. So an unpublished workflow is the first thing to check when a trigger does nothing. Confirm the webhook exists A published workflow can still have no webhook, and n8n reports success either way. Read the webhook list directly: curl -s -H "Authorization: Bearer $APIFY TOKEN" \ "https://api.apify.com/v2/webhooks" \ | grep -o '"requestUrl": " ^" "' Export APIFY TOKEN first, rather than pasting the token inline. Send it as a header rather than in the query string, because a token in a URL is written to server-side access logs. Treat what this prints as a secret, too. Unless you put authentication in front of it, the webhook URL is the only thing protecting your pipeline from the public internet. Do not post it in forum threads or screenshots. That list should hold a webhook pointing at your n8n instance. An empty list means the trigger registered nothing, which is the resource locator problem. Run another crawl and confirm an execution appears in n8n, not only that Apify reports a delivery. Expect that execution to stop at the sync node, because the integration Actor needs a one-time permission approval that Step 4 explains. The trigger passes the whole Actor run object into n8n. The field that the rest of the pipeline needs is resource.defaultDatasetId , and n8n now receives it on every finished crawl. Step 3: Check what the vector store nodes can do The first thing to try is an n8n vector store node in Insert Documents mode. Let the workflow run twice that way and the vector count doubles. That's not a misconfiguration. The Qdrant vector store binding comes from LangChain rather than n8n. It assigns a random UUID to every chunk that arrives without an explicit ID, and n8n passes no IDs in insert mode. Every run, therefore, writes new points for text that already exists. Update Documents is the next thing most people try. In the bundled @n8n/ai-utilities package, version 0.30.4, that mode has no text splitter and requires each item to resolve to exactly one document. Those 2 limits make it a one-vector-per-document upsert. It suits short records with a stable ID. It isn't a way to maintain chunked web pages. Insert, update, and delete across the 13 nodes These counts come from the @n8n/n8n-nodes-langchain package at version 2.37.5, bundled in n8n 2.37.10: | Capability | Nodes | |---|---| | Insert | 13 of 13 root vector store nodes | | Update by ID | 5 of 13: Azure AI Search, MongoDB Atlas, Pinecone, Redis, and Supabase | | Delete | 0 of 13 | The default operation set in the shared node factory is 'load', 'insert', 'retrieve', 'retrieve-as-tool' . Update is opt-in, and the code comment for that mode says most providers omit it. There's one more thing to check before you build the write path from these nodes. On the n8nio/n8n:latest image at 2.37.10, which includes Node v26.5.1, the Qdrant Vector Store node failed with fetch failed . The same container reached the database over plain HTTP with no error. The bundled @qdrant/js-client-rest 1.16.2 raises invalid onError method on that Node version. So the n8n vector store nodes are built around insert, and keeping the vector database up to date belongs somewhere else. Step 4: Hand the write path to an integration Actor Both halves of this pipeline are ready-made Actors from Apify Store https://apify.com/store , and the marketplace holds thousands more. Apify publishes integration Actors for Chroma, Milvus, OpenSearch, PGVector, Pinecone, Qdrant, and Weaviate. The code is public in actor-vector-database-integrations https://github.com/apify/actor-vector-database-integrations . Qdrant Integration https://apify.com/apify/qdrant-integration apify/qdrant-integration takes a dataset ID and syncs it into a collection. It identifies each page by a primary field, hashes the content into a checksum, and compares that checksum against what is already stored. The Actor only refreshes last seen at on unchanged chunks, so it doesn't re-embed them. That only works while the chunking and embedding settings stay the same between runs. Approve the permissions once These Actors read your datasets and write to your storage, so the first run needs a one-time permission approval. The approval is account-wide rather than scoped to a single dataset or collection, so read it before you accept. The API holds the run until you approve: full-permission-actor-not-approved This Actor requires full access to your account. You must approve its permissions before running it. Approve it once on the Actor page in Apify Console. The error is the same whether you start the Actor from n8n, the API, or a schedule. Inside the HTTP Request node In the workflow, this is an HTTP Request node named Sync dataset into Qdrant , with your Apify credential attached. It POSTs to: https://api.apify.com/v2/acts/tqzqfIiXvKWCpbRiv/runs?timeout=900&memory=1024&waitForFinish=60 memory=1024 names the Actor's own default explicitly, so a later build change can't raise what the sync costs. timeout=900 stops a stuck sync after 15 minutes rather than at the Actor's 1-hour default. waitForFinish=60 holds the response open until the run finishes, up to the API maximum of 60 seconds. The counts later in the workflow are then read after the sync rather than during it. A sync longer than 60 seconds returns while still running, and those counts aren't final. The body is one n8n expression, not plain JSON. The JSON body field is in expression mode and its value starts with = . Inside JSON.stringify the values are JavaScript, so $env.QDRANT API KEY is a reference rather than a quoted string: ={{ JSON.stringify { qdrantUrl: 'https://YOUR-CLUSTER.qdrant.io:6333', qdrantApiKey: $env.QDRANT API KEY, qdrantCollectionName: 'docs', qdrantAutoCreateCollection: true, datasetId: $ 'Apify Trigger' .first .json.resource.defaultDatasetId, datasetFields: 'text' , metadataDatasetFields: { url: 'url', title: 'metadata.title' }, embeddingsProvider: 'OpenAI', embeddingsApiKey: $env.OPENAI API KEY, embeddingsConfig: { model: 'text-embedding-3-small' }, dataUpdatesStrategy: 'deltaUpdates', dataUpdatesPrimaryDatasetFields: 'url' , deleteExpiredObjects: true, expiredObjectDeletionPeriodDays: 30, performChunking: true, chunkSize: 1000, chunkOverlap: 200 } }} Drop the = or quote the expressions and the Actor receives the literal text {{ $env.QDRANT API KEY }} as your API key. Check the fields that change the result dataUpdatesStrategy is the one line that changes what the pipeline does. Set to deltaUpdates , it compares checksums and re-embeds what changed. Set to add , it appends everything on every run. The Actor requires qdrantUrl , qdrantCollectionName , embeddingsProvider , embeddingsApiKey , and datasetFields . qdrantApiKey isn't on that list, but any Qdrant Cloud cluster requires authentication on writes, so treat it as required too. The dataset reference points at the trigger by node name rather than at $json , because at that stage $json is whatever the previous node emitted. embeddingsConfig names the embedding model that you want. It's optional, and the underlying embeddings class uses text-embedding-ada-002 when no model is named. That model costs $0.10 per 1M tokens against $0.02 for text-embedding-3-small at published OpenAI prices https://platform.openai.com/docs/pricing . Both produce 1,536 dimensions, so a run on the wrong model completes normally at 5 times the price. performChunking , chunkSize , and chunkOverlap control how each page is split before embedding. The values here suit documentation prose, where a single answer usually sits inside one or two paragraphs. Changing them later invalidates every stored checksum, so choose values before the first scheduled run. dataUpdatesPrimaryDatasetFields names the field that identifies a page, and the checksum comparison uses that field as its key. It's what separates an update from a duplicate. Here that field is url , so the same page reachable at 2 addresses would count as 2 pages. metadataDatasetFields copies fields onto the stored chunk, so a retrieved result can be traced to its source URL. Know how your keys are stored The Actor declares qdrantApiKey and embeddingsApiKey as secret fields, so Apify stores them encrypted. A run's saved input shows ENCRYPTED VALUE: in place of each key. The rest of the input, including your cluster URL and collection name, is stored as written. Scope the OpenAI key to embeddings and the Qdrant key to this collection anyway. Step 5: Prove the second run The proof comes from 4 nodes around the sync. Vectors before sync sits ahead of it, and Vectors after sync , Read sync log , and Freshness report sit after. Both counts are HTTP GETs against /collections/docs on your cluster, each carrying an api-key header set to ={{ $env.QDRANT API KEY }} . Only the first continues on error. The collection doesn't exist before the first run, so Vectors before sync has to tolerate a 404. The sync node creates the collection on that first pass. Vectors after sync must not continue on error, because the report reads a missing count as 0. Make them consistent and a temporary read failure during the sync produces a report that shows an empty collection. Stopping there is the safer failure. Between them, Let the sync settle is a Wait node set to 30 seconds. It gives the Qdrant point count time to update after the Actor reports that it finished. The file sets its unit explicitly, because the Wait node defaults to hours and a number with no unit leaves the run waiting overnight. waitForFinish stops at 60 seconds, and the first pass over 86 pages took 35.4s. On a larger corpus the request returns while the sync is still running. Freshness report therefore checks the run status first, and if the sync hasn't finished it reports that the counts aren't final. Read sync log fetches the Actor run log, and Freshness report parses one line from it: js const raw = $ 'Read sync log' .first .json; const text = typeof raw === 'string' ? raw : raw?.data ?? JSON.stringify raw ; const m = String text .match /to add: \d+ , to update last seen at: \d+ , to delete: \d+ / ; The report's conclusion comes from those 3 captures rather than from the point count. A run that re-embeds 5 changed chunks and expires 5 old ones leaves the count unchanged. Compare insert against delta updates Both strategies ran on 2 crawls of the same 86 pages, into Qdrant 1.19.1: | Strategy | After crawl 1 | After crawl 2 | |---|---|---| | add | 602 vectors | 1,204 vectors | | deltaUpdates | 602 vectors | 602 vectors | On the second delta pass, the Actor reported what it did: Objects: to add: 0, to update last seen at: 602, to delete: 0 Nothing reached the embedding model on that run. The run finished in 17.1s for $0.00053, against 35.4s and $0.00119 for the first pass over the same 86 pages. Those runs changed nothing. The next test started from an empty collection, then edited a single page of the same 86 and left the other 85 untouched. That page's text grew from 6,229 to 8,146 characters: 86 pages, first load to add: 602 to update last seen at: 0 to delete: 0 602 points same 86, one page edited to add: 11 to update last seen at: 594 to delete: 8 605 points The edited page held 8 chunks and now holds 11. The old 8 were removed in the same pass rather than after the 30-day expiry window. The collection never held 2 versions of that page. The other 85 pages, 594 chunks between them, were refreshed without reaching the embedding model. Set the expiry window from your crawl history deleteExpiredObjects defaults to true, and expiredObjectDeletionPeriodDays defaults to 30. Deletion is time-based and scoped to the collection, not to the current crawl. Anything whose last seen at is older than the cutoff is removed. A page deleted from the site stops appearing in crawls, so its chunks expire and the sweep removes them. Pair that with the crawl variation from Step 1 and a failure mode appears. This test used the 86-page crawl as night 1 and a partial 28-page crawl as night 2. The expiry sweep then ran with a window short enough for the night 1 pages to have expired: night 1: full 86-page crawl pages=86 chunks=602 vectors in db=602 night 2: partial 28-page crawl pages=28 chunks=194 vectors in db=604 after expiry sweep vectors in db=194 The 2 extra vectors on night 2 are chunks whose checksum matched nothing already stored. Of the 194 that remained, 192 were originals. The sweep removed 410 of the first 602, 68.1%, for pages that still exist on the site. This doesn't usually happen on the 30-day default, because every crawl has to miss the page for a month before it expires. Shortening the window to match a nightly crawl removes that protection. Fix the crawl so it returns the same pages every run. Then set the window longer than the longest series of incomplete crawls that you're willing to accept. Step 6: Query the index Reading is the job that suits the n8n vector store node here, because it needs neither update nor delete. Add an AI Agent node and attach the vector store in Retrieve Documents As Tool for AI Agent mode. Connect a Chat Trigger to test it. These are nodes that you add yourself: the workflow file handles ingestion only. If you're on the image described in Step 3, the Qdrant node fails here for the same transport reason that it fails on writes. Query the collection over plain HTTP, as the count nodes do. Testing freshness needs a page you can edit. Change a sentence, re-run, and ask a question that only the new sentence answers. A question you can answer from either version proves nothing. If you can't edit the source, the Freshness report is the check. For where retrieval sits in a larger system, see how web data flows into agent retrieval https://blog.apify.com/ai-agent-infrastructure/ . Step 7: Schedule it Schedule the crawl on the Apify side with Actor schedules https://docs.apify.com/platform/actors/running/schedules , which take a cron expression and are timezone aware. The webhook then starts n8n whenever a run finishes, so you're not maintaining 2 schedules that can stop matching. Add a second Apify Trigger on the Failed event so a failed crawl reaches you rather than quietly skipping a night. Then measure your corpus's change interval instead of guessing at a schedule. Run the crawl twice, a few days apart, and compare the page text per URL: | Window | Pages unchanged | Changed | Added | |---|---|---|---| | September 4 to September 7 | 86 of 86 | 0 | 0 | | September 7 to September 8 | 86 of 86 | 0 | 1 | No page was edited in 4 days. A single new page appeared, which raised the crawl from 86 pages to 87. A nightly schedule on this corpus fetches all 86 pages again every night to learn that. Match the schedule to the interval that you measured rather than to a habit. Weekly on a corpus that changes weekly costs a seventh as much as nightly, as long as nothing urgent appears between crawls. One further setting would reduce the crawl bill. Website Content Crawler passes customHttpHeaders to the target site, so an If-Modified-Since header turns an unchanged crawl into a 304. In one before-and-after pair, that reduced the crawl from $0.01089 to $0.00151. A conditional crawl of an unchanged site returns no items. To the expiry deletion in Step 5, a run of no items looks exactly like a failed crawl. On a corpus stable enough to be worth crawling conditionally, 30 such nights in sequence put every point past the 30-day last seen at cutoff. Do not run conditional crawling and expiry deletion together until you have tested that combination against your own collection. What the n8n RAG pipeline costs to run Figures come from measured runs on a free Apify plan in September 2026, plus published list prices for the other components. | Component | Cost | |---|---| | Crawl, 86 pages, HTTP crawler at 1,024 MB | $0.00809 per run | | Conditional request, untested against expiry deletion | $0.00151 per run | | Browser crawler, measured on 4 pages at 4,096 MB | 4.17 times the HTTP crawler on the same 4 pages | | Embeddings, first run, 602 chunks | about $0.003 at list price | | Embeddings, later runs with delta updates | $0 when nothing changed | | Qdrant Cloud free cluster | $0 | | n8n Community, self-hosted | $0 | Delta updates reduce the embedding bill and not the crawl bill, because by default every run fetches every page. Once embeddings drop toward zero, the crawl is nearly the whole cost, and it buys nothing on a corpus that hasn't changed. On a corpus this size, the entire delta saving is the $0.003 embeddings row. Delta updates leave the collection whole, where a full rebuild empties it first. A larger corpus raises both the bill and the time spent reloading. Run your own n8n RAG pipeline twice Measure your own change interval before you choose a schedule. Fix any crawl that returns a different page count each run before you change the expiry window. The fastest check is to point Website Content Crawler https://apify.com/apify/website-content-crawler at your own documentation and let the pipeline run twice. The free Apify plan is enough for that. Either the count is unchanged and nothing was re-embedded, or Freshness report names what changed. FAQ Do repeated inserts create duplicates? Yes. Any code that writes without an explicit ID produces a new random UUID for each chunk, so the same text is stored twice. Running 86 pages twice through the integration Actor on its add strategy produced 602 vectors and then 1,204. On deltaUpdates , the same 2 runs left the count at 602. Can n8n delete vectors from a vector database? No vector store node in n8n exposes a delete operation, checked against version 2.37.5 of the LangChain nodes package. Update by ID works on 5 of the 13 root vector store nodes. To remove records, use the database client directly, empty a namespace on insert, or let an integration Actor manage expiry. Do I need to re-embed everything when one page changes? No. The integration Actor hashes each page into a checksum and compares it against what is stored. Editing a single page of 86 produced 11 chunks added, 8 removed, and 594 refreshed. Only the edited page reached the embedding model, and its old chunks were removed in the same pass. How often should I re-crawl for RAG? Measure it rather than guessing: crawl twice a few days apart and compare the text per URL. One documentation corpus here had no edited pages across 4 days. It gained a single new page in that time, so 3 of those 4 nights would have found nothing. Why did my crawl return fewer pages than the site has? Check whether the start URL redirects. Seeding a redirecting URL returned between 10 and 86 pages across 8 identical runs here, and only 1 of those runs reached the full site. Seed the post-redirect target instead. Crawling it returned 86 pages on all 3 runs here, with identical URL sets.