{"slug": "ai-agent-infrastructure-stack-from-web-data-to-retrieval", "title": "AI agent infrastructure stack: from web data to retrieval", "summary": "Apify's July 2026 measurements show that AI agent failures often stem from the data layer, not the model, with four recurring issues: fetch failures, stale knowledge, and retrieval gaps. The company recommends a stack using its Website Content Crawler, Pinecone Integration, and RAG Web Browser to handle extraction, transformation, storage, and retrieval, while noting that modern bot defenses and Cloudflare's pay-per-crawl model add economic and identity challenges to web data access.", "body_md": "Your agent answers a question about your own product and gets it wrong. The model is fine. It answers from training data, and your product has changed since then.\n\nSo you add retrieval: crawl the docs, embed the pages, store the vectors. The page that your agent needed was last crawled three weeks ago, or it came back as an empty JavaScript shell. You're paying to re-embed content that hasn't changed, and the answers cite URLs that no longer exist. Nothing warned you about any of it.\n\nThese aren't model problems. Every one of them lives in the data layer of the AI agent infrastructure stack: the part that gets web content into your agent and keeps that content fresh.\n\nBuild that layer once, and measure it. The work splits into four jobs: extraction, transformation, storage, and retrieval. Three Apify Actors cover most of the pipeline.\n\n[Website Content Crawler](https://apify.com/apify/website-content-crawler) extracts clean text and handles the fetch problem. [Pinecone Integration](https://apify.com/apify/pinecone-integration) chunks the text, embeds it, and upserts the vectors, skipping pages that haven't changed. [RAG Web Browser](https://apify.com/apify/rag-web-browser) covers the retrieval fallback: the questions that the index cannot answer.\n\nThis walkthrough builds the data layer end to end. Every cost and latency figure below comes from measured July 2026 runs.\n\n## What the AI agent infrastructure stack looks like\n\nThe full stack, from the bottom up:\n\n**Models and inference**- a hosted API or a self-hosted model.** Orchestration**- the framework that runs the agent loop and decides when to call tools. This layer holds[AI agent frameworks](https://blog.apify.com/best-ai-agent-frameworks/)like LangGraph, CrewAI, Mastra, and the OpenAI Agents SDK.**Memory**- conversation history, scratchpads, and state that persists between sessions.** Tools and data**- how the agent reaches the outside world: web data, APIs, and the knowledge it retrieves to ground its answers.** Observability and governance**- tracing, evals, audit trails, and permissions.\n\nThe orchestration layer gets most of the attention, but the retrieval and the answer call are the same whichever framework you pick. The tools and data layer decide where the agent's knowledge comes from, how fresh it is, and what it costs to keep it that way. That is the data layer.\n\nPinecone stores the vectors and serves queries. The [Apify MCP server](https://mcp.apify.com/) adds a second retrieval path for what the index doesn’t cover.\n\n## Why the data layer is where agents break\n\nFour failure modes appear repeatedly in production agent systems.\n\n**Fetch failure, not parse failure.** The hard part is not parsing the page. It is fetching the page at all. Modern sites return HTTP 200 with a challenge page, an empty JavaScript shell, or a \"please verify you are human\" screen. A scraper passes testing, then starts returning unusable text silently.\n\nProduction scraping is an infrastructure problem that needs proxies, browser fingerprints, retries, and session rotation. The difficulty is that modern bot defenses don’t rely on a single signal. They score the whole request: IP reputation, TLS fingerprint, header ordering, and how the page behaves once JavaScript runs. A mismatch between any two of those signals can be enough to block the request.\n\nThat is why a better proxy alone rarely fixes a block.\n\nThe problem now has an economic side too. Cloudflare sorts crawlers into search, agent, and training categories, and verifies agents through cryptographically signed requests under the [Web Bot Auth proposal](https://datatracker.ietf.org/wg/webbotauth/about/). Cloudflare has also announced [pay-per-crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/), which lets sites charge AI crawlers per page. Getting the page is now a question of identity and billing, not just of engineering.\n\n**Stale knowledge.** The failure is silent. A stale index still returns high-scoring matches and still attaches citations. An answer looks exactly the same whether it was built on year-old content or on today's content.\n\nThe pipeline below stamps every chunk with a last-seen timestamp. The timestamp reaches the model only if you add it to the metadata you retrieve. The fix is scheduled re-crawls, which introduce a cost of their own.\n\n**Wasted re-embedding.** A naive refresh pipeline re-embeds the entire corpus on every crawl. Most pages don’t change between crawls, so most of that cost is waste. Re-embed only the pages that actually changed, and the waste drops to near zero. Changing your chunking strategy or embedding model is the exception, because that forces a full re-index.\n\n**Retrieval count creep.** Raise retrieval from the top 3 chunks to the top 8, and you more than double the input tokens you pay for on every call. The cost is easy to miss because no single query looks expensive.\n\nThe architecture below addresses the first three failure modes. The fourth is a practice rather than a component.\n\n## A persisted index first, live retrieval as the fallback\n\nFor most agents, the default is a **persisted vector index**. You crawl your knowledge sources, embed them, store the vectors in a vector database, and let the agent query that index at runtime. This approach works because most of what an agent needs to know is bounded and relatively stable: product docs, knowledge bases, catalogs, and internal wikis.\n\nAn index lookup costs read units, Pinecone's billing unit for a query. In the runs below, each lookup cost 1 read unit, and a warm lookup returned in roughly 0.4 seconds. You also control exactly what is in the index.\n\n**One question now comes before the index.** Model context windows are large enough that a small corpus can skip retrieval entirely. The 24 crawled pages here total about 18k tokens and would fit in one cached prompt.\n\nWe kept retrieval for three reasons. Top-5 retrieval sends about 1.5k tokens into the model per query instead of the whole corpus. The update pipeline keeps knowledge fresh without rebuilding a prompt. Per-chunk metadata gives every answer a source to cite.\n\nWhen your corpus is a hundred times the context window, the question disappears. When it is small enough to fit, run the token math before you build anything.\n\nThe index doesn’t cover fresh events, long-tail questions, or sources you never crawled. For those, you add **live web retrieval**, a tool that the agent calls at query time to search the web, fetch the top results, and return clean Markdown.\n\nIn the measured runs, a 3-result query in raw HTTP mode cost $0.005 and took 10.4 seconds, so live retrieval runs only when the index misses. It also puts pages you never chose into the model's context. That is a security decision.\n\n**The pipeline is one ingestion path and two retrieval paths.** The agent tries the index first and calls the live tool when the index returns nothing useful. You implement that routing either as a system-prompt instruction plus tool descriptions, or as a similarity-score threshold on the index results.\n\nThe threshold approach is the more predictable of the two.\n\nIt is predictable in one direction only, though. A threshold catches an index that returns nothing useful, not an index that returns the wrong page.\n\n## The reference architecture: from crawl to query\n\nBelow, the worked example indexes a documentation site and gives an agent both retrieval paths. Most of the pipeline runs on Apify, and you supply the vector database and the agent code.\n\n### What you need\n\nYou need four things for the walkthrough below:\n\n- An\n[Apify account](https://console.apify.com/sign-up)on the free plan, and its[API token](https://docs.apify.com/integrations/api)from Apify Console under**Settings**, then** API & Integrations**. - A\n[Pinecone account](https://app.pinecone.io/)and its API key. The index itself is created in code, with 1536 dimensions and the cosine metric. The dimension matches what`text-embedding-3-small`\n\noutputs. - An OpenAI API key, used both for the embeddings and for the answer call at the end of the walkthrough.\n- Python 3.11 or newer and the client libraries, installed with\n`pip install \"apify-client>=3\" pinecone openai`\n\n. The version floor matters. The crawl snippet reads attributes from a Run object, and the older 2.x client returns a plain dictionary instead.\n\nThe three steps that follow are one script, with the dataset from step 1 feeding step 2.\n\nApify and Pinecone both have free tiers that cover this walkthrough. The whole walkthrough stays well inside Apify's $5 of free monthly credit. OpenAI's API needs credit on the account, and the embedding cost here stays under a cent. The three walkthrough steps took under two minutes of compute in total, so most of your time will go to account setup, not to the pipeline.\n\nEvery snippet reads its credentials from the environment, so export them once:\n\n```\nexport APIFY_TOKEN=...  PINECONE_API_KEY=...  OPENAI_API_KEY=...\n```\n\nThe snippets below need no other credentials.\n\n**Create the index before the first load.** This runs once rather than on every crawl:\n\n``` python\nimport os\nfrom pinecone import Pinecone, ServerlessSpec\n\npc = Pinecone(api_key=os.environ[\"PINECONE_API_KEY\"])\nif not pc.has_index(\"agent-knowledge\"):\n    pc.create_index(\n        name=\"agent-knowledge\",\n        dimension=1536,  # must match text-embedding-3-small\n        metric=\"cosine\",\n        spec=ServerlessSpec(cloud=\"aws\", region=\"us-east-1\"),\n    )\n```\n\nStep 2 fills this index with embeddings it computes outside Pinecone. You make that choice here rather than later, and it rules out Pinecone's own MCP record-search tool.\n\n### Step 1: Extract with Website Content Crawler\n\nWebsite Content Crawler is an Actor, Apify's name for a serverless cloud program. It deep-crawls a site, removes navigation menus, cookie banners, and other boilerplate, and then outputs text and Markdown per page into a structured dataset. The Actor handles the fetch problem with proxy rotation and browser fingerprinting, and renders JavaScript when a page needs it. It runs on [Crawlee](https://crawlee.dev/), Apify's open-source crawling library, so those same techniques are available if you ever build your own crawler.\n\nThe call below crawls the Apify docs under a 20-page ceiling and hands step 2 a dataset ID:\n\n``` python\nimport os\nfrom apify_client import ApifyClient\n\nclient = ApifyClient(os.environ[\"APIFY_TOKEN\"])\n\ncrawl = client.actor(\"apify/website-content-crawler\").call(\n    run_input={\n        \"startUrls\": [{\"url\": \"https://docs.apify.com/platform/actors\"}],\n        \"crawlerType\": \"cheerio\",  # raw HTTP, lowest compute for static sites\n        \"maxCrawlPages\": 20,\n        \"respectRobotsTxtFile\": True,  # Console prefills this, the API does not\n        \"proxyConfiguration\": {\"useApifyProxy\": True},\n    }\n)\nif crawl is None or crawl.status != \"SUCCEEDED\":\n    raise RuntimeError(f\"crawl ended as {crawl.status if crawl else 'no run returned'}\")\ndataset_id = (\n    crawl.default_dataset_id\n)  # .call() returns a Run object in apify-client 3.x\n```\n\nThe measured run made 25 requests, stored 24 pages with 0 failures, finished in 61 seconds, and consumed 0.136 compute units. A compute unit (CU) is 1 GB of memory for 1 hour, priced at $0.20 per CU on [Apify's free and Starter plans](https://apify.com/pricing). This crawl was billed $0.029.\n\n**Compute is the largest line item, but not the only one.** Storage operations are billed alongside it.\n\nThis crawl recorded 77 writes and 26 reads on the request queue, plus 24 dataset writes. Compute alone comes to $0.027, and the rest of the $0.029 is storage. Every measured cost here is a billed total from the platform rather than an estimate.\n\nThe run stored 24 pages even though `maxCrawlPages`\n\nwas 20. The log explains why: when the crawler hits the limit, it stops enqueueing new links and shuts down gracefully, but it doesn’t cancel the requests that are already running. The setting is a ceiling with a small overshoot, not an exact count.\n\nBy default, the crawler follows only the start URL and its subpages. That is what keeps this run on the Actors section of the docs. `includeUrlGlobs`\n\nextends the crawl to paths elsewhere on the site, and `excludeUrlGlobs`\n\nnarrows it. Both take glob patterns like `https://docs.example.com/api/**`\n\n.\n\n**A run can end in a state other than SUCCEEDED.** Three of the 30 runs behind this article failed, one timed out, and we aborted two that never got a worker.\n\nA run still returns a dataset ID in all of those cases. The ID points to partial or empty results. Without the status check in the snippet above, that thin dataset flows straight into the next step.\n\nA run can sit in a READY state before a worker picks it up, and that wait isn’t part of the run duration. A 4 GB test run sat queued for nearly nine minutes, and an 8 GB run sat queued for over five minutes. We aborted both.\n\nThe same crawls at 1 GB and 2 GB started in about a second. You set the memory with `memory_mbytes`\n\non `.call()`\n\n.\n\n**One setting matters more than the rest.** `crawlerType: \"cheerio\"`\n\nuses raw HTTP, which costs the least compute and works well for static sites like documentation. For JavaScript-heavy sites, switch to `crawlerType: \"playwright:adaptive\"`\n\n, and the crawler renders pages in a headless browser only when a page needs it. Whichever you pick, keep it stable across re-crawls, because a config change alters the extracted text and invalidates the checksums that the next step relies on.\n\nWe tested that switch by pointing the same config at a second site. A raw HTTP crawl of linear.app/docs finished as SUCCEEDED, yet the site's landing page returned only 105 characters of text. That page builds its content in the browser, and the documentation pages around it arrive as full HTML.\n\nThe same crawl in adaptive mode returned 10 times as much text for that page. It also cost 0.081 CU for 5 pages against 0.006 CU for the 7 pages that raw HTTP stored, about 19 times the compute per page. A single site can mix both rendering modes page by page. That is why per-page detection exists, and why a successful run status alone is not a measure of output quality.\n\nTwo newer options match where the web is going. `signHttpRequests`\n\nmakes the crawler sign its requests under Web Bot Auth, so the crawler can identify itself as a verifiable agent on sites that restrict access by identity. Apify still marks this option as experimental.\n\n`useLlmsTxt`\n\nmakes the crawler read the target site's [llms.txt](https://llmstxt.org/) file. That file is a Markdown index that a site publishes to point AI systems at the site's key pages. Turn it on when the target site publishes one.\n\nThe crawl snippet above also sets `respectRobotsTxtFile`\n\n. Apify Console prefills it, but the API doesn’t, so set it explicitly to keep the crawl within the site's stated rules. The measured run used the API default, which is off. A separate run with the flag on returned the same 25 requests and the same 24 pages.\n\nContent behind a login (an internal wiki, for example) needs two more input fields. `initialCookies`\n\nsets cookies on every page that the crawler opens, which covers session logins. `customHttpHeaders`\n\nadds headers to every request, which covers token and basic auth. Both travel as Actor input and are stored with the run record, so scope and rotate them like API keys.\n\n### Step 2: Transform and load with Pinecone Integration\n\nPinecone Integration takes the dataset from step 1, splits the text into chunks, computes embeddings, and upserts vectors with metadata into a Pinecone index. One call configures chunking, embedding, and the update strategy:\n\n```\nload = client.actor(\"apify/pinecone-integration\").call(\n    run_input={\n        \"datasetId\": dataset_id,\n        \"datasetFields\": [\"text\"],\n        \"metadataDatasetFields\": {\"url\": \"url\", \"title\": \"metadata.title\"},\n        \"pineconeApiKey\": os.environ[\"PINECONE_API_KEY\"],\n        \"pineconeIndexName\": \"agent-knowledge\",\n        \"embeddingsProvider\": \"OpenAI\",\n        \"embeddingsApiKey\": os.environ[\"OPENAI_API_KEY\"],\n        \"embeddingsConfig\": {\"model\": \"text-embedding-3-small\"},\n        \"performChunking\": True,\n        \"chunkSize\": 2000,\n        \"chunkOverlap\": 200,\n        \"dataUpdatesStrategy\": \"deltaUpdates\",\n        \"dataUpdatesPrimaryDatasetFields\": [\"url\"],\n        \"deleteExpiredObjects\": True,\n        \"expiredObjectDeletionPeriodDays\": 30,\n    }\n)\nif load is None or load.status != \"SUCCEEDED\":\n    raise RuntimeError(\n        f\"index load ended as {load.status if load else 'no run returned'}\"\n    )\n```\n\nThree of these settings directly address the failure modes:\n\n`dataUpdatesStrategy: \"deltaUpdates\"`\n\ncompares checksums against what is already in the index and re-embeds only the pages that changed. Scheduled re-crawls no longer cost you a full re-embed each time.`deleteExpiredObjects`\n\nwith a 30-day window removes vectors for pages that disappeared from the source site, so the agent doesn’t retrieve deleted content.`dataUpdatesPrimaryDatasetFields: [\"url\"]`\n\nmakes the page URL the primary key for updates, which lets a changed page replace its old vectors instead of adding duplicates. A renamed URL is the exception.\n\nTwo notes before you run this. Both API keys travel as Actor input, which the run record stores. Scope these keys to this job and rotate them if a run link ever leaves your team.\n\nThe content also leaves your boundary here, because the chunk text is sent to the embedding provider and then stored in the index. That is unremarkable for public documentation like the site crawled here. It does matter if you point the same pipeline at an internal wiki.\n\nThe integration embeds through OpenAI or Cohere and has no self-hosted option. If that boundary is firm, compute the embeddings yourself and upsert to Pinecone directly, so the text never leaves your own infrastructure.\n\n**Expiry depends on when a page was last seen in a crawl, not on whether the page still exists.** `deleteExpiredObjects`\n\nis on by default.\n\nWhen a site blocks part of a crawl, the run still finishes successfully with fewer pages. Nothing refreshes the timestamps on the missed pages, so a later run deletes their vectors once the window has passed. That is the real reason to be alert to item counts.\n\nThe measured integration run loaded the 24 crawled pages, split them into 59 chunks, embedded the chunks, and upserted everything into the index in 21 seconds. It consumed 0.0059 CU and was billed $0.002.\n\n**Changing the chunking or metadata configuration invalidates the delta logic.** We re-ran the integration after altering one metadata setting, and the checksums of all 59 chunks changed. The Actor deleted all 59 old vectors and re-embedded every chunk. The index count stayed at exactly 59 with no duplicates.\n\nDecide your chunking and metadata setup before you index at scale. When you do have to change them later, build the new version into a separate namespace, run your eval against it, and switch the query target once it wins. That keeps the old index serving while you build the new one. You set the namespace name in two places: `pineconeIndexNamespace`\n\non the Actor input, and `namespace`\n\non `index.query()`\n\n.\n\nThe unchanged case is the opposite. We re-crawled the same site, got the same 24 pages, and ran the integration again with nothing changed. The log reported zero objects to add and zero to delete, and the Actor only refreshed the last-seen timestamps on all 59 chunks.\n\nWith delta updates on, an unchanged corpus produced no embedding calls at all. That is what makes a scheduled re-crawl cheap to repeat.\n\nCheap is not free: the crawl itself still costs $0.029 each time, and only the embedding drops to zero.\n\nPut the crawler on a [schedule](https://docs.apify.com/actors/running/schedules). Step 2 needs the dataset ID from each new crawl, and nothing holds `dataset_id`\n\nin a variable once the Python process is gone.\n\nAdd an [Actor-to-Actor integration](https://docs.apify.com/integrations/actors) to the crawler. The integration fires on the run-succeeded event, starts Pinecone Integration, and passes `{{resource.defaultDatasetId}}`\n\nas its `datasetId`\n\n. Everything else from the snippet above stays as static configuration on that integration.\n\n**URL-keyed updates have one operational problem.** We re-indexed the same pages, but first we renamed one page's URL, just as a docs reorganization renames whole sections. The integration treated the renamed page as new content and re-embedded its 4 chunks.\n\nIt also left the old URL's 4 vectors in place, so the index grew from 59 to 63. The same content was stored twice, once under a dead URL.\n\nNothing removes those duplicates until the expiry window passes, which is 30 days in this config. After any URL migration, set `expiredObjectDeletionPeriodDays`\n\nto 1 for one run, then put it back. Do not set it to 0, which turns expiry off rather than speeding it up.\n\nThe alternative is to call `index.delete_namespace(name=\"__default__\")`\n\nand re-index, since the walkthrough never sets a namespace. Expect renamed sections to cost a re-embed.\n\nThe crawled pages averaged about 3.8k characters of extracted text each. Across the 24-page corpus, that is roughly 90k characters and 18k tokens. It cost well under a cent to embed the whole corpus, at $0.02 per 1M tokens for `text-embedding-3-small`\n\n([OpenAI's published price](https://developers.openai.com/api/docs/pricing) as of July 2026). Even 1,000 pages of similar density cost about $0.02 in embedding fees.\n\nThe account's own usage dashboard counts 58,544 input tokens across 126 requests, which include both chunking configs and every retrieval-eval query. At $0.02 per 1M tokens, that comes to about $0.0012.\n\nThe model itself is a swappable config field, and the integration also supports Cohere. An eval is how you decide whether to swap.\n\n**One constraint applies to model swaps.** A model with a different output dimension needs a new index and a full re-embed rather than an edit to one config field. Check the dimension that your target model produces before you change anything.\n\nEquivalent integrations exist for [Qdrant](https://apify.com/apify/qdrant-integration) and [Milvus](https://apify.com/apify/milvus-integration) if Pinecone isn’t your storage choice. [pgvector](https://github.com/pgvector/pgvector) inside Postgres is an alternative to a dedicated vector database. It is the better choice when retrieval needs SQL joins or permissions filtering. This example uses Pinecone because it has a free tier, scales without provisioning, and already has an Actor.\n\n### Step 3: Retrieve from Pinecone\n\nAt query time, the agent embeds the user's question with the same model and runs a similarity search against the index:\n\n``` python\nimport os\nfrom openai import OpenAI\nfrom pinecone import Pinecone\n\nquestion = \"How do I schedule an Actor to run daily?\"\nembedding = (\n    OpenAI(api_key=os.environ[\"OPENAI_API_KEY\"])\n    .embeddings.create(model=\"text-embedding-3-small\", input=question)\n    .data[0]\n    .embedding\n)\n\nindex = Pinecone(api_key=os.environ[\"PINECONE_API_KEY\"]).Index(\"agent-knowledge\")\nmatches = index.query(vector=embedding, top_k=5, include_metadata=True)\ncontext = \"\\n\\n\".join(\n    f\"Source: {m['metadata']['url']}\\n{m['metadata']['text']}\"\n    for m in matches[\"matches\"]\n)\n```\n\nThe top 5 chunks go into the model's context. Because step 2 stored the source URLs as metadata, the agent can cite where each answer came from.\n\nThe chunk text itself arrives under `metadata[\"text\"]`\n\nbecause `datasetFields: [\"text\"]`\n\nin step 2 selected it for storage. Passing the text alone is not enough. Without the URLs, a model asked to cite sources will reconstruct plausible ones from page titles.\n\nYou can also filter on that metadata. Pass `filter={\"url\": {\"$eq\": ...}}`\n\nto `index.query()`\n\nto narrow a search to one page or section.\n\n**The integration stores more metadata than step 2 configured.** Every chunk also carries a `checksum`\n\nand `last_seen_at`\n\n, a Unix timestamp of the crawl that last saw the page. Put `last_seen_at`\n\non the source line when you want the model to weigh freshness. Convert it first, because `1784613039`\n\ntells the model nothing and `2026-07-21`\n\ntells it a date.\n\nEach match also carries a `score`\n\n. The threshold routing uses that number:\n\n```\ntop_score = matches[\"matches\"][0][\"score\"] if matches[\"matches\"] else 0.0\nif top_score < THRESHOLD:  # from your own eval, not from this article\n    context = live_search(question)  # fall back to the live path, defined next\n```\n\n`live_search`\n\nruns RAG Web Browser as a direct Actor call, the same `client.actor(...).call()`\n\npattern as steps 1 and 2. It joins the returned pages into one Markdown context string:\n\n``` python\ndef live_search(question):\n    run = client.actor(\"apify/rag-web-browser\").call(\n        run_input={\n            \"query\": question,\n            \"maxResults\": 3,\n            \"scrapingTool\": \"raw-http\",\n        }\n    )\n    items = client.dataset(run.default_dataset_id).list_items().items\n    return \"\\n\\n\".join(m[\"markdown\"] for m in items if m.get(\"markdown\"))\n```\n\nThis direct call suits the threshold approach. The prompt approach reaches the same Actor over MCP instead.\n\n**Derive THRESHOLD from your own eval rather than from an article.** Run your questions, record the top-1 score on each question where the index retrieves the correct page, and set the bar below the lowest of those scores.\n\nIn the measured test, four of the five results for the step 3 query came from the schedules documentation page. The fifth came from a different page:\n\n```\n0.607  https://docs.apify.com/actors/running/schedules\n0.591  https://docs.apify.com/actors/running/schedules\n0.574  https://docs.apify.com/actors/running/schedules\n0.550  https://docs.apify.com/actors/running/schedules\n0.514  https://docs.apify.com/actors/development/automated-tests\n```\n\nFrom the test machine, warm queries returned in roughly 0.4 seconds, and embedding the question took another 0.4 to 0.9 seconds.\n\nOn this corpus, a bar below the lowest correct score gives a floor, not a filter. Across the 15 eval questions, the correct top-1 hits scored 0.452 to 0.717, and the one wrong top-1 scored 0.478, which is inside that range. A bar cannot keep every correct hit without also passing the wrong one.\n\nThe pipeline exists for this final call, handing the retrieved context to the model alongside the question:\n\n```\nanswer = (\n    OpenAI(api_key=os.environ[\"OPENAI_API_KEY\"])\n    .chat.completions.create(\n        model=\"gpt-5-mini\",\n        messages=[\n            {\n                \"role\": \"system\",\n                \"content\": \"Answer only from the context provided. Cite the source URLs.\",\n            },\n            {\"role\": \"user\", \"content\": f\"{context}\\n\\nQuestion: {question}\"},\n        ],\n    )\n    .choices[0]\n    .message.content\n)\n```\n\nThe answer is grounded in those chunks and carries the URL it came from:\n\n```\nUse a schedule with the daily cron expression \"@daily\" (equivalent to\n\"0 0 * * *\") and either create it in Apify Console or via the Apify API.\n(Docs: https://docs.apify.com/actors/running/schedules)\n```\n\nThe chat model is interchangeable here, since nothing upstream depends on which one answers. Both system-message instructions are required. Without the first, the model answers from training data. Without the second, the URLs in your context never reach the user.\n\nAny orchestration framework can run the retrieval and the answer call. Apify has documented integrations for LangChain, LlamaIndex, LangGraph, CrewAI, Haystack, Mastra, the OpenAI Agents SDK, the Vercel AI SDK, and Google ADK.\n\n### How well it retrieves, measured\n\nMeasuring retrieval quality takes more than one demo query. We wrote 15 test questions against the crawled corpus, and each one has a known correct source page. We checked whether that page was returned.\n\nEach question retrieves the top 10 chunks. Those chunks collapse to their distinct source pages in score order, and we record where the correct page lands. This is a wider window than the top 5 chunks that the step 3 snippet retrieves. Read the top-5-pages column as a page-level upper bound, not a guarantee about that snippet.\n\nThen we re-indexed the same 24 pages into a separate namespace at half the chunk size and ran the same questions again.\n\nThree of the 15 questions, with the page that each answer should come from:\n\n| Question | Correct page |\n|---|---|\n| What is the rate limit for requests to a Standby Actor? | `/running/standby` |\n| What happens to rental Actors in 2026? | `/running/actors-in-store` |\n| How do I get alerted when my Actor run fails? | `/running/monitoring` |\n\nThe two chunk sizes scored:\n\n| Chunking config | Chunks | Correct page ranked first | Correct page in top 5 pages |\n|---|---|---|---|\n2,000 chars, 200 overlap |\n59 | 14/15 (93%) | 15/15 (100%) |\n1,000 chars, 100 overlap |\n114 | 13/15 (86%) | 15/15 (100%) |\n\nTwo lessons follow from the comparison. First, the larger chunks won by one question, and neither configuration missed the correct page in its top 5. A margin that thin is not a reason to pick 2,000 characters for your own corpus.\n\nSecond, the first eval against the fresh namespace silently missed content. The vectors had been upserted less than a minute earlier, and the serverless index hadn’t caught up yet. A fresh write may not be queryable the instant it lands. Re-run the eval before you trust a first result against a fresh namespace, because that failure looks like bad retrieval rather than a timing problem.\n\n**An eval that picks your settings needs a hold-out.** Set aside a few questions you never tune against, and check that the winning setting still scores higher on those.\n\nWe used the same 15 questions to pick the chunk size and to report the result, and that favors whichever setting scored higher. A setting can look better on its own tuning data and nowhere else. That is easy to mistake for a real improvement.\n\n**Plain dense retrieval was enough at this scale.** There are two standard upgrades. [Hybrid search](https://docs.pinecone.io/guides/search/hybrid-search) mixes keyword and vector scores. A [reranker](https://docs.pinecone.io/guides/search/rerank-results) reorders the results.\n\nThe same 15-question eval tells you which one you need. If a correct page lands in your top 5 but not first, it is retrieved and then ranked below something else. A reranker fixes that.\n\nQuestions that miss both columns need hybrid search instead. Those are typically the ones that depend on an exact term, an error code, or a proper noun.\n\n**One result from the same setup matters more than the hit rates.** With the retrieved chunks truncated to 200 characters each, the model answered only one of three questions correctly, even though the right pages were retrieved every time. Retrieval hit rate is not answer rate. What you assemble into the context window is a separate failure point.\n\n## The MCP path for runtime tool discovery\n\nThe index is the default path. The live retrieval path can run through the Model Context Protocol (MCP), an open standard that lets agents discover and call external tools at runtime.\n\nThe Apify MCP server at `mcp.apify.com`\n\nexposes Actors as callable tools. MCP clients can use it if they support remote servers. Agents built on the frameworks above can use it too.\n\nPoint your client at the server. Whenever the index doesn’t have the answer, the agent can call [RAG Web Browser](https://apify.com/apify/rag-web-browser), a tool that searches the web and fetches pages:\n\n```\n{\"mcpServers\": {\"apify\": {\"url\": \"https://mcp.apify.com?tools=apify/rag-web-browser\"}}}\n```\n\nWe verified this config with the official Python MCP SDK over streamable HTTP. The server exposed the Actor as a callable tool. A 1-result query through the protocol came back in about 21 seconds.\n\nLeave the `?tools=`\n\nparameter off, and the server loads its default set instead. An MCP server sends the agent the schema of every tool it exposes, before the question is even asked. Apify's [documentation](https://docs.apify.com/integrations/mcp) recommends naming tools explicitly in production so behavior stays stable across updates. Naming them explicitly also saves the tokens those schemas cost.\n\n**The two retrieval paths are not reached the same way.** The live path is a tool that the agent calls over MCP. The index path runs in your own code.\n\nPinecone has its own [MCP server](https://docs.pinecone.io/guides/operations/mcp-server) with a record-search tool. Pinecone's documentation says that the record-search tool doesn’t support indexes built with external embedding models. This index is built exactly that way, with OpenAI embeddings computed by the integration in step 2.\n\nTo put both paths behind MCP, the index has to use Pinecone's integrated inference. You create that kind of index with `create_index_for_model`\n\nrather than with the `create_index`\n\ncall above. That rules out step 2, because the integration computes the embeddings outside Pinecone, so you would chunk and upsert the text yourself. It is a decision you make when you create the index, not afterwards.\n\n## Trust and permissions when the agent picks its own tools\n\nLive retrieval changes what you can trust. Everything in the index came from sources you chose. The live path can put any page on the web into the model's context.\n\nTreat fetched text as data, never as instructions. [Indirect prompt injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) through retrieved content is first on the Open Web Application Security Project OWASP risk list for LLM applications. The risk is concentrated on the live path.\n\nThe risk comes from a combination, not from any single part. Simon Willison named it the [lethal trifecta](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/): private data, untrusted content, and a way to send data out. This pipeline has the first two by design.\n\nThe third is the one you add yourself. A poisoned page becomes an exfiltration route as soon as you add a tool that can post, write, or call an external endpoint. Keep the live path read-only, or put an approval step in front of anything that writes.\n\nThe MCP server isn’t limited to one tool. It can expose thousands of Actors from [Apify Store](https://apify.com/store) through tools that let the agent search the store and load an Actor mid-conversation. The agent can then call tools you never explicitly configured.\n\nThe MCP server isn't limited to one tool. It can expose **thousands of Actors** from Apify Store through tools that let the agent search the store and load an Actor mid-conversation.\n\nThat breadth has an upside. Not every question needs to pass through embeddings. When the agent needs prices, listings, profiles, or map results, the right tool is one of the [AI agent tools](https://blog.apify.com/ai-agent-tools/) that return structured JSON. The index is not involved.\n\n**The agent's autonomy has a guardrail.** Actors declare a permission level. Most Actors run with limited permissions and can reach only their own storages.\n\nSome Actors need full account access. You must approve those once in Apify Console before they can run. The MCP server goes further and leaves full-permission Actors out of search and execution altogether. Escalation stays your decision, not something an agent can do on its own.\n\nWe hit that guardrail during testing, when the integration Actor waited for exactly that approval before its first run. That approval doesn’t block the pipeline here, because step 2 calls the integration from your own code rather than over MCP. Every Actor's page carries its permission level as a badge, so check it before you run anything.\n\n**Payment is moving in the same direction as identity.** Since June 2026, an agent can pay for Actor calls through [x402 agentic payments](https://blog.apify.com/introducing-x402-agentic-payments/) with no Apify account or API key. In the [documented flow](https://docs.apify.com/integrations/x402), the agent authenticates a wallet, buys a prepaid USD Coin (USDC) token, and passes that token as bearer auth on the platform's standard run endpoints.\n\nx402 currently supports Actors with pay-per-event pricing and limited permissions, so the CU-priced crawler and browser in this pipeline aren’t covered yet. x402 is built on HTTP 402 Payment Required, the same status code that Cloudflare's pay-per-crawl uses to charge crawlers. Each shift gives the agent more autonomy, and each one leaves a gate you control.\n\n## What the live path costs\n\nRAG Web Browser takes a search query, pulls the top Google results, fetches those pages, and returns their content as Markdown. In the measured runs, a 3-result query in raw HTTP mode returned in 10.4 seconds, consumed 0.0112 CU, and was billed $0.005. That price includes the charge for the Google search itself.\n\nThe Actor fetched two of the three pages cleanly. The third was blocked and returned only its search-result metadata. That degraded page is the fetch problem, now appearing on the live path.\n\nThe same query in browser mode returned all three pages cleanly. But the run continued for 379 seconds, ended in a timeout, and was billed $0.019. Across nine raw HTTP queries that produced 25 pages between them, 22 had usable content and three were blocked at the source, returning only search-result metadata. Expect a small share of degraded results, and check each one before you hand it to the model.\n\n**For production latency, run the Actor in Standby mode.** Standby keeps it running to serve requests without cold starts. Apify's published Standby benchmarks range from 16 seconds (8 GB memory, 1 result) to 31 seconds (4 GB memory, 3 results), figures that Apify flags as indicative. A handful of our own 1-result requests took 6.7 to 22.3 seconds.\n\nPlan for the spread rather than any single number. One request returned a transient server error and succeeded on retry, so build retries into the client code.\n\nStandby's convenience is metered: the container that served those requests stayed up for 8 minutes 56 seconds and was billed $0.114. Standby bills for the time it stays warm, not only for the requests it answers.\n\nThe **Endpoints** tab in Apify Console confirms that Standby is on and gives the hostname that those requests hit, `rag-web-browser.apify.actor`\n\n. The tab shows the full endpoint URL with the token as a query parameter. Prefer the `Authorization`\n\nheader, which the Actor also accepts. Tokens in a query parameter end up in server logs and in any link you share.\n\nThat latency spread is why live retrieval is the fallback. An index query is two network calls, one to embed the question and one to search the index. A live fetch runs a web search plus one page fetch per result. The runs above used three results per query.\n\n## What the pipeline costs\n\nEvery example ran on one Apify account, so its run list shows the cost of each step and how each was started. The **Origin** column tags each run as API, MCP, Standby, or Web. These are the same Actors started four different ways. We captured the view below partway through the work, so it shows fewer runs and a smaller total than the final figures.\n\nAll figures measured from the runs above or verified against vendor pricing pages, July 2026:\n\n| Component | Measured or verified figure | Cost |\n|---|---|---|\nCrawl (Website Content Crawler, 24 pages, raw HTTP) |\n61 seconds, 0.136 CU | $0.029 per crawl |\nEmbeddings (text-embedding-3-small, ~18k tokens) |\n$0.02 per 1M tokens | <$0.01 |\nChunk, embed, and upsert (Pinecone Integration, 24 pages into 59 chunks) |\n21 seconds, 0.0059 CU | $0.002 per run |\nIndex query (Pinecone, warm, from test machine) |\n~0.4 seconds, plus 0.4-0.9 seconds to embed the question | 1 read unit, free at this scale |\nVector storage (Pinecone Starter) |\n2 GB free tier | $0 at this scale |\nVector storage (Pinecone Standard, at scale) |\n$0.33/GB/month, $16-18 per 1M reads | $50/month minimum |\nLive retrieval (RAG Web Browser, 3 results, raw HTTP) |\n10.4 seconds, 0.0112 CU | $0.005 per query |\nLive retrieval (RAG Web Browser, 3 results, browser mode, all 3 pages returned, then timed out) |\n379 seconds, 0.0798 CU | $0.019 per query |\n\nWrites are priced separately, at $4 to $4.50 per 1M write units on the [Standard serverless plan](https://www.pinecone.io/pricing/). Read and write rates vary slightly by cloud and region. The Starter tier also includes 1M read units a month.\n\nBoth chunking namespaces together hold 173 vectors and use 1.33 MB of the 2 GB. Every eval question across both namespaces consumed 1 read unit.\n\nThe answer call also falls outside the table, because it belongs to the model layer rather than the data layer. Every chunk you retrieve becomes part of its input, which is why the retrieval count matters.\n\nWe re-ran the raw HTTP live query at 1 GB instead of 4 GB. The run took 37 seconds instead of 10, but the bill barely changed, $0.0051 versus $0.0052. The platform allocates CPU in proportion to memory. Extra memory adds far more speed than cost, so provision for the latency you need rather than for the smallest allocation that fits.\n\nQueue time is the counterweight, as the 4 GB and 8 GB crawls in step 1 showed. Choose memory for latency when runs start promptly, and lower it if you see runs sitting in READY.\n\nThe complete set of platform runs came to 30 and was billed $0.35 on Apify, including the six that didn’t succeed. That is about 7% of one month's free credit. The OpenAI total was under a cent for the entire day, across all embedding and chat calls.\n\n**Three costs grow as you scale.** Crawl compute is linear with pages and re-crawl frequency, vector database reads are linear with query volume, and the Pinecone Standard minimum applies once you outgrow the free tier. To estimate your own costs, scale the billed crawl directly.\n\nAt the same page density, $0.029 for 24 pages gives roughly $1.20 for 1,000 pages, with embedding still only about two cents. Embedding cost stays negligible at any scale, as long as delta updates prevent re-embedding of unchanged content.\n\nNote that these figures are estimates scaled from the run above, not direct measurements. Re-measure on your own site.\n\n## A decision checklist before you build\n\nBefore you copy this architecture, run your use case through these decisions.\n\n**Start with the index, not the live tool.** If your agent's knowledge changes on the order of weeks rather than hours, a persisted index serves most queries faster and more cheaply.**Put only fast-changing sources on the live path.** Anything that changes daily belongs there: news, prices, availability. Docs and catalogs do not.**Turn on delta updates from day one.** Re-embedding unchanged pages is pure waste, and adding delta updates later is harder than starting with them.**Start retrieval at top 3 to 5 chunks.** Only raise the count when retrieval quality measurably requires it. Every extra chunk adds input tokens to every query forever.**Write a 15-question eval before tuning anything.** Questions with known source pages turn chunking and retrieval debates into measurements. Ours cost a few cents to run.**Score the answers, not just the retrieval.** A pipeline can fetch exactly the right page and still answer from it badly. Keep a hold-out set that the tuning never sees, and re-run the whole set after every re-crawl to catch quality drifting as pages change.**Alert on output, not just run status.** When a site blocks part of a crawl, the run still exits successfully with fewer or thinner items. Apify's built-in[monitoring](https://docs.apify.com/actors/running/monitoring)can alert on item counts and dataset fields, which catches the silent degradation that run status alone doesn’t measure.**Match the crawler to the site.** Raw HTTP for static sites, adaptive or browser rendering for JavaScript-heavy ones. The wrong choice either misses content or pays browser prices for static pages.**Re-crawl weekly for docs, faster for sources that change more often.** Delta updates make over-crawling cheap, but the crawl compute itself still costs money.\n\n## The AI agent infrastructure data layer, built once\n\nThe data layer decides whether your agent answers from current sources or from stale training data. Vector databases, embedding models, and crawler defaults all move faster than the problems they solve. The four failure modes do not move: fetch failure, stale knowledge, wasted re-embedding, and retrieval count creep.\n\nThe smallest useful first step is a single crawl. Point [Website Content Crawler](https://apify.com/apify/website-content-crawler) at your own documentation and read the text it returns. If an index over that content looks worth building, [Pinecone Integration](https://apify.com/apify/pinecone-integration) builds that index from the same crawl, and an MCP client pointed at [mcp.apify.com](https://mcp.apify.com/) adds the live path.\n\nThe [Apify AI integrations documentation](https://docs.apify.com/integrations/ai) covers every framework mentioned here.\n\n## FAQ\n\n### What is an AI agent infrastructure stack?\n\nThe set of systems that a production agent runs on, including models, orchestration, memory, observability, and the layer that holds tools and data. The data layer covers extracting content, transforming it into embeddings, storing those embeddings in a vector database, and retrieving them at query time.\n\n### Do I need both a vector index and live web retrieval?\n\nUsually yes, but they carry very different shares of the traffic. The index handles bounded, stable knowledge cheaply and quickly. Live retrieval covers fresh or long-tail queries that fall outside the index. A production agent should route most queries to the index and fall back to live retrieval for the rest.\n\n### How much does it cost to build a RAG pipeline for an agent?\n\nThe measured example here crawled and indexed a 24-page site for about $0.03 in total, including embedding costs of well under a cent. The real costs at scale are crawl compute, vector database reads, and the minimum charge on a paid plan.\n\n### What if I want to move off Pinecone or OpenAI later?\n\nBoth are configuration, not architecture. Equivalent integrations exist for Qdrant and Milvus, the embedding model is one field if the dimension matches, and the crawl output is a dataset you can reuse anywhere. The real switching cost is your chunking and metadata setup, because changing it forces a full re-embed.", "url": "https://wpnews.pro/news/ai-agent-infrastructure-stack-from-web-data-to-retrieval", "canonical_source": "https://blog.apify.com/ai-agent-infrastructure/", "published_at": "2026-08-05 10:10:17+00:00", "updated_at": "2026-08-05 10:25:35.309055+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "ai-tools"], "entities": ["Apify", "Pinecone", "Cloudflare", "LangGraph", "CrewAI", "Mastra", "OpenAI Agents SDK", "Apify MCP server"], "alternates": {"html": "https://wpnews.pro/news/ai-agent-infrastructure-stack-from-web-data-to-retrieval", "markdown": "https://wpnews.pro/news/ai-agent-infrastructure-stack-from-web-data-to-retrieval.md", "text": "https://wpnews.pro/news/ai-agent-infrastructure-stack-from-web-data-to-retrieval.txt", "jsonld": "https://wpnews.pro/news/ai-agent-infrastructure-stack-from-web-data-to-retrieval.jsonld"}}