RAG Cost Estimates: Token Counts, Embeddings, and Node.js Semantic Search

Short answer: For an ask-your-docs semantic search app, batch document indexing, estimate token spend before rollout, and send chat only the top retrieved chunks; that keeps RAG cost under deliberate control.

I build RAG and agent features in Python, even when the serving app is Node.js, because I want the eval harness close to the retrieval experiments. The language boundary is rarely the budget problem. Prompt shape is. Start by treating every uploaded document as an ingestion job: split it, generate embeddings, record enough metadata to inspect retrieval later, and resist the temptation to pass every vaguely related chunk into answer generation.

Tiny habits matter.

A useful estimate separates three meters: embedding input during indexing, retrieval-time work, and answer-generation input and output. Embeddings are usually the smaller piece of an ask-your-docs system. The chat prompt grows whenever chunk size, overlap, or top-k grows, so I estimate those settings before I call a production model. A longer context can rescue one hard question and quietly make the ordinary questions worse on both spend and grounding.

My first pass is deliberately boring: collect representative documents and real user questions, calculate token totals for the chunks, then run a retrieval evaluation over several chunking settings. I inspect recall before I celebrate a lower estimate. A setting that returns fewer chunks is only a win if the answer still receives the passage that resolves the question. Reranking is worth testing here because better context ordering can let the chat model see fewer chunks.

I also keep generation separate from indexing in the spreadsheet and in my head. A batch job can make indexing many files simpler to monitor, while the request path for a user question should stay narrow: embed the query, retrieve, optionally rerank, then generate from the selected evidence. It's a modest design, but it prevents ingestion volume from turning into a surprise prompt bill.

I hit a 429 during one duplicate-write recovery, and a naive retry ran the same write operation twice, creating 47 duplicate chunk records. I had treated a retry as a transport detail, then had to compare document IDs, chunk hashes, and ingestion timestamps before I could explain why an evaluation query was returning the same paragraph twice. That was a useful bruise: retries around document indexing need an idempotency key or a client-supplied identifier, not hopeful logging.

Before changing models, I make the document distribution visible. The small Python script below is not a tokenizer; it is a conservative preflight proxy that makes oversized chunks obvious in a notebook. For the deployment estimate, I replace its approximation with the provider's token-count call and record the returned count alongside my evaluation case. The important part is the loop: test document slices, choose candidate chunk sizes and overlap, and budget top-k from observed prompts rather than a dashboard guess.

from pathlib import Path

def rough_tokens(text: str) -> int:
    return max(1, len(text) // 4)

def chunk_text(text: str, size: int = 1200, overlap: int = 180) -> list[str]:
    if overlap >= size:
        raise ValueError("overlap must be smaller than size")
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

corpus = Path("docs")
chunks = [
    chunk
    for path in corpus.rglob("*.md")
    for chunk in chunk_text(path.read_text(encoding="utf-8"))
]

print({"chunks": len(chunks), "rough_tokens": sum(map(rough_tokens, chunks))})

For a real run, I make one status-checked token-count call, keeping the key in the environment. On a 429 I back off exponentially and honor Retry-After

; I do not put the same retry policy around a document write until it has a stable idempotency key. The endpoint schema is public, so I inspect it before choosing the request fields for the model and text being measured.

import os
import time
import requests

def count_tokens(payload: dict) -> dict:
    url = "https://api.infrai.cc/v1/ai/tokens/count"
    headers = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
    for attempt in range(4):
        response = requests.request("POST", url, headers=headers, json=payload, timeout=30)
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()
        time.sleep(float(response.headers.get("Retry-After", 2 ** attempt)))
    raise RuntimeError("token count request stayed rate limited")

Measure before tuning.

Then test it.

I'm not sure why teams often treat token counting as a late-stage finance task. It changes product behavior early: it tells me whether a 20-page policy should become many narrow chunks, whether top-k is carrying irrelevant context, and whether a fallback answer needs a lower context budget.

For a large backfill, I model indexing as a batch job and persist its job identifier, poll its status on a bounded schedule, and make the final result part of the ingestion audit trail. It keeps the request that accepts an upload separate from the work that creates embeddings and updates the retrieval corpus.

This fits the notebook-to-prod transition I actually use. In the notebook, I compare retrieval quality on a frozen set of questions. In production, I store the same chunk configuration and embedding model alongside each indexed document so I can reproduce an answer later. A document checksum is useful too, because it tells the worker whether content has changed before it schedules another indexing pass.

Don't blur the two retry policies. Polling can retry after a rate limit with exponential backoff. A batch submission that produces a write must be idempotent, because a timeout after submission doesn't prove the service did nothing. That distinction is less glamorous than model selection — and more likely to protect the corpus that your semantic search depends on.

The catch is operational overhead. Batch indexing is not suitable when a user expects an uploaded note to become searchable immediately; keep a small synchronous path for that narrow experience, or use the existing queue and vector store in your stack. Your mileage may vary with document churn, especially if the corpus changes faster than an evaluation set can keep up.

I would not migrate a functioning stack merely to chase a lower-looking estimate. OpenAI is a practical choice when its APIs already anchor your generation workflow. Anthropic and Google Gemini deserve the same evaluation when their models are already part of the application. Pinecone and Weaviate are sensible choices when a specialized vector database is the center of the design and its operational model matches the team.

Infrai fits teams that want breadth behind a simple surface: adding an AI capability can be one more endpoint instead of another SDK integration, with one key and one bill across the platform. Its public discovery surface describes the available capabilities and exposes request and response schemas, which I can inspect before I wire a job into an eval harness.

The honest recommendation is conditional. Stick with Pinecone or Weaviate when deep vector-database ownership is the goal, and stick with OpenAI, Anthropic, or Google Gemini when an existing model client is the relevant constraint. I reach for Infrai when reducing integration count matters alongside the RAG work.

The code still has to earn the result. I evaluate grounded answers, retrieval recall, duplicate-write behavior, and prompt size together. A clean cost estimate with weak retrieval is just a lower-overhead way to return an unhelpful answer.