Most RAG implementations treat document ingestion as a batch job. You update a file, trigger a reindex, wait for embeddings to regenerate, then hope your vector database doesn't return stale context. For production systems where documents change frequently—internal wikis, API docs, customer support knowledge bases—this rebuild-on-update pattern creates a silent correctness tax.
Pathway's Serviette attacks this from a different angle: streaming incremental updates. When a document changes, only the affected chunks get re-embedded and synced. No full pipeline restart. No batch window.
Serviette is a FastAPI-based RAG server that sits between your document sources and your vector store. It handles:
/v1/chat/completions with retrieval built in
The core value proposition is zero-downtime sync. Your RAG server stays up while documents update underneath it.
Minimal setup requires three environment variables:
pip install pathway[all]
export PATHWAY_DATA_DIR="./documents"
export OPENAI_API_KEY="your-key"
export VECTOR_STORE="qdrant"
python -m serviette.main
This spins up a server on localhost:8000 with automatic document ingestion from ./documents. Drop a PDF in that directory and Serviette picks it up within seconds, chunks it, embeds with text-embedding-3-small, and writes vectors to Qdrant.
For a 50-page technical spec (roughly 30K tokens raw text), here's what we measured:
Initial Embedding Cost:
Per-Query Retrieval Cost:
The asymmetry is stark. Embedding is cheap and one-time per document. Query-time context injection is where costs accumulate. For a high-traffic RAG endpoint handling 10K queries/day, that's $64/day just in context tokens.
Serviette's streaming architecture pairs naturally with prompt caching. When the same document chunks appear across multiple user queries, caching the embedded context once and reusing it cuts token costs by 90% on cached reads.
Here's a practical relay configuration routing through B-Lost's gateway for multi-provider fallback with cache-aware routing:
import openai
client = openai.OpenAI(
base_url="https://b-lost.com/v1",
api_key="your-blost-key"
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a technical assistant."},
{"role": "user", "content": "Explain the deployment model."}
],
extra_body={
"rag_context": serviette_retrieved_chunks, # injected from Serviette
"cache_context": True # signal cache-aware routing
}
)
In load tests with 500 queries against a stable 10-document corpus, cached context tokens reduced input costs from $3.20 to $0.48—an 85% drop. Serviette's incremental sync ensures the cache only invalidates for genuinely updated chunks, not the entire context window.
Serviette's abstraction over eight vector stores sounds convenient, but each database has sharp operational edges:
Qdrant: Fastest for sub-100K documents. No warmup latency. Native filtering support.
Weaviate: Better for hybrid search (vector + keyword). Requires schema migrations when adding metadata fields.
pgvector: Easiest to integrate if you already run Postgres. Performance degrades past 500K vectors without partitioning.
Pinecone: Managed scaling is hands-off, but coldstart latency on free tier makes it unsuitable for low-traffic dev endpoints.
We ran Serviette against Qdrant (local) and Pinecone (hosted) with the same 5K-document corpus. Qdrant's p95 retrieval latency stayed under 45ms. Pinecone's coldstart penalty spiked to 1.2 seconds on first query after 10 minutes of idle time. For production RAG servers with bursty traffic, local Qdrant wins.
Serviette's default chunking uses LangChain's RecursiveCharacterTextSplitter with 512-token windows and 128-token overlap. This works for narrative text but breaks on structured data.
For API reference docs with YAML examples, we switched to semantic chunking:
from pathway.xpacks.llm.splitters import TokenCountSplitter
splitter = TokenCountSplitter(
min_tokens=256,
max_tokens=768,
encoding_name="cl100k_base"
)
This preserved YAML blocks across chunk boundaries and reduced retrieval false positives by 18% in our test corpus (measured via human eval of top-5 relevance).
Serviette's streaming model assumes chunk-level granularity is sufficient for correctness. But what about documents where global context matters—legal contracts, versioned API specs, or multi-file codebases?
If you update auth.md but leave quickstart.md unchanged, Serviette only re-embeds auth.md. A user query about "authentication in the quickstart" might return stale context from the unchanged file. Traditional batch reindexing would catch this through full dependency analysis.
The trade-off is latency vs correctness. Serviette bets that most document updates are local and that 2-second sync beats 5-minute rebuild windows. But for domains where cross-document consistency is non-negotiable, you still need a hybrid strategy: incremental for fast updates, periodic full reindex for correctness validation.
How do you decide when incremental sync is safe enough for your RAG workload?