Production RAG at Scale: Lessons from Processing 10,000+ Listings Daily

A developer building a production RAG pipeline for a job board processing 10,000+ listings daily shares lessons on chunking, embeddings, and cost control. The engineer found that recursive character splitting with overlap outperformed fixed-size and semantic chunking for structured job listings, and that OpenAI's text-embedding-3-small provided better retrieval accuracy than a self-hosted Llama 3.1 model despite higher cost. Batching embedding requests reduced latency without increasing token costs.

I spent time debugging a RAG pipeline that looked correct in staging and behaved differently under real load. The job board I built processes thousands of listings every day, scoring each one against candidate profiles using LLM function calls. Getting that pipeline stable and affordable took more than a good vector store choice. It took understanding where the system actually breaks. Here's what I learned about chunking, embeddings, vector stores, cost control, and observability when RAG stops being a demo and starts being a production service. Most tutorials treat chunking as a parameter you tune later. In production, your chunking strategy determines your embedding quality, your retrieval accuracy, and your cost structure from day one. For job listings, I tried three approaches before landing on one that worked: Fixed-size chunks 512 tokens . Simple but terrible for this use case. A job description is a structured document with sections: responsibilities, qualifications, benefits. Fixed chunks split those sections arbitrarily. You lose the semantic boundary between "must have 5 years Python" and "nice to have React experience." Retrieval becomes noisy. Semantic chunking with section headers. Better. I split on markdown headers and paragraph breaks. Each chunk preserved a complete thought. But some sections were too long qualifications could run 800 tokens and others too short benefits might be two lines . Uneven chunk sizes meant wasted embedding calls on tiny sections and missed context on large ones. Recursive character splitting with overlap. This is what stayed. I split on double newlines first, then single newlines, then sentences. Each chunk targets roughly 400 tokens with some overlap. The overlap is critical: it catches the boundary cases where a sentence spans two chunks. For job listings specifically, I added a pre-processing step that normalizes the raw ATS output into a consistent structure before chunking. Greenhouse and Lever both return different formats. Normalizing first means the chunker sees clean input. js async function chunkJobListing text: string : Promise<string { const normalized = normalizeListing text // strip HTML, unify line endings const splitter = new RecursiveCharacterTextSplitter { chunkSize: 400, chunkOverlap: 50, separators: '\n\n', '\n', '. ', ' ' , } return splitter.splitText normalized } The key lesson: design your chunking around your document type, not around a generic token budget. Job listings have a natural structure. Legal documents have a different one. Product descriptions have another. Match the chunker to the content. I started with text-embedding-3-small. It works, it's reliable, and the API is trivial to call. At thousands of listings per day, each listing producing several chunks, I was generating tens of thousands of embeddings daily. The cost was manageable but not trivial. I tested a self-hosted alternative: Llama 3.1 via Ollama on the same AWS EC2 instance running the application. The embedding quality was noticeably worse for this use case. Job descriptions contain domain-specific language: "ATS-compliant resume," "Greenhouse integration," "equity compensation." The local model blurred these distinctions. Two listings that required very different skill sets ended up with similar embeddings. The tradeoff was clear. OpenAI's embeddings cost money but returned accurate matches. The self-hosted model was free but produced noisy retrieval that required more LLM calls to correct downstream. I stayed with OpenAI. One optimization that helped: I batch embedding requests. Instead of calling the API once per chunk, I send arrays of up to 100 chunks in a single call. OpenAI charges per token, not per call, so batching saves no token cost. But it reduces latency from N sequential calls to one parallel call, and it keeps the pipeline fast enough to process the daily volume within the scraping window. I evaluated both and ended up with a hybrid that surprised me. Pinecone is easy to set up and fast at query time. I had a working PoC in an afternoon. But at production scale, the cost added up fast. A Pinecone pod index with enough capacity for the vector volume I needed cost more than I wanted to pay per month. And I needed retention beyond a short window for historical matching. pgvector inside PostgreSQL was more work upfront. I had to write the index setup, the query logic, and the maintenance scripts myself. But the savings were dramatic. My existing PostgreSQL instance on the same EC2 box handled the vector workload with no additional infrastructure cost. Query performance was close enough for approximate nearest neighbor search with the right index configuration IVFFlat with appropriate lists and probes . CREATE INDEX idx job embeddings ON job chunks USING ivfflat embedding vector cosine ops WITH lists = 100 ; SELECT FROM job chunks ORDER BY embedding <= $1::vector LIMIT 20; The real win was transactional consistency. With Pinecone, I had to manage sync between PostgreSQL and the vector store. A listing could be in one but not the other during failures. With pgvector, the embedding lives in the same database as the listing data. One transaction, one source of truth. No reconciliation scripts. I use Pinecone for prototyping new RAG features. I use pgvector for production. That pattern has held across multiple projects. The embedding search returns candidate chunks. Then I need an LLM to score each job listing's relevance to a specific candidate profile. This is the expensive part. I use GPT-4o with function calling for scoring. The function schema is strict: it outputs a relevance score 0-100 , a brief justification, and a list of matching skills. The schema enforces that the model cannot fabricate matches. Every skill in the output must be present in the listing text. js const scoreFunction: ChatCompletionTool = { type: 'function', function: { name: 'score job relevance', parameters: { type: 'object', properties: { relevance score: { type: 'number', minimum: 0, maximum: 100 }, justification: { type: 'string', maxLength: 200 }, matching skills: { type: 'array', items: { type: 'string' } }, }, required: 'relevance score', 'justification', 'matching skills' , }, }, } At thousands of listings per day, scoring each one with GPT-4o would cost far more than is sustainable. Three things brought it down: OpenAI Batch API. I send scoring jobs in batches of several hundred at a time. The Batch API gives a significant discount and completes within a few hours. Since the scoring pipeline runs overnight during the scraping window, latency isn't a problem. The batch completes before the morning traffic spike. Caching. If a listing has been scored for a similar candidate profile before, I return the cached result. The cache key is a hash of the listing ID plus the candidate's skill vector. The hit rate is meaningful for repeat candidates. Model tiering. Not every listing needs GPT-4o. Listings for common roles software engineer, sales representative score well enough with GPT-4o-mini. I reserve GPT-4o for niche roles where accuracy matters more compliance officer, biomedical engineer . The routing logic is simple: if the job title matches a list of common titles, use the cheaper model. Total daily LLM cost dropped substantially. The Batch API alone accounted for much of the savings. The pipeline ran silently for a period before I discovered it was failing on a noticeable percentage of listings. The errors were non-fatal: a malformed ATS response would cause the chunker to produce empty chunks, which the embedding step would skip silently, and the listing would never appear in search results. No crashes, no alerts. I added Sentry for error tracking and LogRocket for session replay on the frontend. But the real win was adding structured logging with a correlation ID that traces a single listing through the entire pipeline: ingestion, normalization, chunking, embedding, storage, scoring. logger.info 'Listing processed', { listingId: id, source: 'greenhouse', chunks: chunkCount, embeddingLatency: ms, score: relevanceScore, } This let me build a dashboard showing pipeline health per source. Some ATS sources processed successfully at a high rate. Others failed more often because their API returns HTML in description fields that the normalizer didn't handle. I fixed the normalizer, and the failure rate dropped significantly. Without the correlation ID, I would have found that bug much later, if at all. If I started this pipeline today, I'd skip Pinecone entirely and go straight to pgvector. The setup cost is higher, but the operational simplicity of having one database is worth it. I'd also build the observability layer before the pipeline, not after. The biggest surprise was how much of the work was data normalization, not AI. Most of the bugs came from inconsistent ATS output formats, not from the LLM or the vector store. The AI parts are surprisingly reliable once you handle the data plumbing correctly. If your team is building a production RAG pipeline and hitting the same kind of silent failure modes or cost surprises, that's the kind of thing I help with. Happy to compare notes on what worked and what didn't. Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.