Building a Production-Grade AI Pipeline: Scoring 10,000+ Listings Daily with LLMs A developer built a production-grade AI pipeline using OpenAI's function calling and Batch API to score over 10,000 job listings daily. By switching from real-time to batch processing, the pipeline reduced costs by 50% and eliminated rate-limit issues. The system uses structured JSON output via function calling to ensure consistent, parseable results without post-processing. I learned the hard way that a working LLM pipeline and a production LLM pipeline are two different things. When I first built the scoring system for a job board platform, I thought: throw GPT-4 at each listing, ask it to rate relevance, done. It worked for 100 listings. It worked for 1,000. Then I scaled to 10,000 listings a day and my OpenAI bill hit a number that made the client's eyes go wide. That's when I stopped treating the LLM like a magic black box and started treating it like a production service with cost, latency, and failure modes. Here's what actually matters when you're building an LLM pipeline at volume. The first decision was how to get structured output from the LLM. Every job listing needed a relevance score, a category, and a confidence level. I needed JSON, not prose. Some teams reach for fine-tuning when they want consistent formatting. I don't. Fine-tuning locks you into a specific model version, requires ongoing retraining as your data shifts, and gives you less control over output structure than function calling does. OpenAI's function calling now tool calling lets you define a JSON schema and the model fills it in. I defined a schema with fields for relevance score 0-100 , category enum of job types , and confidence high/medium/low . The model returns valid JSON every time because it's trained to do exactly that. js const scoringFunction = { name: "score listing", description: "Score a job listing for relevance to the candidate profile", parameters: { type: "object", properties: { relevance score: { type: "number", description: "Relevance score from 0 to 100", minimum: 0, maximum: 100 }, category: { type: "string", enum: "engineering", "design", "marketing", "sales", "operations", "other" }, confidence: { type: "string", enum: "high", "medium", "low" } }, required: "relevance score", "category", "confidence" } }; That schema gave me parseable output without regex hacks or post-processing. Every response fits the shape I defined, and if the model deviates, I catch it at the API level. The obvious win with OpenAI's Batch API is cost: 50% cheaper than real-time endpoints. But the real win is operational. When you process 10,000 listings a day, you don't need results in 2 seconds. You need results in 2 hours. Batch mode lets you submit all 10,000 jobs at once, walk away, and come back to a finished file. No rate limits, no connection retries, no backoff logic. I built a pipeline that collects listings throughout the day, submits a batch at midnight, and processes results by morning. The per-listing cost dropped from about $0.003 with GPT-4o mini real-time to $0.0015 with batch. At 10,000 listings a day, that's $15 saved daily. Over a month, nearly $500. The batch API also handles retries internally. If a job fails, OpenAI retries it automatically. I just poll for completion and check the output file. js async function submitBatch listings: Listing : Promise