I learned the hard way that a working LLM pipeline and a production LLM pipeline are two different things.
When I first built the scoring system for a job board platform, I thought: throw GPT-4 at each listing, ask it to rate relevance, done. It worked for 100 listings. It worked for 1,000. Then I scaled to 10,000 listings a day and my OpenAI bill hit a number that made the client's eyes go wide.
That's when I stopped treating the LLM like a magic black box and started treating it like a production service with cost, latency, and failure modes.
Here's what actually matters when you're building an LLM pipeline at volume.
The first decision was how to get structured output from the LLM. Every job listing needed a relevance score, a category, and a confidence level. I needed JSON, not prose.
Some teams reach for fine-tuning when they want consistent formatting. I don't. Fine-tuning locks you into a specific model version, requires ongoing retraining as your data shifts, and gives you less control over output structure than function calling does.
OpenAI's function calling (now tool calling) lets you define a JSON schema and the model fills it in. I defined a schema with fields for relevance_score
(0-100), category
(enum of job types), and confidence
(high/medium/low). The model returns valid JSON every time because it's trained to do exactly that.
const scoringFunction = {
name: "score_listing",
description: "Score a job listing for relevance to the candidate profile",
parameters: {
type: "object",
properties: {
relevance_score: {
type: "number",
description: "Relevance score from 0 to 100",
minimum: 0,
maximum: 100
},
category: {
type: "string",
enum: ["engineering", "design", "marketing", "sales", "operations", "other"]
},
confidence: {
type: "string",
enum: ["high", "medium", "low"]
}
},
required: ["relevance_score", "category", "confidence"]
}
};
That schema gave me parseable output without regex hacks or post-processing. Every response fits the shape I defined, and if the model deviates, I catch it at the API level.
The obvious win with OpenAI's Batch API is cost: 50% cheaper than real-time endpoints. But the real win is operational.
When you process 10,000 listings a day, you don't need results in 2 seconds. You need results in 2 hours. Batch mode lets you submit all 10,000 jobs at once, walk away, and come back to a finished file. No rate limits, no connection retries, no backoff logic.
I built a pipeline that collects listings throughout the day, submits a batch at midnight, and processes results by morning. The per-listing cost dropped from about $0.003 with GPT-4o mini real-time to $0.0015 with batch. At 10,000 listings a day, that's $15 saved daily. Over a month, nearly $500.
The batch API also handles retries internally. If a job fails, OpenAI retries it automatically. I just poll for completion and check the output file.
async function submitBatch(listings: Listing[]): Promise<string> {
const batchItems = listings.map((listing, index) => ({
custom_id: `listing-${index}`,
method: "POST",
url: "/v1/chat/completions",
body: {
model: "gpt-4o-mini",
messages: [
{ role: "system", content: "Score the following job listing..." },
{ role: "user", content: JSON.stringify(listing) }
],
tools: [scoringFunction],
tool_choice: { type: "function", function: { name: "score_listing" } }
}
}));
const response = await openai.batches.create({
input_file_id: await uploadBatchFile(batchItems),
endpoint: "/v1/chat/completions",
completion_window: "24h"
});
return response.id;
}
The batch file upload step is the only extra complexity. After that, it's fire and forget.
LLMs fail in unpredictable ways. Timeouts, malformed JSON, content filters, model overloads. If you handle each failure with a retry to the same endpoint, you burn money and time.
I built a three-tier error handling system:
The key insight: not every listing needs a perfect score. Missing 1% of listings is acceptable if it keeps the pipeline running. Trying to force 100% coverage will cost you in latency and dollars.
I also added a circuit breaker. If error rates exceed 10% in a 5-minute window, the pipeline s and alerts me. That's saved me from runaway costs more than once.
When I started, I logged nothing. Then a batch of 5,000 listings returned zero results and I had no idea why. Turns out a schema change broke the function definition and every response was empty.
Now every step produces structured logs: submission time, batch ID, completion time, token usage, cost, error type. I ship these to a simple logging service and query them to find bottlenecks.
The metric that matters most is cost per useful listing. Not just cost per API call. Some listings get scored but the score is low confidence and needs human review. Those cost money without producing value. I track the ratio of high-confidence scores to total cost. That tells me if my prompt or model choice is efficient.
I also monitor latency distribution. Batch mode has a wide tail. Most jobs complete in 30 minutes, but some take 6 hours. If a batch hasn't completed after 12 hours, I investigate. Usually it's a large file that got queued behind higher-priority customers.
The original pipeline used GPT-4 for scoring. It worked, but it was expensive. I switched to GPT-4o mini and saw no meaningful drop in scoring accuracy for the use case. The categories are broad, the relevance scores are relative. A 2% error rate doesn't matter when you're ranking 10,000 listings.
I'm now evaluating DeepSeek V4 Flash as an even cheaper alternative. Early tests show similar quality at roughly 23x lower cost. If it holds up in production, that's $15/day becoming $0.65/day.
The lesson: test the cheapest model first. Upgrade only when you have evidence that quality suffers. Most teams do the reverse.
If your team is wrestling with LLM integration at scale and shipping slower because of it, that's the kind of thing I help with. I've built pipelines that process tens of thousands of items daily without breaking the bank or the API. Happy to compare notes.
Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.