I spent months building an AI scoring pipeline that processes over 10,000 job listings every day. The first version was a mess. Slow, expensive, and unreliable. The second version worked. Here's what I learned about the architecture decisions that actually matter when you put LLMs in a production data pipeline.
Most tutorials show you how to call an API and get a response. They don't show you what happens when you need to do that 10,000 times a day without burning your budget or losing accuracy. I had to figure that out the hard way.
My first approach was embarrassingly simple. Take a job listing, dump the full text into a GPT prompt, and ask for a relevance score. It worked on three test listings. It fell apart at 500.
Three problems emerged immediately.
First, latency. Each call took 3 to 8 seconds. At 10,000 listings, that's 8 to 22 hours of sequential processing. Even with parallel batching, the wall clock time was unacceptable for a system that needed to surface fresh listings within minutes.
Second, cost. GPT-4 token counts were all over the place. Some job descriptions were 200 words. Others were 2,000. I was paying for the long ones the same way as the short ones, with no control over how much context the model consumed.
Third, inconsistency. The same listing scored differently depending on how I phrased the prompt. Minor wording changes produced wildly different relevance scores. That's fine for a prototype. It's a disaster for a production system where recruiters depend on consistent ranking.
I needed structured output, predictable token usage, and a pipeline that could scale horizontally without breaking the bank.
GPT-4 function calling gave me a way to enforce structure on the output. Instead of asking for a score and hoping the model returned a number, I defined a schema that the model had to follow.
const scoringSchema = {
name: "score_job_listing",
description: "Score a job listing for relevance to a candidate profile",
parameters: {
type: "object",
properties: {
relevance_score: {
type: "number",
description: "Relevance score from 0 to 100",
minimum: 0,
maximum: 100
},
match_reasons: {
type: "array",
items: { type: "string" },
description: "Top 3 reasons for this score"
},
skill_match_count: {
type: "integer",
description: "Number of candidate skills found in the listing"
},
seniority_level: {
type: "string",
enum: ["junior", "mid", "senior", "lead", "executive"]
}
},
required: ["relevance_score", "match_reasons", "skill_match_count", "seniority_level"]
}
};
This forced the model to return a predictable JSON object every time. No more parsing free text. No more guessing whether a score was 75 or 0.75. The schema acted as a contract between the pipeline and the LLM.
The real win was downstream. With structured output, I could store scores directly in PostgreSQL, build SQL queries that filtered by score thresholds, and feed the results into a REST API without any transformation layer. The pipeline became predictable.
The final architecture had four stages, each with a specific job.
Stage one was ingestion. Listings arrived from multiple ATS sources via their public APIs. I normalized them into a common schema: title, description, company, location, skills. No LLM calls at this stage. Just data cleaning and deduplication.
Stage two was pre-filtering. Before any AI call, I ran a fast keyword and rule-based filter. Listings that clearly didn't match the candidate's criteria were dropped without ever touching the LLM. This eliminated about 40% of listings and saved a significant chunk of API costs.
Stage three was the scoring pipeline. This ran in batches of 50 listings per worker, with each batch sent to GPT-4 via function calling. I used a simple queue system with retry logic. Failed calls were retried up to three times with exponential backoff. Listings that failed all retries were logged and sent to a dead-letter queue for manual review.
Stage four was delivery. Scored listings were written to the database with their relevance score and match reasons. The REST API served them to the frontend and to external integrations. The entire pipeline from ingestion to API availability was under 5 minutes for most listings.
The biggest mistake I see teams make is treating LLM calls like regular API calls. They're not. A single GPT-4 call can cost ten times more than another depending on input length and output structure.
I implemented three cost controls that made the difference between viable and unviable.
First, I used a shorter model for pre-filtering. Before sending a listing to GPT-4 for scoring, I ran a lightweight GPT-4o-mini call that classified listings into three buckets: likely match, possible match, and clear miss. Only the first two buckets went to GPT-4. This cut GPT-4 usage by about 60%.
Second, I optimized prompt length aggressively. Every word in the system prompt cost money on every call. I trimmed candidate profiles to the essential signals: skills, experience level, preferred locations. No biography, no career narrative, no fluff.
Third, I cached aggressively. Listings that were scored once and hadn't changed were never rescored. The same candidate profile scored the same listing only once. This sounds obvious, but I've seen production systems that rescored everything on every pipeline run because nobody added a cache check.
Total cost per listing landed well under one cent. At 10,000 listings daily, that's a manageable operational expense. Without these controls, it would have been five to ten times higher.
If I were building this today, I'd make three changes.
I'd use a batch processing strategy from day one instead of the streaming approach I started with. The OpenAI Batch API offers 50% cost reduction for asynchronous workloads. My pipeline was mostly batch-friendly, but I designed it for real-time processing first and had to retrofit.
I'd invest more in the pre-filtering stage. The keyword and rule-based filter was effective, but a lightweight embedding similarity search would have caught more false positives before they reached the LLM. I added this later, but it should have been in the initial design.
I'd build better observability into the scoring pipeline. I had basic logging, but I didn't track score drift over time. I only noticed that scores were trending downward after a prompt change when a client complained. A simple dashboard tracking average scores per day would have caught this immediately.
Production AI pipelines have more in common with ETL systems than with chat applications. The same engineering practices that make data pipelines reliable apply here: idempotency, retry logic, dead-letter queues, monitoring, and cost tracking. The LLM is just one component in a larger system.
If your team is building this kind of pipeline and wondering how to keep accuracy high while managing costs at scale, that's the kind of thing I help with. Happy to compare notes on what's worked and what hasn't.
Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.