My AI integration had terrible costs until I changed my approach

Last month I was working on a SaaS product that needed to summarize long articles for users. Think of it like a TL;DR generator. I built a first prototype using GPT-4 with a straightforward prompt: "Summarize this article in 3 bullet points." It worked beautifully. The summaries were crisp, accurate, and users loved them.

Then the API bills arrived. One month of moderate usage cost me over $1,200. That's not sustainable for a side project. I had to fix it or kill the feature.

First, I tried switching to GPT-3.5-turbo. The price dropped dramatically, but the quality tanked. Summaries became vague, sometimes missing key points. I tried prompt engineering — adding "be specific" or "include numbers" — but nothing reliably matched GPT-4's output.

Next, I tried reducing the input size. I used extractive summarization libraries like sumy

to grab the most important sentences before sending them to GPT. That helped a bit, but the cost was still high because the extracted text was still large, and GPT-3.5 still hallucinated on long inputs.

I also considered using a local model like Llama 2, but my server couldn't handle the inference latency. My users expected summaries in under 3 seconds.

The insight came from reading research papers on summarization. Pure extractive (picking sentences) is fast but rigid. Pure abstractive (generating new sentences) is flexible but expensive. The sweet spot? Use extractive to shrink the text, then use a small abstractive model to rewrite the summary elegantly.

I implemented a pipeline:

textrank

or bert-extractive-summarizer

with a tiny model) to pick the top 5–10 sentences from the article.This slashed costs by 80% while keeping quality closer to GPT-4. The extractive step removes 90% of the input tokens, so the API call is tiny.

Here's a simplified Python version using a generic API endpoint (you can swap in any compatible service, for example https://ai.interwestinfo.com/ or OpenAI):

import requests
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.text_rank import TextRankSummarizer

def extract_key_sentences(text, sentence_count=10):
    parser = PlaintextParser.from_string(text, Tokenizer("english"))
    summarizer = TextRankSummarizer()
    sentences = summarizer(parser.document, sentence_count)
    return " ".join(str(s) for s in sentences)

def abstractive_summarize(key_sentences, api_key, endpoint):
    payload = {
        "model": "gpt-3.5-turbo",  # or any cheap model
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": f"Combine these key sentences into a fluent 3-bullet summary:\n\n{key_sentences}"}
        ],
        "temperature": 0.3,
        "max_tokens": 150
    }
    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
    response = requests.post(endpoint, json=payload, headers=headers)
    return response.json()["choices"][0]["message"]["content"]

article_text = "...long article..."
key_sentences = extract_key_sentences(article_text, 8)
summary = abstractive_summarize(key_sentences, api_key="sk-...", endpoint="https://api.openai.com/v1/chat/completions")
print(summary)

This code runs the extractive summarizer locally (fast, free) and only makes one small API call. I added caching on the extractive step so repeated articles don't re-run the same sentences.

This approach isn't perfect. Some articles require a fully abstractive summary because the extractive phase may miss the connecting logic. For example, if the original text builds an argument step-by-step, picking random key sentences loses the flow. In those cases, I fall back to a single GPT-4 call, but I limit it to articles above a certain complexity (detected by sentence count or named entity density).

Also, the extractive step with TextRank is non-deterministic — you might get different results on re-runs. I switched to a deterministic variant (using a fixed random seed) to ensure consistency.

Another lesson: caching is your best friend. I cache both extractive results (by article hash) and abstractive results (by hash + model). In production, thousands of users read the same articles, so cache hit rates are high.

If I rebuild this, I'd:

The real lesson isn't about any specific API or library. It's about thinking in layers: you don't always need a sledgehammer. By breaking a complex AI task into cheaper sub-tasks, you solve both cost and latency.

Now I'm curious: What's your approach to balancing AI quality and cost in production? Do you use cascading models or other tricks?