{"slug": "my-ai-integration-had-terrible-costs-until-i-changed-my-approach", "title": "My AI integration had terrible costs until I changed my approach", "summary": "A developer building a SaaS article summarizer reduced API costs from $1,200/month to a fraction by combining extractive and abstractive summarization. The pipeline uses TextRank to extract key sentences locally, then sends only those to GPT-3.5-turbo for rewriting. This approach cut token usage by 90% while maintaining summary quality close to GPT-4.", "body_md": "Last month I was working on a SaaS product that needed to summarize long articles for users. Think of it like a TL;DR generator. I built a first prototype using GPT-4 with a straightforward prompt: *\"Summarize this article in 3 bullet points.\"* It worked beautifully. The summaries were crisp, accurate, and users loved them.\n\nThen the API bills arrived. One month of moderate usage cost me over $1,200. That's not sustainable for a side project. I had to fix it or kill the feature.\n\nFirst, I tried switching to GPT-3.5-turbo. The price dropped dramatically, but the quality tanked. Summaries became vague, sometimes missing key points. I tried prompt engineering — adding \"be specific\" or \"include numbers\" — but nothing reliably matched GPT-4's output.\n\nNext, I tried reducing the input size. I used extractive summarization libraries like `sumy`\n\nto grab the most important sentences before sending them to GPT. That helped a bit, but the cost was still high because the extracted text was still large, and GPT-3.5 still hallucinated on long inputs.\n\nI also considered using a local model like Llama 2, but my server couldn't handle the inference latency. My users expected summaries in under 3 seconds.\n\nThe insight came from reading research papers on summarization. Pure extractive (picking sentences) is fast but rigid. Pure abstractive (generating new sentences) is flexible but expensive. The sweet spot? Use extractive to shrink the text, then use a small abstractive model to rewrite the summary elegantly.\n\nI implemented a pipeline:\n\n`textrank`\n\nor `bert-extractive-summarizer`\n\nwith a tiny model) to pick the top 5–10 sentences from the article.This slashed costs by 80% while keeping quality closer to GPT-4. The extractive step removes 90% of the input tokens, so the API call is tiny.\n\nHere's a simplified Python version using a generic API endpoint (you can swap in any compatible service, for example [https://ai.interwestinfo.com/](https://ai.interwestinfo.com/) or OpenAI):\n\n``` python\nimport requests\nfrom sumy.parsers.plaintext import PlaintextParser\nfrom sumy.nlp.tokenizers import Tokenizer\nfrom sumy.summarizers.text_rank import TextRankSummarizer\n\ndef extract_key_sentences(text, sentence_count=10):\n    parser = PlaintextParser.from_string(text, Tokenizer(\"english\"))\n    summarizer = TextRankSummarizer()\n    sentences = summarizer(parser.document, sentence_count)\n    return \" \".join(str(s) for s in sentences)\n\ndef abstractive_summarize(key_sentences, api_key, endpoint):\n    # endpoint could be e.g. \"https://ai.interwestinfo.com/v1/completions\"\n    payload = {\n        \"model\": \"gpt-3.5-turbo\",  # or any cheap model\n        \"messages\": [\n            {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n            {\"role\": \"user\", \"content\": f\"Combine these key sentences into a fluent 3-bullet summary:\\n\\n{key_sentences}\"}\n        ],\n        \"temperature\": 0.3,\n        \"max_tokens\": 150\n    }\n    headers = {\"Authorization\": f\"Bearer {api_key}\", \"Content-Type\": \"application/json\"}\n    response = requests.post(endpoint, json=payload, headers=headers)\n    return response.json()[\"choices\"][0][\"message\"][\"content\"]\n\n# Usage\narticle_text = \"...long article...\"\nkey_sentences = extract_key_sentences(article_text, 8)\nsummary = abstractive_summarize(key_sentences, api_key=\"sk-...\", endpoint=\"https://api.openai.com/v1/chat/completions\")\nprint(summary)\n```\n\nThis code runs the extractive summarizer locally (fast, free) and only makes one small API call. I added caching on the extractive step so repeated articles don't re-run the same sentences.\n\nThis approach isn't perfect. Some articles require a fully abstractive summary because the extractive phase may miss the connecting logic. For example, if the original text builds an argument step-by-step, picking random key sentences loses the flow. In those cases, I fall back to a single GPT-4 call, but I limit it to articles above a certain complexity (detected by sentence count or named entity density).\n\nAlso, the extractive step with TextRank is non-deterministic — you might get different results on re-runs. I switched to a deterministic variant (using a fixed random seed) to ensure consistency.\n\nAnother lesson: caching is your best friend. I cache both extractive results (by article hash) and abstractive results (by hash + model). In production, thousands of users read the same articles, so cache hit rates are high.\n\nIf I rebuild this, I'd:\n\nThe real lesson isn't about any specific API or library. It's about thinking in layers: you don't always need a sledgehammer. By breaking a complex AI task into cheaper sub-tasks, you solve both cost and latency.\n\nNow I'm curious: **What's your approach to balancing AI quality and cost in production? Do you use cascading models or other tricks?**", "url": "https://wpnews.pro/news/my-ai-integration-had-terrible-costs-until-i-changed-my-approach", "canonical_source": "https://dev.to/__c1b9e06dc90a7e0a676b/my-ai-integration-had-terrible-costs-until-i-changed-my-approach-pml", "published_at": "2026-06-21 08:01:02+00:00", "updated_at": "2026-06-21 08:37:29.045030+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "natural-language-processing", "developer-tools", "ai-products"], "entities": ["OpenAI", "GPT-4", "GPT-3.5-turbo", "Llama 2", "TextRank", "sumy", "InterWest Info"], "alternates": {"html": "https://wpnews.pro/news/my-ai-integration-had-terrible-costs-until-i-changed-my-approach", "markdown": "https://wpnews.pro/news/my-ai-integration-had-terrible-costs-until-i-changed-my-approach.md", "text": "https://wpnews.pro/news/my-ai-integration-had-terrible-costs-until-i-changed-my-approach.txt", "jsonld": "https://wpnews.pro/news/my-ai-integration-had-terrible-costs-until-i-changed-my-approach.jsonld"}}