{"slug": "token-cost-blew-up-in-production-how-are-you-handling-context-compression-for", "title": "Token Cost Blew Up in Production: How Are You Handling Context Compression for Mobile-Embedded LLM Features?", "summary": "A mobile app developer reports that an LLM-powered assistant's production token costs ran roughly 4x higher than modeled, driven by quadratic growth from resending full conversation history and re-injecting identical RAG chunks each turn. The developer mitigated costs via prompt caching (about 90% reduction on cached portions) and summarization of older turns, and is seeking advice from others building mobile-embedded LLM features.", "body_md": "Posting this partly as a write-up of what we hit and partly because I’d genuinely like to know how other people building LLM features into mobile apps are handling the same problem, because the docs and blog posts I found were all optimized for server-side / chat-app use cases, not the constraints we’re actually working under.\n\nWe shipped an LLM-powered assistant inside a mobile app a few months back — nothing exotic, RAG over a product knowledge base plus conversational memory, calling a hosted API rather than running anything on-device. Development and QA ran against a few hundred test conversations and the per-session cost looked completely reasonable. Scaled to real users, the bill was roughly 4x what we’d modeled, and it took a while to figure out why, because nothing in the architecture was “wrong” in the sense of throwing errors or producing bad outputs. It was just expensive in a way our testing hadn’t surfaced.\n\nTwo mechanisms, once we dug in, accounted for almost all of the gap.\n\nObvious in retrospect, embarrassing that it took a cost review to notice: every turn in the conversation was sending the full prior history as context. Turn 1 sends the system prompt plus the user’s first message. Turn 6 sends the system prompt plus all eleven prior messages plus the new one. On a provider that bills input tokens per call rather than per session, this means a six-turn conversation costs roughly O(n²) in cumulative input tokens relative to conversation length, not O(n). Our test conversations during QA averaged 2-3 turns because testers were, understandably, not trying to have realistic extended conversations with a support bot. Real users, especially the frustrated ones actually needing help, were running 8-12 turn conversations, where the quadratic cost growth actually bites.\n\nThe mobile constraint that made this worse than it would be for a desktop/server chat product: because the app is stateless between app-kills and has to reconstruct conversation state from a local cache or a backend session store, there was no natural pressure to ever prune history — the full history object was just… there, and got sent, because nobody had written code to not send it.\n\nThe second one is more specific to RAG-in-mobile and I haven’t found much discussion of it. Our retrieval step pulls the top-k chunks from the knowledge base relevant to the current query and injects them into context. Reasonable on turn 1. But the naive implementation was re-running retrieval and re-injecting fresh chunks on every turn, even when the topic of conversation hadn’t shifted, which meant a single support conversation about, say, a return policy was paying for the same three or four knowledge-base chunks repeatedly across 8+ turns because nothing was tracking “we already have this context loaded, don’t refetch/resend it.”\n\nCombined with mechanism 1 (full history resend), the actual input token count for turn 10 of a support conversation was: system prompt + full 9-turn history + freshly retrieved RAG chunks for the current turn, even when the RAG chunks were near-identical to what had already been sent four turns earlier.\n\n**Prompt caching where the provider supports it.** The system prompt and, where feasible, the RAG context block are structured as a stable prefix so repeated calls hit cache on the unchanged portion instead of paying full input rate every time. This was the single biggest lever — the discounted cache-hit rate on a large provider is roughly a 90% reduction versus standard input pricing on the cached portion, and our system prompt plus retrieved-context block was a meaningful fraction of total input tokens on later turns, so this alone materially changed the shape of the cost curve.\n\n**Summarize-and-truncate for conversation history beyond a window.** Rather than sending full verbatim history indefinitely, we now keep the last N turns verbatim and periodically compress older turns into a running summary generated by a cheaper/smaller model call. This trades a small amount of fidelity (the model working from a summary rather than verbatim old messages) for keeping the O(n²) growth from ever compounding past a bounded window. We’re still tuning N — too small and the assistant loses thread on longer support conversations, too large and we’ve just moved the quadratic-growth problem further out instead of solving it.\n\n**RAG retrieval caching keyed to detected topic stability.** Instead of re-retrieving on every turn, we added a lightweight check for whether the current turn is a topic continuation versus a topic shift (cheap classifier call, not a full LLM call) and only re-run retrieval when the topic has actually moved. This cut redundant chunk-resends substantially, though I’ll be honest that the topic-continuation classifier itself isn’t perfect and occasionally either re-fetches when it didn’t need to or fails to re-fetch when the topic genuinely shifted subtly.\n\n**Model routing by turn complexity.** Not every turn in a support conversation needs the same model. A lot of turns are simple confirmations, clarifying questions, or short factual lookups that a smaller/cheaper model handles fine, and we route those away from the larger model we use for the harder reasoning turns. This is probably the least mobile-specific of the four changes but it stacked with the others meaningfully.\n\nNet effect across all four changes: per-session cost came down to roughly a quarter of the naive-implementation number for equivalent conversation lengths, which lines up with the 60-80% reduction range that keeps showing up in general LLM-cost-optimization writeups — it’s just that none of those writeups were written with “conversation state reconstructed from a mobile local cache” as the starting condition, so getting there took more trial and error than I expected going in.\n\nA few things I haven’t found good answers to, and would like this community’s take on:\n\n**On-device summarization vs. server-side.** We’re doing the history-compression summary call server-side right now because running even a small model on-device felt like scope creep for a v1, but it’s an extra round-trip and extra latency on top of everything else. Has anyone had success running a lightweight summarization model on-device (mobile) specifically to compress conversation history before it ever gets sent, rather than sending full history and compressing server-side? Curious what model sizes are actually workable on mid-range Android hardware for this specific narrow task, as opposed to general on-device inference benchmarks which tend to assume flagship hardware.\n\n**Better topic-shift detection for RAG re-triggering.** Our current classifier approach for “should we re-run retrieval” feels like a hack. Is there a more principled way people are handling “when to refresh retrieved context in a multi-turn conversation” that doesn’t require training a bespoke classifier per use case?\n\n**Session boundaries and cache invalidation on mobile specifically.** Because the app can be backgrounded, killed, and restored, our conversation state persistence doesn’t map cleanly onto how most prompt-caching documentation assumes a session works (continuous, server-managed). If the user backgrounds the app for six hours and comes back, is it worth invalidating the cached prefix and starting fresh, or does the caching mechanism handle staleness gracefully enough that it doesn’t matter? We’ve been treating it conservatively (short cache TTL assumptions) without much evidence either way.\n\nFor anyone hitting this same wall, the part that took longest to get right wasn’t the concept (stable prefix = cacheable) but the actual prompt structuring discipline required to keep the prefix genuinely stable across calls. It’s easy to say “put the system prompt and RAG context first, user-specific stuff last” and much harder to enforce in practice once you have multiple engineers touching the prompt-assembly code and someone innocently inserts a timestamp, a user ID, or a dynamically-generated instruction ahead of the cacheable block — which invalidates the cache for that entire prefix on every single call without any obvious error or warning, just a cost graph that doesn’t move the way you expected after you “fixed” caching. We ended up writing a small internal lint-style check that flags any prompt-assembly change touching the first N tokens of the template, specifically because this bug was silent and expensive and we didn’t want to reintroduce it by accident three sprints later.\n\nThe other non-obvious part: caching only helps if the prefix is actually reused within whatever TTL window the provider maintains it for. A support conversation with long gaps between user replies (someone gets distracted, comes back forty minutes later) may fall outside that window depending on provider and plan tier, silently reverting to full-price input on the next call with no error thrown — which made our early cost dashboards confusing until we correlated cache-miss spikes with conversation gap duration and realized what was happening.\n\nNone of the four changes above are individually novel — caching, summarization, retrieval gating, and model routing are all well-documented techniques in isolation. What made this specific to mobile, in our experience, was that the standard reference implementations for all four assume a session model (continuous connection, server-managed state, predictable turn cadence) that doesn’t map cleanly onto how a mobile app’s lifecycle actually behaves — backgrounding, app kills, variable-length gaps between turns, and conversation state that has to be reconstructed rather than continuously held. Every optimization technique needed a mobile-specific adaptation layer on top of the “textbook” version, and that adaptation layer is where most of our actual engineering time went, not in implementing the underlying techniques themselves.\n\nIf it’s useful context: I do this work at a[ mobile app development company](https://devtechnosys.com/mobile-app-development.php), and this specific project was client-facing rather than internal tooling, which is part of why the cost surprise mattered enough to dig into properly rather than just bumping the budget and moving on. Happy to share more implementation detail on any of the four changes above if useful to anyone hitting the same wall — and genuinely interested if anyone has cracked the on-device summarization piece, since that feels like the most mobile-specific unsolved part of this for us.", "url": "https://wpnews.pro/news/token-cost-blew-up-in-production-how-are-you-handling-context-compression-for", "canonical_source": "https://discuss.huggingface.co/t/token-cost-blew-up-in-production-how-are-you-handling-context-compression-for-mobile-embedded-llm-features/179178#post_1", "published_at": "2026-08-24 07:23:57+00:00", "updated_at": "2026-08-24 09:14:07.556162+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-products"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/token-cost-blew-up-in-production-how-are-you-handling-context-compression-for", "markdown": "https://wpnews.pro/news/token-cost-blew-up-in-production-how-are-you-handling-context-compression-for.md", "text": "https://wpnews.pro/news/token-cost-blew-up-in-production-how-are-you-handling-context-compression-for.txt", "jsonld": "https://wpnews.pro/news/token-cost-blew-up-in-production-how-are-you-handling-context-compression-for.jsonld"}}