{"slug": "shopify-introduces-gisting-compressing-llm-system-prompts-into-learned-tokens", "title": "Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens", "summary": "Shopify introduced Gisting, a technique that compresses long LLM system prompts into learned 'gist' tokens, reducing the Sidekick GraphQL agent's prompt from about 6,000 tokens to 1,500 tokens without sacrificing prediction quality. At 350 requests per minute, median time to first token dropped from 438ms to 354ms, median end-to-end latency fell from 6.8s to 4.2s, and throughput rose from 20.2 to 23.4 queries per second, enabling reduced GPU allocation.", "body_md": "Shopify's engineering introduced [Gisting, a novel technique for compressing long LLM prompts](https://shopify.engineering/gisting) into a smaller set of learned \"gist\" tokens, improving throughput and reducing inference cost.\n\nSpotify emphasizes that replacing lengthy text for concise gist tokens at inference time reduces end-to-end latency, drops infrastructure costs, and boosts token throughput without modifying the model's core weights.\n\nThe company says that gisting reduced the Sidekick GraphQL agent’s system prompt from about 6000 tokens to 1500 gist tokens without sacrificing prediction quality. This implies a 4:1 reduction in context size:\n\nAt 350 requests per minute (RPM), the median time to first token (TTFT) dropped from 438ms to 354ms, the median end-to-end request latency dropped from 6.8s to 4.2s, and throughput rose from 20.2 to 23.4 queries per second (QPS).\n\nThese improved metrics allowed Spotify to reduce the number of allocated GPUs.\n\nGisting is based on a technique pioneered in a 2022 paper, [\" Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models\"](https://arxiv.org/abs/2210.03162), and consists of a two-step process to learn the embeddings of the new compressed gist token. In a first pass, the\n\n*teacher pass*, the model is run with the real prompt to derive the\n\n*teacher logits*of the response. In the\n\n*student pass*, the model is run with the gist tokens to derive the student logits. Finally, the gist are trained to minimize the KL divergence between the teacher logits and the student logits, that is until the student's predictions closely match the teacher's.\n\nWhen training finishes, we write the gist embeddings straight into the model's embedding matrix, and register the new gist tokens as special tokens in the model’s tokenizer. The model loads and runs like any other at inference time: no custom attention mask, extra encoder, or special serving path.\n\nThe key advantage of gisting is that the model does not process a conventional summary of the original prompt, but rather a learned representation designed to make the LLM to behave as close as possible to how it would if it had seen the original prompt.\n\nGisting can reduce latency and increase throughput. In Shopify's case, Time to First Token (TTFT) dropped from 438ms to 354ms, and end-to-end latency fell from 6.8s to 4.2s. At the same time, queries per second (QPS) increased from 20.2 to 23.4, allowing engineering teams to scale down overall GPU allocation.\n\nAs a final note, Shopify also emphasizes that gisting is complementary to other optimization techniques, such as [prefix caching](https://handbook.modular.com/inference-optimization/prefix-caching/). Prefix caching avoids recomputing the KV tensors for cached prompt sequences, but the model must still process those cached tensors during the decoding phase. Gisting further reduces this overhead by replacing a long prompt with a shorter sequence of learned gist tokens. The two optimizations therefore compound, and Shopify uses them together.\n\nThere is much more to gisting than can be covered here. Make sure to read the original article if you are interested in the full details, which covers topics such as the role of autosearch in tuning the Gisting process and other implementation details that significantly affect performance.", "url": "https://wpnews.pro/news/shopify-introduces-gisting-compressing-llm-system-prompts-into-learned-tokens", "canonical_source": "https://www.infoq.com/news/2026/09/spotify-gisting-llm-performance/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global", "published_at": "2026-09-03 20:00:00+00:00", "updated_at": "2026-09-03 20:22:26.560211+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure"], "entities": ["Shopify", "Gisting", "Sidekick GraphQL agent"], "alternates": {"html": "https://wpnews.pro/news/shopify-introduces-gisting-compressing-llm-system-prompts-into-learned-tokens", "markdown": "https://wpnews.pro/news/shopify-introduces-gisting-compressing-llm-system-prompts-into-learned-tokens.md", "text": "https://wpnews.pro/news/shopify-introduces-gisting-compressing-llm-system-prompts-into-learned-tokens.txt", "jsonld": "https://wpnews.pro/news/shopify-introduces-gisting-compressing-llm-system-prompts-into-learned-tokens.jsonld"}}