Paying for Cached Tokens Sucks

A developer argues that paying for cached tokens is inefficient, especially for conversational agents that reuse the same context, and proposes that inference providers expose an API allowing developers to send their own KV cache with new tokens, run inference, and receive the updated cache back, so developers own storage and lifecycle while providers sell compute, eliminating arbitrary TTLs.

paying for cached tokens sucks. even more when you already have the storage to keep the KV cache yourself this is especially painful for conversational agents: where long conversations constantly reuse the same context, but developers are forced to either pay for cached input or lose the cache when the provider's TTL expires so, what if inference providers exposed a simple API? 𐊔 send your KV cache with the new tokens, run inference, and get the updated cache back. 𐊔 developers own the storage and lifecycle of their state. providers sell the compute 𐊔 no arbitrary TTLs KV cache shouldn't be locked inside the model provider. customers should at least have the option to store and send it back