Paying for Cached Tokens Sucks

paying for cached tokens sucks. even more when you already have the storage to keep the KV cache yourself this is especially painful for conversational agents: where long conversations constantly reuse the same context, but developers are forced to either pay for cached input or lose the cache when the provider's TTL expires so, what if inference providers exposed a simple API? 𐊔 send your KV cache with the new tokens, run inference, and get the updated cache back. 𐊔 developers own the storage and lifecycle of their state. providers sell the compute 𐊔 no arbitrary TTLs KV cache shouldn't be locked inside the model provider. customers should at least have the option to store and send it back