Fitting LLM Reply Suggestions Into Every Provider's Prompt Cache — Without Structured Output
A developer implemented inline markers embedded in LLM responses to generate reply suggestions for a voice roleplay chat, avoiding structured output, stream interruption, and cache invalidation. The s…