I'm a developer working on Paramour.chat, an LLM roleplay app where, of course, exchanges need moderation. The annoying part is that all those checks take time. A few seconds of dead air may not sound like much, but in a chat they feel endless. Streaming keeps the conversation moving, but it creates a nastier problem: once a word appears in the browser, it has crossed the network, been delivered, and may already have been read by the user.
I tried two good approaches: buffer the whole reply until the gate check finishes, or hold one sentence ahead and release it only after it clears the gate.
A usual chat stream has almost no protocol. The provider emits a token, the server forwards it, and the client appends it to a message. By the time generation stops, delivery has already happened.
Moderation adds a boundary between held and cleared. Held text has not left the server and may still be discarded. Cleared text has passed the required gates and may be delivered. There is no client-side draft that can be retracted later.
Full buffering is easy to reason about: generate the reply, classify it and resolve any escalation, then send it. A rejected reply never reaches the client.
It also makes five seconds of generation look like five seconds of nothing. That is a very annoying experience, although some people prefer the because it feels more like chatting with another person. To stream without giving up the same default, I let a turn earn the fast path.
The input gate starts alongside the writer. The first output remains held. At the boundary between reasoning and prose, the server usually has the input verdict.
Anything sent down an escalation path stays buffered too. The delivery layer does not need to know how that escalation will be resolved; it only needs to avoid releasing unchecked output.
Basically, when we break, we break closed.
Content classifying models are small and fast. That is good for more than cost: their latency can fit inside the time already spent displaying the reply.
Consider the response as a sequence of sentences. Sentence n has already cleared moderation and is being streamed to the user. Meanwhile, the writer finishes sentence n + 1 and sends it through moderation. If that check completes before the display reaches the end of sentence n, the next cleared sentence follows with no visible .
The check stays hidden when sentence n takes at least as long to display as sentence n + 1 takes to finish generating and pass classifiers.
We can stretch the left side slightly by pacing output below the writer's native speed. The display rate stays at the greater of two values: the rate needed to cover the next calls to classifiers, and the minimum rate that still feels like a live response. If the minimum wins, the buffer runs out and the user sees a .
This works well only when both models are fast. I treat about 50 tokens per second as the lower bound for the writer; below that, there is too little surplus output to hide another network call. Faster writers build a deeper lead over the display pacer and make the latency disappear more often. Fast classifiers matter for exactly the same reason.
The result is a small pipeline:
The user watches sentence n stream while the next unit of work moves through generation and classification. Streaming is no longer the raw decode rate of the writer. It is the final stage of a pipeline with a little inventory between each stage.
“Moderate before sending” is too easy to break during a later refactor. The raw writer stream should have no path to the client at all. In my implementation, the writer can only write into a server-side sentence buffer. The browser transport can only read from a separate queue of cleared sentences.
A sentence moves between those two places after every required check passes. A timeout or classifier error leaves it held. A failure stops the release and sends the turn down the escalation path. If a gate needs the complete reply, that turn never enters the sentence pipeline and stays fully buffered.
This separation means we can change models, thresholds, and pacing without changing the delivery guarantee. If unchecked writer output can reach the socket anywhere except through the clearing step, the boundary is in the wrong place.
When the writer slows down or moderation takes too long, the cleared queue runs dry and the user waits. That is the trade we value: the experience may become slower, but the gate does not become weaker.
This is the design behind moderation at Paramour.chat. If you've built something similar, how did you handle all this?