📝 Originally published (in Japanese) at
[forge.workstyle.tech].
When creating a live stream where an AI avatar operates autonomously, the first major hurdle you encounter is the issue of "silence." It’s not that there are no viewers—quite the opposite. Yet the avatar falls silent for long stretches, or ignores comments for tens of seconds. What human streamers do unconsciously—creating "space" in the conversation—is entirely missing from AI behavior.
In this article, I’ll summarize two key challenges we tackled to prevent unmanned streams from becoming boring. The first: how to fill the silence when no comments arrive. The second: how to handle response delays when comments do arrive. The former deals with behavior during "no input," while the latter concerns the time between input and reaction. Both are two sides of the same coin in live streaming, and neither worked with a straightforward implementation.
What they had in common was that brute-force attempts to "make it faster" or "make it smarter" missed the mark. We had to observe long-running streams, measure breakdowns, and redesign priorities—mundane but essential work.
Our initial implementation was straightforward: "Respond when a comment arrives." Functionally, it worked correctly and passed tests.
The problem was what happens when no comments arrive.
In an unmanned stream, the avatar stands frozen on screen for tens of seconds—blinking, but doing nothing. This is nearly an accident for a live stream. And for newly launched channels, this is the default state. Comments come only after the stream has grown; until then, silence is the norm.
This was a design philosophy issue. If built as a chatbot, the AI only outputs in response to input—just like a web request/response model.
But a streamer is different. Their job is to keep talking even when no one says anything.
So we needed a mechanism that generates speech regardless of input. Here’s what we implemented:
If 75 seconds pass after the last utterance with no activity →
Generate and speak a topic based on the stream’s theme
The number 75 seconds isn’t backed by strong theory—we determined it by observing actual streams. Here’s what guided us:
Another critical factor was viewer delay. Viewers’ screens lag by 15–30 seconds. So when they react to a comment and hit "send," it takes time to reach the stream. If the threshold is shorter than this round-trip, conversation breaks down.
This viewer delay reappears later as a dominant factor in response speed. Though unmanned speech and response delay seem like separate features, they’re both constrained by the same "platform latency" in live streaming.
We made the silence threshold configurable so it can be adjusted per stream type.
In implementation, utterances are managed in a priority queue. Spontaneous topics go at the lowest priority.
| Priority | Type |
|---|---|
| Highest | Closing remarks at stream end |
| High | Events like tips or subscriptions |
| Medium | Responses to viewer comments |
| Low | |
| Spontaneous topics to fill silence |
This ensures that if a comment arrives while a spontaneous topic is being generated, the response to the comment takes precedence. Prioritizing monologue over waiting for viewers is clearly wrong for a stream.
"Filler content" should always be interruptible.
Topics are generated from the stream’s theme, set when the stream is registered. Since we already had a system where each stream specifies a character (personality) and theme, we reused that data.
We made sure to avoid repeating the same topic. By referencing conversation history, we prevent immediate repetition. In long streams, the amount of history kept directly determines how often topics cycle back.
Features like this can’t be validated with short tests. Artificial silence doesn’t mimic real streaming behavior.
We ran continuous 2-hour streams to confirm:
Long-running tests reveal issues that short ones never will. The "topics cycling back" problem, for example, never appears in a 10-minute test.
Once we solved the silence problem, attention shifted to the other side: the slowness of responses to comments. Our initial experience felt like this:
Viewer types a comment →
25–45 seconds later, the avatar replies
As dialogue, this is painfully slow. At first, we wanted to "speed up LLM generation" or "lighten TTS," but before doing that, we measured the breakdown. The results changed our approach entirely.
| Segment | Time | Can We Reduce It? |
|---|---|---|
| Comment posted → Chat retrieved | ||
| Max 5 sec (polling interval) | ||
| Yes (but API limits constrain this) | ||
| Waiting in response queue | Varies | Partially |
| Response generation (LLM) | Several seconds | Yes |
| Speech synthesis (TTS) | Several seconds | Yes |
| Delivery → Viewers see it | ||
| 15–30 sec | ||
| Almost impossible to reduce |
The dominant factor was the last row. Viewers "hear" the reply 15–30 seconds after we send it. This is platform-level streaming delay, independent of our implementation. It’s the same viewer delay we used earlier to set the silence threshold.
In other words: Even if generation took 0 seconds, the perceived delay would still be ~20 seconds.
Still, we optimized where possible:
1. Enable low-latency mode
When creating a stream, enabling low-latency settings reduces viewer delay. This is the only way to affect the dominant factor, so we always use it.
2. Polling interval
Chat retrieval uses polling, so the interval directly adds to delay. Shorter intervals help, but we balance against API quota limits. We settled on 5 seconds.
3. Keep responses short
This affects not just generation time, but utterance duration. Long replies mean the avatar can’t move to the next comment quickly, increasing wait time for subsequent comments.
We instructed the system to respond in 2–3 sentences. For live dialogue, short exchanges feel more natural than long monologues. Constraints improved quality.
Even after optimization, ~20 seconds of delay remained. That’s when we shifted strategy:
If total time can’t be reduced, reduce theunresponsivetime.
Think of human streamers: when they read a comment and prepare a reply, they don’t stay silent. They hum, say "Ah, I see," or "Good question"—they keep making sound while thinking.
So we implemented immediate acknowledgments:
On comment detection:
→ Immediately play an acknowledgment phrase ("I see," "Good question," etc.)
※ Pre-synthesized audio, so no generation delay
→ In parallel, generate the full response
→ When ready, continue speaking seamlessly
Acknowledgments are fixed phrases, so we pre-synthesize and cache the audio. Generation delay becomes zero, so sound plays the moment a comment is detected.
The viewer experience changed like this:
| Before | After | |
|---|---|---|
| Comment → First reaction | 25–45 sec (silent) | |
| Immediate (ack) | ||
| Comment → Full response | 25–45 sec | 25–45 sec (unchanged) |
Total time didn’t change by a second. Yet the perception was completely different. "Being ignored" became "being heard and considered."
The "filler speech" we added for silence and this acknowledgment system are variations of the same idea: If you can’t eliminate wait time, don’t leave it silent.
The lessons from these two efforts apply beyond live streaming.
First: What should the system do when there’s no input?
Many conversational AIs assume "wait for input" is the default. That’s correct for chatbots (imagine one talking at you unprompted), but for roles like broadcasting, exhibits, reception, or monitoring—where being present is the job—silence is a specification gap. A simple rule like "if N seconds of silence pass, do X" is a minimal fix.
Second: Latency isn’t just about total time.
Users don’t experience total duration—they experience time spent with no response. These are different, and the latter can sometimes be reduced even when the former can’t. This is the same principle behind spinners or optimistic UI updates—it works in voice interaction too.
And both share two key lessons:
In our case, we almost wasted time optimizing LLM and TTS speed, only to find over half the delay came from platform-level viewer lag. Without measuring breakdowns first, we’d have burned cycles on speedups that barely improved perception.
An AI that stays silent looks broken. If your role is to be present, silence is a gap in the spec. And once you accept that "you can’t go faster," real UX design begins.
Making unmanned streams less boring isn’t about eliminating wait time—it’s about designing how to fill it.