📝 Originally published (in Japanese) at
[forge.workstyle.tech].
When building an AI avatar live streaming system that runs unattended, the development focus shifts from a typical application. If the system crashes during unmonitored hours, no one is there to fix it. This means that instead of just "working well," the key to quality is making sure the system "understands when it breaks and can recover on its own."
This article summarizes the pitfalls encountered while getting such a system to run continuously on Twitch, broken down into three layers:
ffmpeg
is sending video, the channel doesn’t go live. Twitch silently accepts invalid keys.All of these issues share a common pattern: "Everything looks normal from the sender’s side." This makes them especially tricky. Let’s go through them one by one.
First, to just read chat, an anonymous IRC connection is sufficient—no app registration required. That part was straightforward.
OAuth became necessary when we wanted to do two things:
Only these two operations require authorization. The overall flow looks like this:
1. Register an app in the Developer Console (Confidential type)
2. Obtain client_id / client_secret
3. Open the authorization URL in a browser and authorize as the channel owner
4. Extract the authorization code (code) from the redirect URL
5. Exchange the code for an access token and refresh token via the token endpoint
6. Save the refresh token on the server
Only steps 3 and 4 require human interaction; the rest can be automated. Below are the specific pain points we encountered during this process.
When registering the app, you must select a type. If your server holds the client_secret
and performs token exchange, choose Confidential. Choosing the wrong type later means the secret can’t be used for token exchange, and you’ll have to recreate the app.
The redirect URI is only needed to complete authorization, so something like http://localhost:3000
is fine. You don’t even need to run a server to receive it—just copy the code from the address bar after authorization.
If scopes are missing, you’ll have to restart the authorization process. The three scopes we needed were:
| Scope | Purpose |
|---|---|
bits:read |
|
| Subscribe to bits (cheers) events | |
channel:read:subscriptions |
|
| Subscribe to subscription events | |
channel:manage:broadcast |
|
| Set stream metadata like the title |
Adding scopes later requires the user to open a browser again. List all required functionality upfront and request all necessary scopes in one authorization.
force_verify
to Avoid Authorizing the Wrong Account
We added force_verify=true
to the authorization URL.
Without it, if you’re already logged into the browser, authorization might complete without a confirmation screen, especially if you have both a personal account and a character account. This can lead to accidentally authorizing the wrong account. With force_verify=true
, a confirmation screen always appears, making it clear which account you’re authorizing.
This was the most nerve-wracking issue.
Authorization codes ( code) expire in just a few minutes. If the flow involves human steps—opening a browser, copying the code, passing it to the server, and exchanging it—the code may already be dead by the time you receive it.
The fix is simple: prepare the token exchange process in advance and execute it the moment the code arrives. We pre-assembled the exchange command and waited. As soon as the code came in, we executed it immediately, and it worked on the first try.
Once you get the token, hit the validation endpoint (/oauth2/validate
) to inspect its contents.
curl -H "Authorization: OAuth <access_token>" https://id.twitch.tv/oauth2/validate
Check the returned login
(username) and the list of scopes. Are you authorized as the expected channel owner? Are all required scopes present?
Skipping this step makes it impossible to distinguish between "insufficient scopes," "wrong account," and "implementation error" when EventSub subscriptions fail. Immediately after obtaining credentials, verify what they represent.
This issue became apparent only after implementation and is worth sharing.
Twitch may replace the refresh token itself when updating tokens.
A naive implementation looks like this:
If the old token is still valid, it works. But if it’s invalidated, authentication fails. Worse, the failure doesn’t appear until the process restarts, making the root cause hard to diagnose.
The correct approach is to persist the new refresh token every time it’s updated. This issue appears in other services too, so it’s worth confirming whether you’re assuming “refresh tokens are immutable.”
A quick operational note. Avoid pasting client secrets or refresh tokens into chats or tickets. We used a file-based approach, writing to a secure location and deleting it afterward.
touch ~/.twitch-cred && chmod 600 ~/.twitch-cred
shred -u ~/.twitch-cred
In a previous project, a stream key accidentally ended up in logs, forcing us to reset it. Plan ahead for where secrets end up.
Once authentication is sorted, the next step is sending video to Twitch. Here we hit a situation where everything looked normal on the sender side, yet the channel never went live.
ffmpeg
was running and sending frames continuouslyWhere to even start debugging? The root cause boiled down to two issues:
-
Twitch’s RTMP ingest accepts invalid stream keys and silently discards the data. From the sender’s side, success is indistinguishable.
-
If the stream itself is malformed, Twitch won’t mark the channel as live. Even the dashboard’s Stream Inspector shows nothing.
A common misconception with RTMP is that “if the connection is established and maintained, the key is valid.” That’s not true. The discrepancy between what we saw and Twitch’s state was:
| What We Saw | State |
|---|---|
| ffmpeg logs | Normal. Frames being sent |
| TCP connection | Established & maintained |
| Twitch channel | Still offline |
In our case, the cause was account-side settings (2FA and stream key status). Once fixed, running the exact same command made the channel go live. We hadn’t changed anything on the sender side.
The operational rule we derived is simple:
Determine streaming success not by sender-side logs, but by receiver-side state.
Another issue we encountered. At the time, we were generating video via CPU rendering. When we checked the recording, only 6 seconds of video were saved out of 90 seconds (due to an audio track exhaustion bug).
Sending this timeline-dropped stream to Twitch resulted in a connection being established, but the channel never going live. The Stream Inspector showed no information. From Twitch’s perspective, it received a stream with timestamps far behind real time, making it impossible to treat as a valid broadcast.
Again, from the sender’s side, it “looked fine.” Just because ffmpeg is running doesn’t mean a valid stream is being sent.
We incorporated a mechanism to fetch Twitch’s state externally as part of our validation process. We used a public endpoint that returns the channel’s uptime.
curl -s "https://decapi.me/twitch/uptime/<channel_name>"
This allowed us to mechanically determine whether the channel was actually live. We changed our acceptance criteria for stream tests from sender-side logs to this output.
However, there’s a trap. Third-party APIs like this often cache responses. If you check immediately after starting a stream, you might still get offline
, leading to a false negative and wasted debugging effort.
for i in $(seq 1 20); do
curl -s "https://decapi.me/twitch/uptime/<channel_name>"
echo
sleep 15
done
The same applies to stopping. After stopping the stream, offline
isn’t immediate. For both start and stop, poll until the state stabilizes.
This isn’t specific to Twitch. The same pattern appears in many places:
In all cases, there’s a gap between “sender-side success” and “receiver-side state.” Relying only on one side leads to silently broken states. After this incident, our streaming validation checklist always includes a line: “External confirmation of receiver state.” Silence is not a synonym for success.
Once streaming works, the next challenge is surviving unattended operation. How the system behaves when it crashes determines almost the entire quality. We identified three failure paths, each with different detection and recovery methods—and the third was the most dangerous.
| # | Failure Mode | Symptoms | Recovery |
|---|---|---|---|
| 1 | GPU host instability | Renderer recovers every few dozen seconds to minutes | Restart on a different host |
| 2 | Pod disappears | Video feed completely drops | Restart on a different host |
| 3 | Only the interaction server dies | ||
| Video continues, but avatar stays silent | |||
| 3-stage chain recovery |
Paths 1 and 2 were handled together. Instead of trying to fix the broken instance, we discard it and restart.
“Killing yourself” feels counterintuitive, but it’s the most reliable way to propagate failure upward. We verified this by manually deleting a Pod mid-stream. Detection happened in 32 seconds, and streaming resumed on a different host.
This was the real problem. When only the interaction server (the speech generation side) restarts, the renderer knows nothing. Video continues flowing. The page stays alive. The avatar is on screen, blinking. It just stops talking.
From the viewer’s perspective: “The stream continues, but the AI has gone silent.” From a monitoring standpoint, this is catastrophic:
ffmpeg
are alive)None of the health checks catch this. Even our external monitoring doesn’t detect it—the channel is still marked as live.
The root cause was that death wasn’t propagated through the layers. The architecture was multi-tiered:
Interaction Server ←WebSocket→ Audio Pipeline ←WebRTC→ Page
When the WebSocket to the interaction server died, the audio pipeline didn’t forward the disconnection downstream. The WebRTC connection remained alive, so from the page’s perspective, it was “connected but receiving nothing.” The disconnection was absorbed mid-path and never reached the end. That was the true issue.
We made death propagate reliably to the end:
Step 3 was subtle but critical. Without it, if the reload happened while the server was still restarting, the connection would fail and hang. Recovery logic must account for recovery timing.
We performed a rolling restart of the interaction server and measured recovery time.
Server restart
→ Detect WebSocket death
→ Actively close WebRTC
→ Page reload
→ Reconnect (retry if server still starting)
→ Restore state from snapshot
→ Resume speech
Total: 41 seconds
Including state restoration, it took 41 seconds to resume seamlessly from where it left off, without re-greeting the audience.
1. Design for “silent continuation” as the worst-case scenario. A process dying is detectable and relatively safe. The danger lies in appearing alive while not functioning. In our case: video streaming but no speech. Adding conditions like “if I detect I’m not functioning, terminate honestly” across layers improved overall stability. Trying to survive at the cost of availability actually reduces availability.
2. Disconnections must propagate to the end. In multi-tier systems, intermediate layers can absorb upstream deaths. Each connection must define: “When the upstream dies, what does this layer do?” The default behavior is often “do nothing.”
3. Recovery logic must work even during recovery. Code like “reconnect because the server restarted” often fails to account for the server still being down. Retries are part of recovery logic.
4. Fault-tolerant code without fault injection is likely broken. We manually crashed all three paths: deleted Pods, restarted servers. Just writing recovery code rarely works end-to-end. The need for “initial connection retry” only became clear after actual failure injection.
Across all three layers, one principle held true:
“Information visible from the sender side does not guarantee the receiver’s state.”
force_verify=true
to prevent accidental authorization with the wrong account./oauth2/validate
to confirm authorization scope and account.OAuth procedures and RTMP specs are documented, but the real pitfalls lie in:
For unattended operation, aiming for a system that "understands when it breaks and recovers automatically" is far more practical than trying to build one that never breaks.