📝 Originally published (in Japanese) at
[forge.workstyle.tech].
Streaming software makes it easy to go live on YouTube or Twitch, but when you try to build the system yourself, you suddenly find yourself lost in a fog of terminology: RTMP, HLS, WebRTC, SRT, ffmpeg. Which one is used where?
This article summarizes the challenges, causes, and solutions encountered while building a system where an AI avatar starts an unmanned live stream at a set time, responds to comments with voice, and automatically ends with a closing message when the time is up. All humans have to do is register the program in advance; no one opens the streaming screen on the day of the stream.
Here's what happens in sequence:
When the program's start time arrives
→ Create a broadcast using the API
→ Bind to the RTMP stream
→ Start the GPU Pod and send out the video
→ Transition to live
→ Closing message when the time is up
→ End the stream, destroy the Pod
→ Archive (VOD) remains
This process runs unmanned on both YouTube and Twitch. First, we'll outline the overall layout, then dive into specifics like simultaneous streaming, API automation, pitfalls in automatic start triggers, termination handling, and latency.
Note that platform protocols and API specifications change, so check each company's latest documentation before implementing. Here, we'll focus on the structural role division and the actual pitfalls encountered.
[1. Production] → [2. Ingest] → [3. Delivery]
Create video and audio Deliver to platform Deliver to viewers
OBS / Browser / Camera RTMP, etc. HLS, etc.
~Local You → Provider Provider → Viewers
These three segments use completely different technologies with distinct requirements. Much confusion arises from blurring the distinctions between these segments while discussing terms.
| Segment | Main Requirements | Commonly Used |
|---|---|---|
| 1. Production | Flexibility, Real-time | OBS, Browser, Camera, ffmpeg |
| 2. Ingest | Reliability, Platform compatibility | |
| RTMP (also SRT, WHIP, etc.) | ||
| 3. Delivery | Scalability, CDN distribution | |
| HLS variants |
You can only choose segments 1 and 2. Segment 3 is the platform's domain and cannot be controlled externally. This is why latency cannot be reduced beyond a certain point, as discussed later.
RTMP is a technology from the 2000s. It remains the standard for ingest because the recipients support it:
rtmp://
URLs and stream keys)Newer protocols like SRT and RIST offer better loss resilience, and WebRTC-based ingest (WHIP) is emerging. However, if the recipient doesn't support it, it's useless. In practice, the most mature and widely supported option is chosen, which is RTMP.
One critical point to note is that RTMP doesn't report failures. Even with an invalid stream key, the connection is accepted, and data is silently discarded. The sender cannot distinguish success from failure. Therefore, you must externally verify the receiver's status. This was a significant pitfall (related to the external status verification discussed later).
WebRTC is designed for sub-second bidirectional communication, ideal for video conferencing.
It's not the primary path for large-scale live streaming. Its peer-to-peer structure for each viewer doesn't scale well, and it doesn't integrate with CDN distribution mechanisms (while specialized services offer WebRTC-based low-latency streaming, it's not the main path for general platforms like YouTube or Twitch).
Does that mean it's unused? We used it internally:
[Server] AI voice generation
↓ WebRTC (low-latency, bidirectional)
[Browser] Avatar lip-syncs and plays audio
↓ Capture screen and audio
[ffmpeg] Encode
↓ RTMP
[Platform] → HLS → Viewers
WebRTC is used within production, while RTMP handles ingest. WebRTC's low latency is leveraged between the server and browser, and the output is switched to a scalable delivery mechanism.
Instead of choosing between WebRTC and RTMP, we used the right tool for each segment.
ffmpeg is often described as a "video conversion tool," but in streaming pipelines, it serves three simultaneous roles:
1. Encoding
Raw video and audio are compressed into deliverable formats like H.264/AAC. This is the most CPU-intensive task and was the bottleneck for simultaneous streams in our setup. Hardware encoders (NVENC, etc.) are used when possible.
2. Multiplexing (mux)
Video and audio are combined into a single stream. A/V sync is determined here. When video and audio arrive via separate paths, you must decide which timestamp to use as the reference.
3. Distribution (tee)
Encoded packets are duplicated to multiple outputs. Simultaneous streaming to YouTube and Twitch is achieved with just this feature. Since encoding happens only once, adding more destinations barely increases CPU load. This distribution is key to the next section on simultaneous streaming.
To stream an AI avatar to both YouTube and Twitch, a naive approach would be to run two renderers, each sending to a different platform. This doubles GPU usage and encoding load.
This is unnecessary. ffmpeg's tee
multiplexer duplicates encoded packets to multiple outputs:
Page (video + audio)
→ ffmpeg (encoding happens once)
→ tee ─┬→ rtmp://a.rtmp.youtube.com/live2/<key>
└→ rtmp://<ingest>.twitch.tv/app/<key>
The command looks like this:
ffmpeg <input specifications> \
-c:v libx264 -c:a aac <encoding settings> \
-f tee -map 0:v -map 1:a \
"[f=flv:onfail=ignore]rtmp://a.rtmp.youtube.com/live2/KEY1|[f=flv:onfail=ignore]rtmp://INGEST.twitch.tv/app/KEY2"
Two key points:
1. Encoding happens once
tee
duplicates the encoded stream, so the CPU-intensive H.264 encoding occurs only once. In our setup, the bottleneck was the CPU, not the GPU, so this optimization was effective. Adding more platforms barely increases CPU load.
Conversely, both platforms receive the same quality and bitrate. If you need different resolutions per platform, this method won't work. We used 720p30 uniformly, so it was fine.
2. onfail=ignore allows partial failure
This is crucial. By default, if one output fails, ffmpeg stops entirely. So, if the Twitch connection drops, the YouTube stream also stops.
With onfail=ignore
, failed outputs are disconnected, and the rest continue. If one platform has issues, viewers on the other are unaffected. In streaming, "partial failure" is always preferable to "complete failure," so this option is essential.
Even with identical video, peripheral handling differs:
| YouTube | Twitch | |
|---|---|---|
| Broadcast creation | API creates and explicitly transitions to live | No broadcast concept; live when ingest starts |
| Chat retrieval | Data API (API key suffices) | Anonymous IRC (no app registration needed) |
| Donations/subscriptions | Sent as chat messages | |
| EventSub (OAuth required) | ||
| Metadata setting | Set during broadcast creation | Helix API (OAuth required) |
While video ingest can be unified, peripheral control requires platform-specific code. We normalized events into a common internal format, routing them to a single response logic layer. This isolates platform differences in the ingest and event collection layers, making the response logic platform-agnostic.
Twitch lacks a broadcast concept, so the avatar must be connected for the stream to go live. Unlike YouTube, where a "created but no video" state exists, Twitch failures manifest as "nothing happens." Subsequent broadcast-related discussions focus on YouTube's asymmetric design.
YouTube requires controlling "broadcast creation and live transition." First, clarify credential boundaries.
Misunderstanding this wastes time. YouTube Data API v3 operations require different credentials:
| Action | Required |
|---|---|
| Fetch public data, subscribe to live chat | |
| API key suffices | |
| Create/bind/transition/end broadcasts | |
| OAuth (channel owner authorization) | |
| Check post-stream archive status | |
| OAuth (API key returns 403) |
Initially, we only implemented viewer comment reading, so an API key was enough. Adding automatic broadcast creation required OAuth, a significant hurdle. Practically, any write operation to a channel requires OAuth.
OAuth involves obtaining a refresh token once and using it to update access tokens. This requires one-time human interaction, the only exception to "no human intervention."
Use liveBroadcasts.insert
to create a broadcast. Settings here define the stream's nature:
latencyPreference
)RTMP streams are reused, with IDs resolved via API. This avoids embedding stream keys in settings, simplifying key rotation.
This is more about responsibility than implementation.
YouTube requires flagging synthetic media (AI-generated content that could be mistaken for real). This flag is set during broadcast creation. Since our stream uses AI for voice and responses, we enable this flag.
In automated systems, such disclosures might seem optional. However, automated systems should adhere to the same checks as manual ones. Platforms don't waive responsibilities for automation. As more flags emerge, grouping "disclosure items" near broadcast creation code simplifies additions.
Creating a broadcast, sending video via RTMP, and enabling enableAutoStart
should automatically start the stream when video arrives—or so we thought.
It didn't. Video arrived (active
stream status), the broadcast existed, but the status remained ready
.
The cause:
enableAutoStart
doesn't trigger for broadcasts bound to alreadyactive
streams.
Our setup reused streams (RTMP endpoints and keys), creating broadcasts per program:
1. Create broadcast with `liveBroadcasts.insert`
2. Bind to existing stream with `liveBroadcasts.bind`
3. Start renderer Pod and begin RTMP ingest
If the stream was already active
(from previous programs or tests), binding in step 2 means YouTube sees an "already streaming" stream with a new broadcast attached.
Auto-start likely triggers on the transition from inactive
to active
. Binding to an already active stream doesn't trigger this transition, so it fails.
Viewer URLs kept changing, requiring constant resharing. Initially, we tweaked insert
parameters, but the issue was event ordering.
We abandoned auto-start:
enableAutoStart = False
monitorStream = Disabled
Instead, the scheduler explicitly calls liveBroadcasts.transition
:
On stream `active` confirmation
→ Call `transition(broadcastStatus='live')`
→ Retry on failure
Retries are needed because there's a lag between stream activation and YouTube readiness. The first attempt often fails, but a retry after a short wait succeeds.
Explicit state-based actions are more reliable than event-driven ones. The former relies on observable states, while the latter depends on internal platform states. "Automagic" features often lack documented triggers, making explicit control vs. automation a reliability design choice.
403 errors on bind.
liveBroadcasts.bind
sometimes returned 403 errors, not permanent permission issues but
Retry after 20s → 40s → 60s (max 4 attempts)
In this domain, 403 doesn't always mean "permission denied." Treating temporary failures as permanent ones leads to unnecessary abandonment.
Silent streams are treated as audio-less. During failure recovery, restored state snapshots sometimes skipped greetings, resulting in silent starts. YouTube treats these as audio-less streams, undesirable for live broadcasts. Platform health must be considered, not just application logic, when deciding whether to output audio.
Automated streaming often overlooks termination. Starting is motivated, but ending is not—and unterminated streams incur ongoing charges.
We implemented:
Honestly, archive (VOD) status checks post-stream still return 403 with API keys, leaving OAuth implementation or warning-based operation as pending decisions. Even read operations require OAuth if accessing non-public channel data.
Interactive streams are sensitive to viewer latency. Here's the breakdown:
| Segment | Time | Reducible by Us? |
|---|---|---|
| Production (generation/encoding) | A few seconds | Reducible |
| Ingest (RTMP) | <1 second | Almost irrelevant |
| Delivery (to viewers) | ||
| 15–30 seconds | ||
| Almost unreducible |
The dominant factor is segment 3, the platform's domain. HLS delivery involves creating segments (multi-second chunks) for CDN distribution, inherently introducing multi-segment latency. This latency is traded for scalability.
Low-latency modes (latencyPreference
) reduce this but don't eliminate it. Viewer latency is beyond our control, so optimize all reducible settings and design UX assuming 15–30 second latency.
We shifted focus from "reducing latency" to minimizing unresponsive periods. Instantly respond to comments with pre-synthesized acknowledgments. Total time remains unchanged, but the experience is vastly different. Design UX assuming unreducible viewer latency.
Implementation alone isn't enough; we tested with actual programs:
One issue was external status verification. Twitch's public live status endpoint is cached, returning offline
immediately after stream start. This led to false failure detections.
Poll status until it stabilizes for start and end. Single checks are unreliable. Since RTMP doesn't report failures, repeatedly verify receiver status externally.
Design choices effective for unmanned streaming:
tee
.onfail=ignore
is essential). All outputs share the same qualityenableAutoStart
fails for already active streams. Explicitly call transition
bind
403s with staged delays (20→40→60s, max 4 attempts)Confusion arises from mixing segment-specific terms. Separating segments clarifies choices. The hardest parts of unmanned streaming are ensuring termination and avoiding dependence on platform internal states. Unstarted streams are noticed immediately; unterminated ones and silently dropped ingest are noticed in bills and misdiagnoses.