TL;DR
I come from film and VFX. The point of this project was never "generate a video". It was "run a channel": something that is on the air whether or not the next shot is ready. That constraint changes everything about the pipeline, and this post is about those changes.
The fleet this channel needs is 24 RTX 5090s. Budget got me to 16 before I stopped. I will label what the maths requires, what actually ran, and what is extrapolated, because the gap between those three is where the engineering lives.
A real live stream has a camera. This one has a queue.
screenplay ──► shot queue (max 12) ──► claim_idle_box ──► box N: ComfyUI + LTX-2.3
│ │
one clip per box ▼ mp4 (3 s)
└──── completed[sequence] reorder buffer
│ strictly in order
▼
ffmpeg ──► HLS EVENT playlist ──► player
DISCONTINUITY per segment
buffer target 5 clips, ceiling 60 s
The channel keeps a buffer of finished clips ahead of the playhead. The target is 5 clips, the ceiling is 60 seconds of video. Stories are prefetched 4 shots ahead. When the buffer is healthy, the fleet idles. When it is not, the fleet is the only thing between you and a frozen player.
The HLS side is boring on purpose. The playlist is an EVENT type that only grows, every segment gets a DISCONTINUITY tag because every clip is a separate encode, and the playlist gets an ENDLIST only when the channel finishes. The player is told to hold a long buffer (120 seconds max) so a late clip is a hiccup, not a stall.
Each rented machine runs ComfyUI on loopback and a small stdlib HTTP server in front of it. The server owns a JobManager with a queue and N worker threads. Every call is Bearer-authenticated.
| Endpoint | What it does |
|---|---|
GET /health |
{ok, gpu, comfy, models, busy_slots, max_slots} .models is empty until ComfyUI listens. |
POST /generate |
Enqueue. A full queue returns 429 {"error":"busy"} . |
GET /generate/{id} |
Status. An unknown id returns 404 {"status":"unknown"} . |
GET /outputs/{name} |
Regex-guarded mp4 fetch, no path traversal. |
GET /logs |
Ring of the last 500 lines, for debugging a box you cannot SSH into fast enough. |
Task ids are box-local. A shot leases a box, and submit, poll and fetch all go to that same box. The lease is thread-local, so two worker threads can never confuse each other's box.
Job state is one JSON file per job, written to .part and then os.replace d. If the shim restarts, a job that was queued or running becomes succeeded if its mp4 exists and interrupted if it does not. There is exactly one function that writes job state. That rule sounds trivial until you have two writers and a status that flips back and forth.
The model is LTX-2.3, the 22B distilled transformer in FP8 with a Gemma-3-12B text encoder in FP4. Together they fit in 30 GB of system RAM, and the text encoder runs on CPU so the GPU holds only the transformer and VAE. The workflow uses 8 manual sigmas, euler_ancestral_cfg_pp, CFG 1.0, and native audio.
Things that cost me whole evenings:
aria2c -x16 -s16, and offers are filtered on inet_down >= 500./health lists the model. The go-live gate polls every 15 seconds with a 900-second cap.
Measured: deploy to live took a median of 8.95 minutes across 15 boxes, with a range of 3.7 to 53.7 minutes. Of 38 instances rented over the experiment, 11 never made it to live: 5 SSH timeouts, 3 scp or ssh failures, 2 pip or apt failures, 1 that never started.
Required fleet: 24 boxes (FLEET_MAX_BOXES=24). At 512p the stream needs 22 boxes for real time, plus spares for the ones that die. The plan was interruptible bids to keep the experiment cheap. Interruptible 5090s on Vast.ai list around $0.09 to $0.10 an hour. On-demand ones run $0.32 at the cheapest and about $0.63 at the median.
What happened: on 2026-09-24 every spot box under $1/hr was outbid within minutes of finishing provisioning. Provisioning takes 4 to 54 minutes. You cannot amortise a 9-minute boot on a box that lives for 3. The fleet code has been on-demand only since 2026-10-04.
The single-box path still has the bidding optimiser, and it is worth describing because it is the part people ask about:
min(min_bid x (1 + buffer), cap). The buffer starts at 0.13 and is clamped to 0.05 to 0.40.
The lesson from that code, written in its own docs: the biggest cost lever is not smarter bidding, it is not paying for idle hours.
The fleet path keeps the guard rails and drops the cleverness:
verified, reliability >= 0.98, inet_down >= 500, disk >= 100. boxes = ceil(1.5 x render_seconds / clip_seconds) + 1 spare. If production ratio drops below 1.0, the stream will stall and the code says so.destroy_fleet proves nothing remains listed before it returns.
Measured peak: 16 boxes rented in one batch. The run stopped there on money, not on code. The $60 API cap and the $20/hr fleet cap are both hard refusals I wrote myself, and I chose not to raise them. The scheduler, the reorder buffer and the replacement loop are all sized for 24 and tested at 24; the wallet was sized for 16.
Sixteen boxes finishing at different times would air shots out of order. The fix is a sequence number handed out under the pop lock, one clip in flight per box (claim_idle_box), and a completed[sequence] reorder buffer on the publisher. A failed shot leaves a hole at its sequence, the hole is skipped, and the stream continues. Order is a property of the publisher, not of the renderers.
Before this landed, one render thread per channel meant no concurrency at all, and the ledger had explicitly ruled an ordered publisher out because of it. The parallel scheduler and the ordered publisher shipped together on 2026-10-04.
Tests on the branch went from 410 to 669. Almost all of the new ones are about what happens when something lies.
The box reports queued, running, succeeded, failed, lost, interrupted. The studio maps these to SUBMITTED, SUCCESS, FAILURE, LOST. Both unknown and interrupted map to LOST. Anything else raises UnknownRenderStatus. Nothing falls through to a default.
LOST means "the box cannot vouch for this job". It does not mean the job failed. That distinction decides money.
429 busy is retried 8 times at 2 seconds, detected by HTTP code, not by string match. Ten busy-exhausted clips do not the channel. Three genuine failures do.UnknownBackend on a backend name it does not know. A typo like "syray" used to send paid work to the cloud backend while skipping the budget guard.
Two consecutive render failures retire a box. That number is deliberately below the channel's threshold of 3, so one bad box cannot take the channel down. An unreachable box is retired immediately. There is a documented lost-update race in mark_dead, and the worst case is one wasted attempt, so it stays.
I keep these in the ledger rather than fixing them because each one is bounded and the fix would touch money paths.
GPU_BOX_MAX_SLOTS defaults to 2 and is clamped to at least 1. The contract doc still says 1. Docs rot faster than code.poll. They are diagnostic only and can never fail a job.
| Number | Value |
|---|---|
| Warm i2v render, one 5090 | 66 to 70 s, median 67, n=7 |
| Cold first t2v clip | 128 s, n=1 |
| Clip | 3.04 s, 512x896, 24 fps |
| Render-to-video ratio | about 22 to 1 |
| Deploy to live | median 8.95 min, range 3.7 to 53.7, n=15 |
| Provision failures | 11 of 38 instances |
| Shot outcomes, final run | 8 SUCCESS, 11 FAILURE, 5 LOST, 1 SUBMITTED of 25 |
| Final stable box | 8 of 8 |
| Box-hours | 26.57 across 38 instances |
| GPU spend, upper bound | at most $20.82 if every box hit the $0.99 ceiling |
| Keyframe API spend | $60.55 against a $60 cap |
Two caveats on the 67-second figure. It comes from one stable box on the last two story packs, so the 17 shots before it are not timed. And the 5 LOST shots landed right after a batch deploy, which smells like spot reclamation, but the log has no outbid event type, so that is inference.
The honest summary of the final run: once a box was stable, it rendered 8 of 8. Getting a box to stable was the whole fight.
Hourly GPU cost is boxes times price per box-hour. Keyframes are $0.05 per i2v shot.
| Scenario | Boxes | $/box-hr | GPU $/hr | Output | Keyframe $/hr |
|---|---|---|---|---|---|
| Required 24, spot | 24 | 0.10 | 2.40 | 1,290 clips/hr | 64 (capped at 5) |
| Required 24, on-demand | 24 | 0.47 | 11.28 | 1,290 clips/hr | 64 (capped at 5) |
| Required 24 at bid ceiling | 20 | 0.99 | 19.80 | 1,075 clips/hr | 54 (capped at 5) |
| Reached 16, on-demand | 16 | 0.47 | 7.52 | 860 clips/hr, 0.73x real time | 43 (capped at 5) |
| Real-time, 3 s clips, measured | 22 | 0.47 | 10.34 | 1 hr of air per hr | 59 (capped at 5) |
| Real-time, 5 s clips | 14 | 0.47 | 6.58 | 1 hr of air per hr | 36 (capped at 5) |
Notes on that table:
So the full fleet is about $11 an hour in GPUs plus $5 an hour in stills, on-demand, with a stream that holds real-time at 512p and no slack. The spot version at $2.40 an hour was never achievable, because the boxes did not live long enough to boot. At 16 boxes the channel runs at 0.73x real time: the buffer drains, and the gauge tells you exactly when. That is the state I stopped in.
The experiment ended on budget, not on a bug. Everything a 24-box run needs is written, tested and capped. What is left is the list I would spend the next round on:
If you are building anything that has to be on the air whether or not the model cooperates, the render is the easy part. Write the status vocabulary first.
The box shim and the status mapping are the two files I would hand to someone starting the same thing. Ask me for them.
I build systems like this one for founders and studios: AI video pipelines, render fleets on rented GPUs, live streaming backends, and the cinematic web front ends that sit on top of them. If your product needs generated video that stays on the air, or you want this pipeline running under your own brand, talk to me.
Comments on this post reach me too.
Numbers in this post come from the branch's event log and SQLite state, not from memory. Where a figure is extrapolated, it says so.