{"slug": "the-14-open-source-repos-powering-code-driven-video-in-2026-how-we-built-3d-aj", "title": "The 14 Open-Source Repos Powering Code-Driven Video in 2026: How We Built 3D AJ the Cat", "summary": "CodingCat.dev wired a 14-repository open-source stack into its production site and a local RTX 5090 video studio to build a 3D Vinyl-Toy AJ the Cat mascot, using Three.js, Blender and Blender MCP (ahujasid/mcp-for-blender) for the aj-3d-mascot.glb model. The runtime (AjAnimatedMascot.astro and Aj3dMascotViewer.astro) runs a shared deterministic window.seek(t) motion engine in src/lib/aj-motion-engine.ts with closed-form harmonic springs and seeded noise, and the site added 120 BPM audio-reactive lip-sync to its podcasts and automated a 1080p Course Factory with faster-whisper, FFmpeg and headless Chrome DevTools MCP. The build rests on the argument, drawn from a viral breakdown by @0xnicc0 and @seeconvm, that foundation models cannot output an MP4 directly and that code-rendered video requires a strict frame contract plus an FFmpeg contact-sheet critique loop.", "body_md": "[AI & MCP Agents](https://codingcat.dev/topic/ai)\n\n# The 14 Open-Source Repos Powering Code-Driven Video in 2026: How We Built 3D AJ the Cat\n\nEvery AI engineering timeline this month is flooded with launch films, kinetic typography reels, and 3D product videos captioned `\"one prompt in Opus 5.5\"`. If you actually believed a 20-word vibe prompt inside a standard chat window produced those 60fps showreels, you probably tried it yourself and got a centered title over a purple gradient with drifting CSS animations.\n\nA recent viral breakdown by `@0xnicc0` and `@seeconvm` pulled back the curtain on what those viral videos are actually hiding: **foundation models cannot output an MP4 file directly**. Every viral code-rendered film is a deterministic program written by an agent with shell access, driven by a strict `window.seek(t)` frame contract, closed-form spring physics, beat-locked audio (`beats.json`), and a contact-sheet critique loop powered by **14 free open-source repositories**.\n\nOn **CodingCat.dev**, we took that exact 14-repo stack and wired it directly into our production site and local RTX 5090 video studio:\n\n1. We built our canonical **3D Vinyl-Toy AJ the Cat GLB model** (`aj-3d-mascot.glb` ) using**Three.js** ,**Blender** , and**Blender MCP** (`ahujasid/mcp-for-blender` ), complete with a live**2D Vector vs. 3D GLB toggle** on[`/mascot-lab`](https://codingcat.dev/mascot-lab) and our homepage Hero.\n2. We upgraded AJ's runtime (`AjAnimatedMascot.astro` and`Aj3dMascotViewer.astro` ) with a shared**deterministic `window.seek(t)` motion engine** (`src/lib/aj-motion-engine.ts` ) using**closed-form harmonic springs** and**seeded noise** .\n3. We wired **120 BPM audio-reactive lip-sync** into our**Podcasts** (`/podcasts` and`/podcast/[slug]` ) and automated our 1080p**Course Factory** with**faster-whisper** ,**FFmpeg** , and headless**Chrome DevTools MCP** .\n\nLet's break down the mental model behind deterministic code-driven video, see the Architecture Diagrams, rank all 14 repositories in an **Alex Patterson Architectural Tier List**, and walk through the exact TypeScript and Python code powering 3D AJ.\n\n## 1. The Mental Model: Why window.seek(t) Beats Non-Deterministic Timers\n\nWhen you ask an LLM to write a web animation without a render contract, it defaults to `requestAnimationFrame`, `setTimeout`, CSS transitions, and `Math.random()`. That works fine for a casual decorative hover effect, but it falls apart the moment a headless browser tries to capture 900 frames for a 15-second 60fps video:\n\n- If frame 240 takes 45ms to paint instead of 16.6ms, timer-based animations drift out of sync with the audio track.\n- If a headline lands 10 frames too early at second `4.0` , you cannot jump straight to frame 240 to inspect it without re-simulating frames`0` through`239` .\n- If particle positions use `Math.random()` , rendering frame 300 twice produces two different image hashes, breaking loop continuity and automated visual diffs.\n\nThe solution is a strict **Frame Function Contract** enforced via your agent rules (`CLAUDE.md` or `GEMINI.md`):\n\n1. **`window.seek(t)` Paints Frame `t`** : Nothing else drives the screen. Passing` window.seek(4.0)` renders the exact visual state at`4.000` seconds deterministically.\n2. **Closed-Form Springs (`spring(t - t_i)`)** : Instead of accumulating velocity state frame-by-frame in a physics loop, every movement is evaluated as an analytical function of elapsed time since impulse`t_i` .\n3. **Seeded Hash Noise Only** : Replace`Math.random()` with a deterministic 32-bit integer hash (`seededNoise1D(seed, t)` ) so frame`300` hashes identically on every run.\n4. **The FFmpeg Contact-Sheet Critic** : Before the agent shows you an MP4, it uses**FFmpeg** to extract a 2-fps contact sheet and a 12-frame strip around the fastest motion, inspects its own rendered frames via**Chrome DevTools MCP** , scores them from 1 to 10, and fixes the worst 3 defects automatically.\n\n| Architectural Piece | What Goes Wrong Without It | What Fixes It in Production | \n|---|---|---|\n| Frame Function | Renders drift under GPU load; every edit requires replaying from t = 0. | window.seek(t) + pure analytical functions of time + zero CSS timers. | \n| Closed-Form Springs | Fixed linear/cubic-bezier easing feels robotic; iterative physics breaks random frame access. | track(t) = v0 + Σ (v_i - v_(i-1)) * spring(t - t_i) with analytical damping. | \n| The Agent Rig | Chat-only models never see or hear their own broken layouts or overlapping text. | Agent shell + Headless Chromium / Chrome DevTools MCP + FFmpeg. | \n| Audio & Beat Clock | Voiceovers and music float on top of visuals with off-beat state transitions. | Chatterbox / F5-TTS + faster-whisper word timestamps + librosa beats.json. | \n| The Contact-Sheet Critic | The first render ships with clipped text, dead beats, or loop stutters. | 6-across FFmpeg contact sheet + 360px mobile readability test + automated self-critique. | \n\n## 2. Alex Patterson's Architectural Tier List: All 14 Code-Driven Video Repos Ranked\n\nInstead of treating all 14 repositories from `@0xnicc0`'s list as interchangeable, here is how they rank in our production **CodingCat.dev** stack based on determinism, GPU efficiency, MCP agent ergonomics, and how heavily we rely on them to build **3D AJ** and our **Courses**:\n\n### Alex's 2026 Code-Driven Video & 3D Web Tier List (All 14 Repos Ranked)\n\nRanked from S-Tier to C-Tier across deterministic window.seek(t) support, GPU performance, MCP agent integration, and how we use them for 3D AJ.\n\n#### Three.js (mrdoob/three.js)\n\nBrowser 3D & WebGPU\nThe undisputed king of browser 3D. Zero plugin install for users, native glTF 2.0 .glb support, and instant seek(t) scrubbing.\n\n#### FFmpeg (FFmpeg/FFmpeg)\n\nMedia Compositor & Critic\nNon-negotiable infrastructure. Every frame pipeline, audio normalizer, and contact-sheet critic ends in FFmpeg.\n\n#### faster-whisper (SYSTRAN/faster-whisper)\n\nWord-Level Audio Alignment\n4x faster than stock Whisper with lower VRAM via CTranslate2. Gives agents an exact millisecond clock for every spoken word.\n\n#### Blender MCP (ahujasid/mcp-for-blender)\n\nAgentic 3D Scene Control\nTurns Blender from a manual GUI into an agent-programmable 3D compiler. Essential for automated mesh inspection and rigging.\n\n#### Blender (blender/blender)\n\n3D Modeling, Rigging & Baking\nThe open-source 3D standard. Combined with Python bpy, it eliminates manual 3D artist bottlenecks.\n\n#### Remotion (remotion-dev/remotion)\n\nReact Video Composition\nBest-in-class DX when your team already builds in React and Tailwind CSS. Deterministic frame rendering out of the box.\n\n#### Remotion Skills (remotion-dev/skills)\n\nAgent Procedural Memory\nCrucial context upgrade. Without this skill, LLMs write brittle React effects instead of pure useCurrentFrame() math.\n\n#### Chrome DevTools MCP (ChromeDevTools/chrome-devtools-mcp)\n\nLive DOM & Frame Inspection\nCloses the blind-agent gap. Your agent sees the exact DOM, computed styles, and pixel output in real time.\n\n#### GSAP (greensock/GSAP)\n\nTimeline & UI Motion\nBattle-tested timeline control. Just remember to pause auto-ticking and drive tl.time(t) from window.seek(t) in render mode.\n\n#### HyperFrames (heygen-com/hyperframes)\n\nAgent-Native HTML-to-Video\nPurpose-built for coding agents that prefer writing raw HTML/Canvas over full React bundles. Fast and lightweight.\n\n#### Chatterbox (resemble-ai/chatterbox)\n\nOpen-Source Voice Synthesis\nImpressive open-source expressive voice cloning with emotion control. Great local alternative to paid ElevenLabs seats.\n\n#### ComfyUI (Comfy-Org/ComfyUI)\n\nNode-Based Diffusion & 3D Gen\nUnmatched local GPU control for image-to-3D plates and texture maps, though JSON workflow graphs require disciplined versioning.\n\n#### Impeccable (pbakaus/impeccable)\n\nAutomated UI & Design Critic\nGreat automated design gate to stop agents from shipping generic purple-gradient SaaS clichés.\n\n#### yt-dlp (yt-dlp/yt-dlp)\n\nReference Media Ingestion\nIndispensable utility for reference grammar extraction, though strictly an ingestion CLI rather than a rendering engine.\n\n## 3. Deep Dive: How We Built and Upgraded 3D AJ the Cat Across CodingCat.dev\n\nBefore this upgrade, AJ the Cat lived in two separate worlds on **CodingCat.dev**:\n\n1. **Static 3D Turnaround Renders** : High-resolution Pixar-style vinyl-toy studio JPEGs (`assets/AJ-3D-Headphones-Front.jpg` ,`Left.jpg` ,`Right.jpg` ) used in YouTube thumbnails and video bumpers.\n2. **2D Articulated Vector SVG (`AjAnimatedMascot.astro`)** : A 57-path inline SVG rig with 14 bone pivot groups (`#aj-head-group` ,`#aj-arm-right` ,`#aj-tail` , etc.) driven by CSS`@keyframes` and random`setTimeout` blinks.\n\nWhile the 2D SVG rig had zero external JavaScript dependencies, its CSS `@keyframes` and `Math.random()` blink timers meant **it was not frame-seekable** for headless video rendering, and visitors could not orbit or inspect a real 3D mesh in the browser.\n\nHere is the exact 4-stage architecture we implemented to unify **2D Vector AJ** and **3D GLB AJ** under a single deterministic motion engine.\n\n### Stage 1: Programmatic 3D Vinyl-Toy GLB Generation (scripts/build-aj-3d-glb.mjs) & Blender MCP Verification\n\nTo ensure 3D AJ matches every single anatomical law in our Design System (`09-assets.md`) while keeping the binary payload compact (` 756 KB`) and 100% deterministic, we built `scripts/build-aj-3d-glb.mjs` and paired it with our headless **Blender MCP** verification script (`scripts/blender-aj-3d-pipeline.py`).\n\nEvery anatomical feature of AJ is mapped to a named glTF 2.0 node so both **Blender** (`bpy.data.objects`) and **Three.js** (` gltf.scene.getObjectByName`) can target the exact same bones:\n\n- `AjRoot` ->`AjPedestal` +`AjBodyRoot`\n- `AjTorso` (Matte-black`#0B0B0E` PBR fur with roughness`0.62` + dark charcoal`#2D2D30` oval belly patch)\n- `AjCollar` +`AjHexTag` (Soft Lavender`#B4A7D6` collar with hexagonal gold & Royal Purple`\"AJ\"` tag)\n- `AjHeadGroup` ->`AjEarLeft` ,`AjEarRight` ,`AjWhiskersLeft` ,`AjWhiskersRight`\n- `AjEyeLeftPupil` &`AjEyeRightPupil` (Electric Lime Green`#C5E86C` eyes with vertical black pupils and dual white catchlights)\n- `AjJawMouth` (Articulated jaw and pink tongue viseme group for 120 BPM audio-reactive lip-sync)\n- `AjHeadphonesGroup` (`CodingCat.dev Podcast` charcoal headband + Royal Purple`#7C3AED` & Soft Lavender`#B4A7D6` earcups fitting directly over his cat ears)\n\n### Stage 2: Closed-Form Springs & Deterministic window.seek(t) (src/lib/aj-motion-engine.ts)\n\nIn `@0xnicc0` and `@seeconvm`'s guide, the secret to lifelike motion that remains 100% seekable at any arbitrary frame `t` is **stacking closed-form harmonic springs**:\n\nBecause `closedFormSpring(dt, omega, zeta)` is a pure analytical equation using `Math.exp(-zeta * omega * dt)` and `Math.cos(omegaD * dt)`, evaluating frame `812` (`t = 13.533s`) requires **zero knowledge of frames `0` through `811`**.\n\nHere is the production motion engine we shipped in `apps/site/src/lib/aj-motion-engine.ts`:\n\nBoth `<AjAnimatedMascot />` (2D SVG) and `<Aj3dMascotViewer />` (3D Three.js GLB) subscribe to `window.seek(t)` (`cc:seek` custom event) and call `computeAjPoseAtTime(t, state)`. You can test this live right now on [`/mascot-lab`](https://codingcat.dev/mascot-lab) using the **` window.seek(t)` frame scrubber** or by opening DevTools and running `window.seek(4.25)`.\n\n### Stage 3: Zero-LCP-Penalty 3D Three.js Viewer (Aj3dMascotViewer.astro)\n\nA common mistake when adding **Three.js** to a content site is mounting a heavy WebGL canvas above the fold that blocks Largest Contentful Paint (LCP).\n\nOn **CodingCat.dev**, we kept our zero-JS 57-path 2D SVG rig as the instant-paint SSR default and added an interactive **2D SVG / 3D GLB toggle** in the Homepage Hero (`AnimatedHero.astro`) and on [`/mascot-lab`](https://codingcat.dev/mascot-lab). When you switch to `3D GLB`, `Aj3dMascotViewer.astro` loads `/models/aj-3d-mascot.glb` (`756 KB`), lights AJ with ACES Filmic tone mapping plus Royal Purple (`#7C3AED`) and Electric Lime (`#C5E86C`) spot rim lights, and binds cursor coordinates to `AjEyeLeftPupil` and `AjEyeRightPupil`.\n\n### Stage 4: 120 BPM Audio-Reactive Lip-Sync on Podcast Pages & Video Studio\n\nIn Section 6 of the Opus 5.5 Motion Design guide (*\"Springs and Sound, or Why Cheap Motion Feels Cheap\"*), sound and picture must share the exact same clock. We wired `PodcastListen.astro` and `/podcasts` to `window.setAjAudioAmplitude(amp, bpm)`:\n\n- Whenever you play an episode in `PodcastListen.astro` (or click**Test AJ 120 BPM Beat Sync** on`/podcasts` ), AJ's jaw viseme (`#aj-talking-frame` in 2D SVG and`AjJawMouth` in 3D GLB), head nod, and ear bounce lock to the audio envelope at 120 BPM.\n- In our local RTX 5090 `tools/video-studio/synthesize-voiceover.py` pipeline,**faster-whisper** extracts word-level timestamps into`storyboard.json` so every rendered Course Lesson video aligns captions, diagram transitions, and AJ reactions to the millisecond.\n\n**AJ Pro-Tip Callout: Never Use `Math.random()` or Unpaused Timelines in Headless Video Renders** When capturing frames with Playwright or Headless Chrome, even a single `Math.random()` inside a blink timer or particle emitter will cause a visual stutter at your video's loop point. Always derive blinks and noise from `hashUint32(seed + Math.floor(t * 60))` and drive every animation from `window.seek(t)`.\n\n## 4. Full Course Roadmap: 5 Flagship Stack Courses + 14 Deep-Dive Tool Courses\n\nTo help you master this entire code-driven media and 3D web stack without paying `$1,200` per launch video, we have prepped **19 new Courses** (5 Flagship Architectural Stack Courses + 14 dedicated Tool-by-Tool Courses, totaling **95 Lessons**) in the **CodingCat.dev** Course Factory (`/courses`):\n\n### The 5 Flagship Stack Courses\n\n1. **Flagship 1: Programmatic Video with React, Remotion & Agent Skills** (`Remotion` +`Remotion Skills` )\n2. **Flagship 2: Agentic 3D Web Pipelines with Blender, Blender MCP & Three.js** (`Blender` +`Blender MCP` +`Three.js` )\n3. **Flagship 3: Deterministic HTML Motion & Visual Critique with HyperFrames, GSAP, Impeccable & Chrome DevTools MCP** (`HyperFrames` +`GSAP` +`Impeccable` +`Chrome DevTools MCP` )\n4. **Flagship 4: Local GPU Audio & Diffusion Pipelines with Chatterbox, faster-whisper & ComfyUI** (`Chatterbox` +`faster-whisper` +`ComfyUI` )\n5. **Flagship 5: Automated Media Engineering & Reference Extraction with FFmpeg & yt-dlp** (`FFmpeg` +`yt-dlp` )\n\n### The 14 Dedicated Tool-by-Tool Courses (1 Per GitHub Repo)\n\n| Course # | Course Title & Slug (/course/[slug]) | Primary GitHub Repo | Core Production Deliverable | \n|---|---|---|---|\n| 01 | Remotion in Production: React Video Composition (/course/remotion-react-video-production) | remotion-dev/remotion | Multi-aspect-ratio 60fps launch video rendered from React + Tailwind CSS. | \n| 02 | Remotion Skills for AI Coding Agents (/course/remotion-skills-ai-agents) | remotion-dev/skills | Custom SKILL.md harness that teaches Claude Code & Antigravity deterministic video rules. | \n| 03 | Blender 5.0 for Web Developers: Headless Python & GLB (/course/blender-headless-python-web-devs) | blender/blender | Automated bpy script that builds, lights, and exports a <1MB .glb mascot. | \n| 04 | Blender MCP: Driving 3D Scenes with AI Agents (/course/blender-mcp-agentic-3d-scenes) | ahujasid/mcp-for-blender | Live TCP socket MCP server letting an agent model, rig, and render 3D AJ. | \n| 05 | Three.js & WebGPU in Production: Interactive 3D Mascots (/course/threejs-webgpu-3d-mascots) | mrdoob/three.js | Lazy-loaded <Aj3dMascotViewer /> with orbital camera, rim lights, and seek(t). | \n| 06 | GSAP Deterministic Timelines & SVG Morphing (/course/gsap-deterministic-timelines-svg) | greensock/GSAP | Frame-locked tl.time(t) UI morphs and 57-path SVG bone animations. | \n| 07 | HyperFrames: Agent-Native HTML-to-Video Rendering (/course/hyperframes-html-to-video-agents) | heygen-com/hyperframes | Zero-React standalone HTML/Canvas video compiler driven by window.seek(t). | \n| 08 | faster-whisper on Local GPU: Word-Level Captions & Lip-Sync (/course/faster-whisper-word-level-captions) | SYSTRAN/faster-whisper | Sub-20ms word alignment pipeline driving dynamic captions and mascot visemes. | \n| 09 | Chatterbox TTS: Zero-Shot Local Voice Cloning (/course/chatterbox-local-voice-cloning) | resemble-ai/chatterbox | Expressive local voiceover synthesizer with phonetic code identifier rules. | \n| 10 | FFmpeg for AI Video Pipelines & Contact-Sheet Critics (/course/ffmpeg-ai-video-contact-sheet-critics) | FFmpeg/FFmpeg | -14 LUFS EBU R128 audio normalization, 6-across contact sheets, and H.264/AV1 mastering. | \n| 11 | Chrome DevTools MCP: Self-Healing Visual Agent Loops (/course/chrome-devtools-mcp-visual-agent-loops) | ChromeDevTools/chrome-devtools-mcp | Automated DOM/WebGL inspection loop that catches overlapping text before export. | \n| 12 | Impeccable: Automated Design & Motion Polish Gates (/course/impeccable-automated-design-critique) | pbakaus/impeccable | Pre-export design critic enforcing W3C design tokens and banning generic AI gradients. | \n| 13 | ComfyUI for 3D Turnarounds & Generate-Then-Trace (/course/comfyui-3d-turnarounds-video-workflows) | Comfy-Org/ComfyUI | Reproducible API workflow for multi-view mascot sheets and base motion plates. | \n| 14 | yt-dlp & Librosa: Reference Grammar & Beat Extraction (/course/yt-dlp-librosa-reference-beat-extraction) | yt-dlp/yt-dlp | Automated reference shotlist extractor (style_guide.md) and 120 BPM beats.json generator. | \n\n## 5. Verdict: Purrfect or Pass?\n\n- **Purrfect For** : Developer educators, SaaS founders, and frontend engineers who want agency-grade launch films, interactive 3D mascots (like**3D AJ** ), and automated video courses rendered deterministically from Git-versioned TypeScript and Python code for`$0` in licensing fees.\n- **Pass If** : You need photorealistic live-action human cinema with complex cloth physics (where diffusion video models like Veo 3.1 still belong) or you are unwilling to write a structured`brief.md` and`window.seek(t)` render contract.\n\nHead over to [`/mascot-lab`](https://codingcat.dev/mascot-lab) right now to toggle between **2D Vector AJ** and **3D Three.js GLB AJ**, scrub the **`window.seek(t)`** timeline, download `aj-3d-mascot.glb`, and explore all 19 new Courses on [`/courses`](https://codingcat.dev/courses)!\n\n## how this post connects together.\n\nFollow AJ from first principles to production-ready architecture.\n\n## Watch the Related Video Deep Dive: Cloudflare Agents, Durable Objects & Stateful Workflows in TypeScript\n\nBuild stateful, long-running AI agents and resilient multi-step workflows using the Cloudflare Agents SDK, Durable Objects with embedded SQLite, WebSockets hibernation, and Workflows.\n\n[View Full Page →](https://codingcat.dev/course/cloudflare-agents-durable-workflows-typescript)", "url": "https://wpnews.pro/news/the-14-open-source-repos-powering-code-driven-video-in-2026-how-we-built-3d-aj", "canonical_source": "https://codingcat.dev/post/14-open-source-repos-code-driven-video-how-we-built-3d-aj", "published_at": "2026-11-04 05:00:00+00:00", "updated_at": "2026-11-04 05:50:25.029157+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "generative-ai", "developer-tools", "artificial-intelligence"], "entities": ["CodingCat.dev", "Three.js", "Blender", "Blender MCP", "ahujasid/mcp-for-blender", "FFmpeg", "faster-whisper", "Chrome DevTools MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-14-open-source-repos-powering-code-driven-video-in-2026-how-we-built-3d-aj", "markdown": "https://wpnews.pro/news/the-14-open-source-repos-powering-code-driven-video-in-2026-how-we-built-3d-aj.md", "text": "https://wpnews.pro/news/the-14-open-source-repos-powering-code-driven-video-in-2026-how-we-built-3d-aj.txt", "jsonld": "https://wpnews.pro/news/the-14-open-source-repos-powering-code-driven-video-in-2026-how-we-built-3d-aj.jsonld"}}