{"slug": "building-framemill-turning-hours-of-gameplay-into-an-editable-first-cut", "title": "Building FrameMill: Turning Hours of Gameplay Into an Editable First Cut", "summary": "A developer built FrameMill, an AI-powered tool that turns long game streams into editable video projects. The tool analyzes gameplay, uses Gemini to assemble a first cut, and provides a Timeline that creators can inspect and modify. The project was developed for the Gemini All Agentic AI Hackathon and currently supports Call of Duty: Warzone DMZ.", "body_md": "*Written for the Gemini All Agentic AI Hackathon.*\n\nA game stream can last four hours. The moments worth turning into a video might last only a few minutes.\n\nFinding those moments usually means rewatching the recording, marking timestamps, cutting clips, and rebuilding the story manually. Existing AI clipping tools can reduce that work, but they often return disconnected clips or a finished video with little explanation or control.\n\nThat is the problem I’m exploring with **FrameMill**.\n\nFrameMill turns long game streams into editable video projects. It analyzes what happened, uses AI to assemble a first cut, and gives the creator a real Timeline they can inspect, change, and render again.\n\nI did not want FrameMill to be a black box that decides what a “good highlight” is.\n\nA combat event is a fact. Whether it belongs in an intense montage, a tactical breakdown, or a funny recap depends on the creator’s goal.\n\nThat led to the central ownership model:\n\n```\nProcessor observes.\nGemini interprets, plans, edits, and reviews.\nServer persists and executes capabilities.\nClient presents and changes the Timeline.\nRenderer derives video from a Timeline revision.\n```\n\nThe media processor detects factual gameplay Activities with timestamps, confidence, and source evidence. It does not assign universal highlight scores.\n\nGemini receives those Activities alongside bounded pieces of source media. It can map the relevant parts of a game, plan an edit, select footage, and construct a revisioned Timeline.\n\nThe Timeline—not the generated video—is the canonical result.\n\nFrameMill currently supports *Call of Duty: Warzone DMZ* as its first game.\n\nThe local workflow looks like this:\n\nThe backend is written in Go and contains the local server, processing core, and agent runtime. The editing workspace uses Vue, TypeScript, Vite, Tailwind CSS, and Pinia.\n\nThe hosted architecture is designed around MediaMTX, Cloud Storage, Eventarc, Cloud Run, and Firestore. A relay can forward the creator’s stream to its publishing destination while recording small, immutable segments for asynchronous analysis.\n\nOne of the hardest technical problems has been processing long recordings without treating the entire video as one giant request.\n\nFrameMill performs sparse checks across the footage, then uses denser 100 ms refinement only near promising regions. Persistent decoder contexts and bounded parallelism avoid repeatedly reopening the same media.\n\nAn important lesson was that decoding architecture mattered more than micro-optimizing individual detectors. If every detector independently reads the video, small optimizations elsewhere do not help much.\n\nThe better design is to decode once and let multiple detectors consume the same frame evidence.\n\nVideo analysis, model calls, and rendering can all take longer than a browser session. They can also fail independently.\n\nFrameMill therefore treats recovery as part of the product:\n\nIf the browser closes, accepted work should continue. When the creator returns, the client should reconstruct the latest durable state rather than starting over.\n\nI used Codex as an engineering collaborator throughout the project.\n\nIt helped me inspect unfamiliar code, investigate real gameplay footage, create tests, compare performance measurements, and coordinate changes across isolated Git worktrees.\n\nThe most valuable part was not generating code quickly. It was being able to challenge architectural decisions, trace ownership across the system, and verify whether an implementation actually matched the intended product.\n\nI still retained control over the product model, system boundaries, tradeoffs, and final implementation decisions.\n\nThe current MVP proves a real local vertical slice: source analysis, Activity mapping, Timeline validation, and FFmpeg preview rendering.\n\nThe browser workspace demonstrates evidence review, Timeline editing, provenance, revision-conflict recovery, and the render workflow. Some of that workspace still uses canonical fixtures while the remaining server-backed Timeline and control endpoints are completed.\n\nThe hosted path is also still being integrated end to end. I would rather label those boundaries clearly than present a demo fixture as production infrastructure.\n\nThe biggest lesson is that AI does not need to own the entire system to be useful.\n\nDeterministic software is better at observing facts, validating contracts, tracking state, and reproducing outputs. Gemini is better used where interpretation and editorial judgment are actually required.\n\nI also learned that **editability is part of AI trust**.\n\nA creator should be able to see why a clip was chosen, change the decision, and render a different result. The goal is not to remove the creator from editing. It is to remove the hours of searching that happen before the creative work can begin.\n\nFrameMill is my attempt to make that distinction real.", "url": "https://wpnews.pro/news/building-framemill-turning-hours-of-gameplay-into-an-editable-first-cut", "canonical_source": "https://dev.to/prof2k/building-framemill-turning-hours-of-gameplay-into-an-editable-first-cut-4gb2", "published_at": "2026-08-31 23:07:51+00:00", "updated_at": "2026-08-31 23:22:58.888426+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-agents", "developer-tools", "computer-vision"], "entities": ["FrameMill", "Gemini", "Codex", "Call of Duty: Warzone DMZ", "Go", "Vue", "TypeScript", "FFmpeg"], "alternates": {"html": "https://wpnews.pro/news/building-framemill-turning-hours-of-gameplay-into-an-editable-first-cut", "markdown": "https://wpnews.pro/news/building-framemill-turning-hours-of-gameplay-into-an-editable-first-cut.md", "text": "https://wpnews.pro/news/building-framemill-turning-hours-of-gameplay-into-an-editable-first-cut.txt", "jsonld": "https://wpnews.pro/news/building-framemill-turning-hours-of-gameplay-into-an-editable-first-cut.jsonld"}}