{"slug": "teaching-claude-code-to-direct-a-stateful-video-editing-skill-built-on-gemini-s", "title": "Teaching Claude Code to Direct: A Stateful Video-Editing Skill Built on Gemini's Interactions API and MCP", "summary": "A developer built omni-skill-claude, a Claude Code skill that wraps Google's gemini-omni-flash-preview model in a FastMCP server, enabling stateful video editing from the terminal. The skill leverages Gemini's Interactions API to allow iterative edits without re-prompting, supporting tasks like generating videos from text, animating images, interpolating keyframes, restyling clips, and uploading to YouTube.", "body_md": "TL;DR:[omni-skill-claude]wraps Google's`gemini-omni-flash-preview`\n\nmodel (Omni Flash) in a tiny FastMCP server and packages it as a Claude Code skill. You type \"generate a video of a fox running through snow\" into Claude Code, and it just... does it. Then you say \"make it nighttime with snowfall\" and it editsthe same videowithout re-prompting the whole scene. It can also animate a still image, interpolate between two keyframes, restyle a video you already have — and when you're happy, upload the result to YouTube. Without leaving your terminal.\n\nMost video-generation workflows are **stateless**. You send a prompt, you get frames back, and the model immediately forgets everything. Want to tweak the result? You re-describe the *entire scene* and pray the character, lighting, and camera work survive the round trip. (Narrator: they don't.)\n\nGoogle's **Omni Flash** — `gemini-omni-flash-preview`\n\n— takes a different approach. It's the video-generation model in Google's Gemini \"Omni\" line: built for fast, high-fidelity clips, and — the headline feature — wired into the **stateful Interactions API**, which lets you iterate on a video across multiple turns while the model keeps the visual context server-side.\n\nThe \"Omni\" part isn't branding fluff — the model accepts genuinely mixed multimodal input. A single request's `input`\n\ncan be a plain string, or a list of typed parts: `text`\n\nparts, base64-encoded `image`\n\nparts, and `document`\n\nparts pointing at a video you've uploaded via the Gemini File API. The model composes whatever you hand it into one clip. That single mechanism covers five distinct ways to make a video:\n\n`.mp4`\n\nout — landscape `16:9`\n\nor portrait `9:16`\n\n, chosen at generation time.And on top of all five sits the stateful layer: every one of those calls (made with `store=True`\n\n) returns an **interaction ID**, and any result can then be refined turn after turn with incremental edit prompts — same characters, same lighting, same camera language — because the model retrieves the stored visual context instead of making you re-describe it.\n\nThree practical realities to know going in: generation is **synchronous and slow** (the call blocks until the video is ready), it's **billable per generation**, and outputs get big fast — past ~4 MB you want File-API delivery instead of inline base64. The server and skill below exist largely to absorb those realities for you.\n\nThis repo glues all of that into **Claude Code**, so your coding agent can generate and iteratively refine videos as a natural part of a session. It ships as two things in one repo:\n\n`omni-video-agent`\n\n, a single-file FastMCP app in `server.py`\n\n) exposing exactly eight tools.`omni-video`\n\n) that teaches Claude The Interactions API is Gemini's stateful endpoint. The core loop looks like this:\n\n`client.interactions.create(...)`\n\nwith a prompt and `store=True`\n\n.`interaction_id`\n\n`previous_interaction_id`\n\n, and the model edits the So instead of this (stateless suffering):\n\n\"A tracking shot of a red fox running through fresh snow at golden hour, birch trees, low sun, shallow depth of field,\n\nand now alsoat night with heavy snowfall\"\n\n...you write this:\n\n\"Make it nighttime with heavy snowfall.\"\n\nThat's it. The stored context holds the rest.\n\nA few practical details the server handles for you:\n\n`16:9`\n\nlandscape or `9:16`\n\nportrait) and `inline`\n\n(default — the video comes back as base64, fine for short clips) or `uri`\n\n— the output lands on the Google File API and the server polls until it's ready, then downloads it. Videos get big fast; past ~4 MB, `uri`\n\nsaves you from payload-limit failures you'd otherwise discover the hard way.The **Model Context Protocol** is an open standard for connecting AI assistants to tools and data. Before it, giving a model access to some service meant writing a bespoke integration for each assistant — N assistants × M services, everyone reinventing the same plumbing. MCP collapses that: a tool author writes one **MCP server** that exposes typed tools, and any MCP-capable client (Claude Code, Claude Desktop, and a growing list of others) can discover and call them with no per-client glue code.\n\nAn MCP server is usually a small local process that speaks JSON-RPC over stdio. The client launches it, asks \"what tools do you have?\", and from then on the model can call them like functions.\n\nThe `omni-video-agent`\n\nserver exposes exactly eight:\n\n| Tool | What it does |\n|---|---|\n`generate_video` |\nText → video. Saves locally as `.mp4` , returns the path + an interaction ID. |\n`edit_video` |\nStateful edit: takes the previous interaction ID + a description of only the change. |\n`animate_image` |\nA still image + a motion prompt → the image comes to life. |\n`interpolate_images` |\nTwo keyframe images + a transition prompt → the video between them. |\n`generate_with_subjects` |\nReference images of people/objects + a scene prompt → those subjects, directed. |\n`edit_user_video` |\nUploads a video you already have via the Gemini File API and restyles it (\"Make it a Pixar animation style\"). |\n`upload_to_youtube` |\nPublishes a finished `.mp4` via the YouTube Data API v3 (one-time OAuth setup; defaults to `private` ). |\n`get_help` |\nThe full tool reference, delivery-mode guidance, and cinematic prompting tips. |\n\nErrors come back as `🔴 ...`\n\ntext strings rather than protocol errors, so the agent can read and react to them.\n\nTwo conventions run through the whole surface. Every video tool takes a `delivery`\n\nparameter — `'inline'`\n\n(default; the video comes back as base64 in the response) or `'uri'`\n\n(the output lands on the Google File API and the server polls until it's `ACTIVE`\n\n, then downloads — use it for anything over ~4 MB). And every video tool returns a text report carrying the saved local path plus the **interaction ID** to chain into the next edit. Videos land on disk as `<prefix>_<unix-timestamp>.mp4`\n\n, with a prefix per tool.\n\n`generate_video`\n\n— text → video\n\n```\ngenerate_video(prompt: str, aspect_ratio: str = \"16:9\", delivery: str = \"inline\") -> str\n```\n\nThe starting point. `aspect_ratio`\n\nis `'16:9'`\n\n(landscape) or `'9:16'`\n\n(portrait) — this is the **only** tool that accepts one, because stateful edits inherit it; any other value silently falls back to the model default. Under the hood it's a single `client.interactions.create(...)`\n\nwith `store=True`\n\n, so the result is immediately editable. Saves as `gen_*.mp4`\n\n.\n\n`edit_video`\n\n— the stateful edit\n\n```\nedit_video(previous_interaction_id: str, edit_prompt: str, delivery: str = \"inline\") -> str\n```\n\nThe tool the whole architecture is built around. Pass the interaction ID from the **latest** turn and describe *only the change* — the stored context holds the rest. Each call returns a *new* ID; chain that one next, because editing from a stale ID silently forks the session from an older state. Deliberately has no `aspect_ratio`\n\nparameter — it's inherited. Saves as `edit_*.mp4`\n\n.\n\n`animate_image`\n\n— still image → motion\n\n``` php\nanimate_image(image_path: str, motion_prompt: str, delivery: str = \"inline\") -> str\n```\n\nReads a local image (png/jpg/jpeg/webp — mime type inferred from the extension, anything else sent as png), base64-encodes it, and sends `[image, text]`\n\nas the multimodal input. Saves as `animated_*.mp4`\n\n.\n\n`interpolate_images`\n\n— two keyframes → the footage between them\n\n```\ninterpolate_images(start_image_path: str, end_image_path: str, prompt: str, delivery: str = \"inline\") -> str\n```\n\nSame encoding as `animate_image`\n\n, but the input is `[start_image, end_image, text]`\n\nand the prompt describes the transition (\"a smooth timelapse from sunrise to sunset\"). Saves as `interpolation_*.mp4`\n\n.\n\n`generate_with_subjects`\n\n— reference images, directed\n\n```\ngenerate_with_subjects(subject_image_paths: list[str], prompt: str, delivery: str = \"inline\") -> str\n```\n\nEvery path in the list becomes an image part, the scene prompt goes last, and the model generates a video featuring those subjects. Saves as `subject_*.mp4`\n\n.\n\n`edit_user_video`\n\n— restyle footage you already have\n\n``` php\nedit_user_video(video_path: str, edit_prompt: str, delivery: str = \"inline\") -> str\n```\n\nThe one tool that touches the Gemini **File API on input**: it uploads your local video, polls until processing completes (up to 5 minutes), then sends `[document, text]`\n\n— the uploaded video referenced by URI plus your edit instruction. Saves as `user_edit_*.mp4`\n\n.\n\n`upload_to_youtube`\n\n— publish the final cut\n\n```\nupload_to_youtube(video_path: str, title: str, description: str,\n                  category_id: str = \"22\", privacy_status: str = \"private\") -> str\n```\n\nYouTube Data API v3. Needs a one-time OAuth setup (`client_secrets.json`\n\nin the server's working directory; first run opens a browser and caches `token.pickle`\n\n). `category_id`\n\ndefaults to `'22'`\n\n(People & Blogs); `privacy_status`\n\nis `'private'`\n\n, `'public'`\n\n, or `'unlisted'`\n\n— defaulting to `private`\n\n, so nothing goes live by accident. Its errors use `❌ ...`\n\ninstead of `🔴 ...`\n\n, and its extra dependencies are optional — the tool reports the exact `pip install`\n\ncommand if they're missing.\n\n`get_help`\n\n— the built-in manual\n\n``` php\nget_help() -> str\n```\n\nNo parameters. Returns the full tool catalog, delivery-mode guidance, and a cinematic prompting guide — so an agent (or a curious human) can orient without leaving the session.\n\nIf MCP is the *hands* (the tools Claude can physically call), a **skill** is the *muscle memory* — a markdown file (`SKILL.md`\n\n) plus bundled resources that load into Claude's context and teach it the workflow: which tool to reach for, in what order, with which constraints.\n\nFor `omni-video`\n\n, the skill encodes things like:\n\n`delivery='uri'`\n\nfor anything long or high-motion.`public`\n\nto YouTube.The skill also bundles the MCP server itself (`mcp/server.py`\n\n), its requirements, an installer script, and the Interactions API video guide — so it's self-contained: install the skill, and you have everything needed to also stand up the server.\n\nYou need three things: **Python 3.10+**, **Claude Code**, and a **Gemini API key** (free from [Google AI Studio](https://aistudio.google.com/)). Pick *one* of the paths below.\n\nInside Claude Code, type:\n\n```\n/plugin marketplace add xbill9/omni-skill-claude\n/plugin install omni-video@omni-skill-claude\n```\n\nThis installs the skill **and** auto-registers the MCP server. The plugin manifest carries no API key (as it should!) — the server reads `GEMINI_API_KEY`\n\nfrom your environment, so make sure it's exported before launching Claude Code.\n\n```\n# 1. Get the code\ngit clone https://github.com/xbill9/omni-skill-claude.git\ncd omni-skill-claude\n\n# 2. One-command setup: installs deps, registers the MCP server\n#    in .mcp.json, and prompts for your API key (stored in ~/gemini.key)\n./init.sh\n\n# 3. Restart Claude Code in this directory and approve the server\n#    when prompted. Verify with:\n/mcp        # should list omni-video-agent\n```\n\nThat's genuinely it. `init.sh`\n\nis safe to rerun if anything looks off.\n\nFrom a clone of the repo:\n\n```\nmake init TARGET=/path/to/your/project\n```\n\nThis copies the skill into `<project>/.claude/skills/omni-video/`\n\nand writes the `omni-video-agent`\n\nentry into that project's `.mcp.json`\n\n. It reuses `~/gemini.key`\n\nif you've set one up. Restart Claude Code in the target project, approve the server, done. Generated videos land in the project directory.\n\nThe repo ships a Dockerfile that builds an image containing only the server and its deps — no keys, no Claude Code:\n\n```\nmake docker-build   # builds xbill9/omni-video-agent\n\nclaude mcp add omni-video-agent --env GEMINI_API_KEY=\"$(cat ~/gemini.key)\" -- \\\n  docker run --rm -i -e GEMINI_API_KEY -v \"$PWD:$PWD\" -w \"$PWD\" xbill9/omni-video-agent\n```\n\nThe `-v \"$PWD:$PWD\" -w \"$PWD\"`\n\nmount matters: the server saves videos to disk and reads local files for the image/video-input tools, so the container must see your project at the *same absolute path* as the host. (One caveat: `upload_to_youtube`\n\n's first-run OAuth flow opens a browser, which containers famously don't have — run that one from a host install.)\n\n`/mcp`\n\ndoesn't list the server → restart Claude Code in the project directory.`source set_env.sh`\n\n(or export `GEMINI_API_KEY`\n\n) and restart.`get_help`\n\n; failures come back as readable `🔴 ...`\n\nstrings.Once installed, you talk to it in plain English. A real flow looks like:\n\n**You:** *\"Generate a video of a red fox running through fresh snow at golden hour, 16:9.\"*\n\nClaude calls:\n\n```\ngenerate_video(\n    prompt=\"A tracking shot of a red fox running through fresh snow at golden hour\",\n    aspect_ratio=\"16:9\",\n    delivery=\"uri\",\n)\n# 🟢 Video successfully saved!\n# • Saved to: ./gen_1784759001.mp4\n# • Interaction ID: v1_ChdpRU5...\n```\n\n**You:** *\"Nice. Make it nighttime, heavy snowfall.\"*\n\n```\nedit_video(\n    previous_interaction_id=\"v1_ChdpRU5...\",\n    edit_prompt=\"make it nighttime with heavy snowfall\",\n    delivery=\"uri\",\n)\n# 🟢 Video successfully saved!\n# • Saved to: ./edit_1784759050.mp4\n# • Interaction ID: v1_Xk9mPq2...   ← a NEW id; the next edit chains this one\n```\n\nSame fox, same trees, same camera move — only the time of day and weather change. No re-prompting, no continuity roulette.\n\nAnd for footage that didn't come from the model at all:\n\n**You:** *\"Take ./team-photo.png and animate it — everyone waves at the camera.\"*\n\n```\nanimate_image(\n    image_path=\"./team-photo.png\",\n    motion_prompt=\"the group smiles and waves at the camera, subtle handheld motion\",\n)\n```\n\n**You:** *\"Turn ./demo-screencast.mp4 into a Pixar-style animation.\"*\n\n```\nedit_user_video(\n    video_path=\"./demo-screencast.mp4\",\n    edit_prompt=\"Make it a Pixar animation style\",\n    delivery=\"uri\",\n)\n```\n\nBoth return interaction IDs too — so follow-up refinements switch to `edit_video`\n\nand go stateful from there. And when the cut is final:\n\n**You:** *\"Ship it to YouTube, unlisted.\"*\n\n```\nupload_to_youtube(\n    video_path=\"./edit_1784759050.mp4\",\n    title=\"Fox in the Snow — generated with Omni Flash\",\n    description=\"Generated and edited with the omni-video Claude Code skill.\",\n    privacy_status=\"unlisted\",\n)\n# 🟢 Video successfully uploaded to YouTube!\n# • URL: https://www.youtube.com/watch?v=...\n```\n\nFirst run, the tool walks you through the one-time OAuth setup (a `client_secrets.json`\n\nfrom Google Cloud Console; the token is cached after that). Prompt to published URL, all inside one Claude Code session.\n\nIf the term is new to you: **\"eating your own dog food\"** means using your own product for real work, not just demoing it. It's the difference between \"this should work\" and \"I ship with this every day.\" If a tool is good enough for your users, it should be good enough for you — and if it isn't, you'll be the first to feel the pain and fix it.\n\nThis repo dogfoods itself at every layer:\n\n`omni-video`\n\nskill and `omni-video-agent`\n\nserver are already wired up, so every development session doubles as an integration test.\n\n```\ngenerate_video(\n    prompt=\"A tracking shot of a red fox running through fresh snow at golden hour, \"\n           \"birch trees in the background, low sun flaring through the branches, \"\n           \"shallow depth of field, photorealistic, cinematic\",\n    aspect_ratio=\"16:9\",\n    delivery=\"uri\",\n)\n# 🟢 Video successfully saved!\n# • Saved to: gen_1784824947.mp4\n# • Interaction ID: v1_Chdja1JpYXFyVkVkcWVqTWNQaHFULW9BWRIXY2tSaWFxclZFZHFlak1jUGhxVC1vQVk\n```\n\nOne incremental edit later — note that only the change is described, nothing about the fox, the trees, or the camera:\n\n```\nedit_video(\n    previous_interaction_id=\"v1_Chdja1JpYXFyVkVkcWVqTWNQaHFULW9BWRIXY2tSaWFxclZFZHFlak1jUGhxVC1vQVk\",\n    edit_prompt=\"make it nighttime with heavy snowfall, moonlight instead of golden hour\",\n    delivery=\"uri\",\n)\n# 🟢 Video successfully saved!\n# • Saved to: edit_1784825027.mp4\n# • Interaction ID: v1_Chdja1JpYXFyVkVkcWVqTWNQaHFULW9BWRIXd2tSaWF1RGdHXzZhX3VNUHI4LThzUTQ\n```\n\nA detail you only notice with real receipts in hand: the two interaction IDs share their first half. The session lineage is visible in the ID itself — the common prefix is the stored context both turns belong to, and the differing tail is the new turn. Also worth noting: both clips came out around 2.6 MB, under the ~4 MB inline ceiling — but `delivery=\"uri\"`\n\nwas the right call anyway, because you don't know the size until it's too late.\n\nAnd here is that final cut — published straight from the same session with the skill's own `upload_to_youtube`\n\ntool (`privacy_status=\"unlisted\"`\n\n), so the publishing step got dogfooded too:\n\n`gen_1784824947.mp4`\n\n, two seconds in) — so the header art was generated by the tool the article describes, too.`edit_video`\n\nwith the latest interaction ID (`...UHI4LThzUTQ`\n\n, the second one, not the first) and describe the change. That's the whole point.Dogfooding is the cheapest credibility there is: no cherry-picked gallery, no \"results may vary\" fine print — the tool's real output is embedded right here, receipts and all. If the model had mangled the motion or lost the fox between edits, you'd be looking at the evidence right now.\n\n*This is a third-party community project, not affiliated with or endorsed by Anthropic or Google. Bring your own Gemini API key — and remember video generations are billable and slow, so nail the prompt, batch your edits, and save the YouTube upload for the final cut.*", "url": "https://wpnews.pro/news/teaching-claude-code-to-direct-a-stateful-video-editing-skill-built-on-gemini-s", "canonical_source": "https://dev.to/gde/teaching-claude-code-to-direct-a-stateful-video-editing-skill-built-on-geminis-interactions-api-2h7l", "published_at": "2026-07-23 17:17:43+00:00", "updated_at": "2026-07-23 17:34:48.445535+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "developer-tools", "ai-products"], "entities": ["Google", "Gemini", "Claude Code", "FastMCP", "YouTube", "Omni Flash", "Interactions API", "Model Context Protocol"], "alternates": {"html": "https://wpnews.pro/news/teaching-claude-code-to-direct-a-stateful-video-editing-skill-built-on-gemini-s", "markdown": "https://wpnews.pro/news/teaching-claude-code-to-direct-a-stateful-video-editing-skill-built-on-gemini-s.md", "text": "https://wpnews.pro/news/teaching-claude-code-to-direct-a-stateful-video-editing-skill-built-on-gemini-s.txt", "jsonld": "https://wpnews.pro/news/teaching-claude-code-to-direct-a-stateful-video-editing-skill-built-on-gemini-s.jsonld"}}