# Teaching Claude Code to Direct: A Stateful Video-Editing Skill Built on Gemini's Interactions API and MCP

> Source: <https://dev.to/gde/teaching-claude-code-to-direct-a-stateful-video-editing-skill-built-on-geminis-interactions-api-2h7l>
> Published: 2026-07-23 17:17:43+00:00

TL;DR:[omni-skill-claude]wraps Google's`gemini-omni-flash-preview`

model (Omni Flash) in a tiny FastMCP server and packages it as a Claude Code skill. You type "generate a video of a fox running through snow" into Claude Code, and it just... does it. Then you say "make it nighttime with snowfall" and it editsthe same videowithout re-prompting the whole scene. It can also animate a still image, interpolate between two keyframes, restyle a video you already have — and when you're happy, upload the result to YouTube. Without leaving your terminal.

Most video-generation workflows are **stateless**. You send a prompt, you get frames back, and the model immediately forgets everything. Want to tweak the result? You re-describe the *entire scene* and pray the character, lighting, and camera work survive the round trip. (Narrator: they don't.)

Google's **Omni Flash** — `gemini-omni-flash-preview`

— takes a different approach. It's the video-generation model in Google's Gemini "Omni" line: built for fast, high-fidelity clips, and — the headline feature — wired into the **stateful Interactions API**, which lets you iterate on a video across multiple turns while the model keeps the visual context server-side.

The "Omni" part isn't branding fluff — the model accepts genuinely mixed multimodal input. A single request's `input`

can be a plain string, or a list of typed parts: `text`

parts, base64-encoded `image`

parts, and `document`

parts pointing at a video you've uploaded via the Gemini File API. The model composes whatever you hand it into one clip. That single mechanism covers five distinct ways to make a video:

`.mp4`

out — landscape `16:9`

or portrait `9:16`

, chosen at generation time.And on top of all five sits the stateful layer: every one of those calls (made with `store=True`

) returns an **interaction ID**, and any result can then be refined turn after turn with incremental edit prompts — same characters, same lighting, same camera language — because the model retrieves the stored visual context instead of making you re-describe it.

Three practical realities to know going in: generation is **synchronous and slow** (the call blocks until the video is ready), it's **billable per generation**, and outputs get big fast — past ~4 MB you want File-API delivery instead of inline base64. The server and skill below exist largely to absorb those realities for you.

This repo glues all of that into **Claude Code**, so your coding agent can generate and iteratively refine videos as a natural part of a session. It ships as two things in one repo:

`omni-video-agent`

, a single-file FastMCP app in `server.py`

) exposing exactly eight tools.`omni-video`

) that teaches Claude The Interactions API is Gemini's stateful endpoint. The core loop looks like this:

`client.interactions.create(...)`

with a prompt and `store=True`

.`interaction_id`

`previous_interaction_id`

, and the model edits the So instead of this (stateless suffering):

"A tracking shot of a red fox running through fresh snow at golden hour, birch trees, low sun, shallow depth of field,

and now alsoat night with heavy snowfall"

...you write this:

"Make it nighttime with heavy snowfall."

That's it. The stored context holds the rest.

A few practical details the server handles for you:

`16:9`

landscape or `9:16`

portrait) and `inline`

(default — the video comes back as base64, fine for short clips) or `uri`

— the output lands on the Google File API and the server polls until it's ready, then downloads it. Videos get big fast; past ~4 MB, `uri`

saves you from payload-limit failures you'd otherwise discover the hard way.The **Model Context Protocol** is an open standard for connecting AI assistants to tools and data. Before it, giving a model access to some service meant writing a bespoke integration for each assistant — N assistants × M services, everyone reinventing the same plumbing. MCP collapses that: a tool author writes one **MCP server** that exposes typed tools, and any MCP-capable client (Claude Code, Claude Desktop, and a growing list of others) can discover and call them with no per-client glue code.

An MCP server is usually a small local process that speaks JSON-RPC over stdio. The client launches it, asks "what tools do you have?", and from then on the model can call them like functions.

The `omni-video-agent`

server exposes exactly eight:

| Tool | What it does |
|---|---|
`generate_video` |
Text → video. Saves locally as `.mp4` , returns the path + an interaction ID. |
`edit_video` |
Stateful edit: takes the previous interaction ID + a description of only the change. |
`animate_image` |
A still image + a motion prompt → the image comes to life. |
`interpolate_images` |
Two keyframe images + a transition prompt → the video between them. |
`generate_with_subjects` |
Reference images of people/objects + a scene prompt → those subjects, directed. |
`edit_user_video` |
Uploads a video you already have via the Gemini File API and restyles it ("Make it a Pixar animation style"). |
`upload_to_youtube` |
Publishes a finished `.mp4` via the YouTube Data API v3 (one-time OAuth setup; defaults to `private` ). |
`get_help` |
The full tool reference, delivery-mode guidance, and cinematic prompting tips. |

Errors come back as `🔴 ...`

text strings rather than protocol errors, so the agent can read and react to them.

Two conventions run through the whole surface. Every video tool takes a `delivery`

parameter — `'inline'`

(default; the video comes back as base64 in the response) or `'uri'`

(the output lands on the Google File API and the server polls until it's `ACTIVE`

, then downloads — use it for anything over ~4 MB). And every video tool returns a text report carrying the saved local path plus the **interaction ID** to chain into the next edit. Videos land on disk as `<prefix>_<unix-timestamp>.mp4`

, with a prefix per tool.

`generate_video`

— text → video

```
generate_video(prompt: str, aspect_ratio: str = "16:9", delivery: str = "inline") -> str
```

The starting point. `aspect_ratio`

is `'16:9'`

(landscape) or `'9:16'`

(portrait) — this is the **only** tool that accepts one, because stateful edits inherit it; any other value silently falls back to the model default. Under the hood it's a single `client.interactions.create(...)`

with `store=True`

, so the result is immediately editable. Saves as `gen_*.mp4`

.

`edit_video`

— the stateful edit

```
edit_video(previous_interaction_id: str, edit_prompt: str, delivery: str = "inline") -> str
```

The tool the whole architecture is built around. Pass the interaction ID from the **latest** turn and describe *only the change* — the stored context holds the rest. Each call returns a *new* ID; chain that one next, because editing from a stale ID silently forks the session from an older state. Deliberately has no `aspect_ratio`

parameter — it's inherited. Saves as `edit_*.mp4`

.

`animate_image`

— still image → motion

``` php
animate_image(image_path: str, motion_prompt: str, delivery: str = "inline") -> str
```

Reads a local image (png/jpg/jpeg/webp — mime type inferred from the extension, anything else sent as png), base64-encodes it, and sends `[image, text]`

as the multimodal input. Saves as `animated_*.mp4`

.

`interpolate_images`

— two keyframes → the footage between them

```
interpolate_images(start_image_path: str, end_image_path: str, prompt: str, delivery: str = "inline") -> str
```

Same encoding as `animate_image`

, but the input is `[start_image, end_image, text]`

and the prompt describes the transition ("a smooth timelapse from sunrise to sunset"). Saves as `interpolation_*.mp4`

.

`generate_with_subjects`

— reference images, directed

```
generate_with_subjects(subject_image_paths: list[str], prompt: str, delivery: str = "inline") -> str
```

Every path in the list becomes an image part, the scene prompt goes last, and the model generates a video featuring those subjects. Saves as `subject_*.mp4`

.

`edit_user_video`

— restyle footage you already have

``` php
edit_user_video(video_path: str, edit_prompt: str, delivery: str = "inline") -> str
```

The one tool that touches the Gemini **File API on input**: it uploads your local video, polls until processing completes (up to 5 minutes), then sends `[document, text]`

— the uploaded video referenced by URI plus your edit instruction. Saves as `user_edit_*.mp4`

.

`upload_to_youtube`

— publish the final cut

```
upload_to_youtube(video_path: str, title: str, description: str,
                  category_id: str = "22", privacy_status: str = "private") -> str
```

YouTube Data API v3. Needs a one-time OAuth setup (`client_secrets.json`

in the server's working directory; first run opens a browser and caches `token.pickle`

). `category_id`

defaults to `'22'`

(People & Blogs); `privacy_status`

is `'private'`

, `'public'`

, or `'unlisted'`

— defaulting to `private`

, so nothing goes live by accident. Its errors use `❌ ...`

instead of `🔴 ...`

, and its extra dependencies are optional — the tool reports the exact `pip install`

command if they're missing.

`get_help`

— the built-in manual

``` php
get_help() -> str
```

No parameters. Returns the full tool catalog, delivery-mode guidance, and a cinematic prompting guide — so an agent (or a curious human) can orient without leaving the session.

If MCP is the *hands* (the tools Claude can physically call), a **skill** is the *muscle memory* — a markdown file (`SKILL.md`

) plus bundled resources that load into Claude's context and teach it the workflow: which tool to reach for, in what order, with which constraints.

For `omni-video`

, the skill encodes things like:

`delivery='uri'`

for anything long or high-motion.`public`

to YouTube.The skill also bundles the MCP server itself (`mcp/server.py`

), its requirements, an installer script, and the Interactions API video guide — so it's self-contained: install the skill, and you have everything needed to also stand up the server.

You need three things: **Python 3.10+**, **Claude Code**, and a **Gemini API key** (free from [Google AI Studio](https://aistudio.google.com/)). Pick *one* of the paths below.

Inside Claude Code, type:

```
/plugin marketplace add xbill9/omni-skill-claude
/plugin install omni-video@omni-skill-claude
```

This installs the skill **and** auto-registers the MCP server. The plugin manifest carries no API key (as it should!) — the server reads `GEMINI_API_KEY`

from your environment, so make sure it's exported before launching Claude Code.

```
# 1. Get the code
git clone https://github.com/xbill9/omni-skill-claude.git
cd omni-skill-claude

# 2. One-command setup: installs deps, registers the MCP server
#    in .mcp.json, and prompts for your API key (stored in ~/gemini.key)
./init.sh

# 3. Restart Claude Code in this directory and approve the server
#    when prompted. Verify with:
/mcp        # should list omni-video-agent
```

That's genuinely it. `init.sh`

is safe to rerun if anything looks off.

From a clone of the repo:

```
make init TARGET=/path/to/your/project
```

This copies the skill into `<project>/.claude/skills/omni-video/`

and writes the `omni-video-agent`

entry into that project's `.mcp.json`

. It reuses `~/gemini.key`

if you've set one up. Restart Claude Code in the target project, approve the server, done. Generated videos land in the project directory.

The repo ships a Dockerfile that builds an image containing only the server and its deps — no keys, no Claude Code:

```
make docker-build   # builds xbill9/omni-video-agent

claude mcp add omni-video-agent --env GEMINI_API_KEY="$(cat ~/gemini.key)" -- \
  docker run --rm -i -e GEMINI_API_KEY -v "$PWD:$PWD" -w "$PWD" xbill9/omni-video-agent
```

The `-v "$PWD:$PWD" -w "$PWD"`

mount matters: the server saves videos to disk and reads local files for the image/video-input tools, so the container must see your project at the *same absolute path* as the host. (One caveat: `upload_to_youtube`

's first-run OAuth flow opens a browser, which containers famously don't have — run that one from a host install.)

`/mcp`

doesn't list the server → restart Claude Code in the project directory.`source set_env.sh`

(or export `GEMINI_API_KEY`

) and restart.`get_help`

; failures come back as readable `🔴 ...`

strings.Once installed, you talk to it in plain English. A real flow looks like:

**You:** *"Generate a video of a red fox running through fresh snow at golden hour, 16:9."*

Claude calls:

```
generate_video(
    prompt="A tracking shot of a red fox running through fresh snow at golden hour",
    aspect_ratio="16:9",
    delivery="uri",
)
# 🟢 Video successfully saved!
# • Saved to: ./gen_1784759001.mp4
# • Interaction ID: v1_ChdpRU5...
```

**You:** *"Nice. Make it nighttime, heavy snowfall."*

```
edit_video(
    previous_interaction_id="v1_ChdpRU5...",
    edit_prompt="make it nighttime with heavy snowfall",
    delivery="uri",
)
# 🟢 Video successfully saved!
# • Saved to: ./edit_1784759050.mp4
# • Interaction ID: v1_Xk9mPq2...   ← a NEW id; the next edit chains this one
```

Same fox, same trees, same camera move — only the time of day and weather change. No re-prompting, no continuity roulette.

And for footage that didn't come from the model at all:

**You:** *"Take ./team-photo.png and animate it — everyone waves at the camera."*

```
animate_image(
    image_path="./team-photo.png",
    motion_prompt="the group smiles and waves at the camera, subtle handheld motion",
)
```

**You:** *"Turn ./demo-screencast.mp4 into a Pixar-style animation."*

```
edit_user_video(
    video_path="./demo-screencast.mp4",
    edit_prompt="Make it a Pixar animation style",
    delivery="uri",
)
```

Both return interaction IDs too — so follow-up refinements switch to `edit_video`

and go stateful from there. And when the cut is final:

**You:** *"Ship it to YouTube, unlisted."*

```
upload_to_youtube(
    video_path="./edit_1784759050.mp4",
    title="Fox in the Snow — generated with Omni Flash",
    description="Generated and edited with the omni-video Claude Code skill.",
    privacy_status="unlisted",
)
# 🟢 Video successfully uploaded to YouTube!
# • URL: https://www.youtube.com/watch?v=...
```

First run, the tool walks you through the one-time OAuth setup (a `client_secrets.json`

from Google Cloud Console; the token is cached after that). Prompt to published URL, all inside one Claude Code session.

If the term is new to you: **"eating your own dog food"** means using your own product for real work, not just demoing it. It's the difference between "this should work" and "I ship with this every day." If a tool is good enough for your users, it should be good enough for you — and if it isn't, you'll be the first to feel the pain and fix it.

This repo dogfoods itself at every layer:

`omni-video`

skill and `omni-video-agent`

server are already wired up, so every development session doubles as an integration test.

```
generate_video(
    prompt="A tracking shot of a red fox running through fresh snow at golden hour, "
           "birch trees in the background, low sun flaring through the branches, "
           "shallow depth of field, photorealistic, cinematic",
    aspect_ratio="16:9",
    delivery="uri",
)
# 🟢 Video successfully saved!
# • Saved to: gen_1784824947.mp4
# • Interaction ID: v1_Chdja1JpYXFyVkVkcWVqTWNQaHFULW9BWRIXY2tSaWFxclZFZHFlak1jUGhxVC1vQVk
```

One incremental edit later — note that only the change is described, nothing about the fox, the trees, or the camera:

```
edit_video(
    previous_interaction_id="v1_Chdja1JpYXFyVkVkcWVqTWNQaHFULW9BWRIXY2tSaWFxclZFZHFlak1jUGhxVC1vQVk",
    edit_prompt="make it nighttime with heavy snowfall, moonlight instead of golden hour",
    delivery="uri",
)
# 🟢 Video successfully saved!
# • Saved to: edit_1784825027.mp4
# • Interaction ID: v1_Chdja1JpYXFyVkVkcWVqTWNQaHFULW9BWRIXd2tSaWF1RGdHXzZhX3VNUHI4LThzUTQ
```

A detail you only notice with real receipts in hand: the two interaction IDs share their first half. The session lineage is visible in the ID itself — the common prefix is the stored context both turns belong to, and the differing tail is the new turn. Also worth noting: both clips came out around 2.6 MB, under the ~4 MB inline ceiling — but `delivery="uri"`

was the right call anyway, because you don't know the size until it's too late.

And here is that final cut — published straight from the same session with the skill's own `upload_to_youtube`

tool (`privacy_status="unlisted"`

), so the publishing step got dogfooded too:

`gen_1784824947.mp4`

, two seconds in) — so the header art was generated by the tool the article describes, too.`edit_video`

with the latest interaction ID (`...UHI4LThzUTQ`

, the second one, not the first) and describe the change. That's the whole point.Dogfooding is the cheapest credibility there is: no cherry-picked gallery, no "results may vary" fine print — the tool's real output is embedded right here, receipts and all. If the model had mangled the motion or lost the fox between edits, you'd be looking at the evidence right now.

*This is a third-party community project, not affiliated with or endorsed by Anthropic or Google. Bring your own Gemini API key — and remember video generations are billable and slow, so nail the prompt, batch your edits, and save the YouTube upload for the final cut.*