{"slug": "dev-log-python-create-short-videos-from-photos-and-clips-with-gemini-3-7-flash", "title": "[Dev Log][Python] Create short videos from photos and clips with Gemini 3.7 Flash: ReelCraft", "summary": "A developer built ReelCraft, a Python CLI that uses Gemini 3.7 Flash to automatically edit photos and videos into short vertical videos. The tool analyzes each asset individually, generates an editing plan, and uses ffmpeg to render the final video with background music and subtitles. The developer clarified that Gemini Omni Flash is not suitable for multi-video understanding, so the pipeline uses standard Gemini models for analysis and planning.", "body_md": "It all started with a misunderstanding.\n\nI noticed a new page in the Gemini API documentation called [Omni](https://ai.google.dev/gemini-api/docs/omni), introducing a model named Gemini Omni Flash, described as \"natively multimodal, processing text, images, audio, and video simultaneously.\" My first thought was straightforward: if I throw a whole folder of videos and photos from my phone into it, let it understand what each asset is about, and then tell it in one sentence to edit them into a short video—isn't that a video editing app?\n\nAfter reading the documentation, I realized I had misunderstood, and the misunderstanding happened to be at the most critical point. However, after bypassing that limitation, the rest was actually feasible. The result is [ReelCraft](https://github.com/kkdai/reelcraft): a Python CLI where you feed in a bunch of videos and photos, Gemini 3.7 Flash understands the assets one by one and provides editing suggestions. Once I confirm the edit list, `ffmpeg`\n\ncuts it into a 9:16 vertical short video, background music is generated using Lyria 3, and subtitles are automatically burned in.\n\nAlong the way, there were three issues where both `ffmpeg`\n\nand Gemini reported success, but the output was wrong—the kind of errors you only discover by actually playing the video.\n\nThis article will cover:\n\nGemini Omni Flash (`gemini-omni-flash-preview`\n\n) is a video generation and editing model that uses the Interactions API. It allows you to use natural language to apply effects to a single video, such as \"when the person touches the mirror, make the mirror ripple beautifully like liquid.\" It is not a tool for \"understanding a bunch of videos.\"\n\nThe limitation section states clearly:\n\nReferencing or reasoning across multiple videos is not supported. Attempting multi-video prompting may result in degraded model performance or unexpected outputs.\n\nAdditionally:\n\nVideo references up to 3 seconds in duration are accepted by the API schema but are not correctly processed by the model at this time.\n\nSo the path of \"throwing a bunch of videos in and letting it understand and edit them\" was blocked for Omni Flash. The models that can actually perform multi-video understanding are the standard Gemini models: starting from version 2.5, a single request can include up to 10 videos. With a 1M context window, it can handle about an hour of footage at default resolution, tokenize it second-by-second, and output scene descriptions with timestamps.\n\nThe time spent on this misunderstanding wasn't wasted. The verification process helped clarify \"which task should be handled by which model,\" and the architecture followed naturally.\n\nThe entire pipeline is split into five stages, with states stored in files:\n\n```\n[Asset Folder]\n     │ poc ingest: Scan videos/photos → catalog.json\n     ▼\n     │ poc analyze: Call Gemini for each file individually → analysis/*.json\n     ▼\n     │ poc plan: Aggregate all analysis results, call once for editing suggestions\n     ▼ → summary.md (for humans) + edl.yaml (for machine execution)\n     ⏸ Human inspection and editing of edl.yaml\n     ▼\n     │ poc render: ffmpeg editing, 9:16 cropping, xfade transitions\n     ▼\noutput/final.mp4\n```\n\nThe key design decision is in the second and third steps: call Gemini once for each video to get precise internal timestamps and descriptions; then feed these text results (not the raw videos) into a second call for cross-asset aggregation, sequencing, and editing suggestions.\n\nThis approach has two benefits. First, it completely avoids the \"multi-video reasoning not supported\" issue because the second call only sees text, not ten videos. Second, it isn't limited by the 10 videos/request cap; no matter how many assets there are, it just means more independent calls in the `analyze`\n\nphase. Those calls can be retried or fail individually without affecting each other.\n\nTesting also proved that timestamps are more reliable when processed separately. When asking about ten videos in a single prompt (\"which seconds are the highlights?\"), the model easily confuses the timelines of different videos.\n\nFailure handling in the `analyze`\n\nphase is recorded separately: if a file fails after three retries, it's logged in `analysis/_errors.json`\n\n, while other files continue. This later revealed a loophole during review, which I'll discuss later.\n\nI decided from the start not to make it \"one-click fully automatic.\" Between inputting assets and outputting the final product, there must be a place where I can manually intervene, because LLM-provided edit points will inevitably have some irrationalities, and re-running the entire pipeline incurs API costs again.\n\nThat interface is a YAML file:\n\n```\ntarget_duration_sec: 23\naspect_ratio: '9:16'\nclips:\n- source: /abs/path/808327978.mp4\n  note: Opening shot: Showing the COSCUP x UbuCon Asia main visual backdrop.\n  in: '00:00.000'\n  out: '00:02.500'\n- source: /abs/path/S__1908753.jpg\n  note: Fun venue easter egg: Creative semiconductor chip snacks distributed on-site.\n  duration_sec: 4.0\ntransitions: crossfade 0.3s\nmood_tags: [Professional, Joyful, Community Cohesion]\n```\n\nVideos use `in`\n\n/`out`\n\nto mark the range, photos use `duration_sec`\n\nfor duration, and `note`\n\nis the reason for selection written by Gemini (this field was later used for subtitles, see below). To change an edit point, just change the numbers; to change the order, move the clip; after saving, run `poc render`\n\n.\n\nThe outputs of each stage remain in the project directory, so any step can be re-run individually. `analyze`\n\nalso skips files that already have analysis results, so re-running doesn't incur double charges—this is very helpful when iterating on prompts.\n\n`poc plan --theme`\n\nwas added later: you can provide a sentence as the editing theme, e.g., `--theme \"Participating in the COSCUP open source community\"`\n\n. This affects the narrative angle of the summary, the priority of clip selection, and the wording of each clip's `note`\n\n. Since it only affects the `plan`\n\nstage, changing the theme doesn't require re-analyzing assets, making it very cheap to try different narratives on the same set of materials.\n\nThe understanding and aggregation stages initially used `gemini-2.5-flash`\n\n, then switched to [ gemini-3.7-flash](https://ai.google.dev/gemini-api/docs/models/gemini-3.7-flash). This is the GA stable version, not a preview:\n\n| Item | Specification |\n|---|---|\n| Model ID | `gemini-3.7-flash` |\n| Input | 1,048,576 tokens |\n| Output | 65,536 tokens |\n| Input Types | Text, Image, Video, Audio, PDF |\n| Capabilities | structured outputs, function calling, caching, thinking (low/medium/high) |\n| Not Supported | Video/Image/Audio generation, Live API |\n\nFor this project, the most important features are structured outputs and video input, as the `analyze`\n\nstage involves feeding in a video and requesting a JSON with a fixed schema.\n\nAfter switching, I didn't just change the string and call it a day; I verified it with actual API calls, running `analyze_file`\n\non real assets. For the same lecture video, the difference in descriptions between the two models was quite noticeable.\n\n`gemini-2.5-flash`\n\nversion:\n\nAt the start of the video, a woman on stage uses a microphone to introduce herself to the audience. The large screen behind her shows her name \"Zona Wang\" and her job description.\n\n`gemini-3.7-flash`\n\nversion:\n\nIn the video, a female speaker (Zona Wang, LINE Technology Evangelist) is giving a self-introduction and presentation on a stage in a lecture hall, followed by a camera pan across the audience listening intently.\n\nThe difference lies in \"job description\" vs. \"LINE Technology Evangelist.\" The latter actually read the small text on the slide, while the former only knew there was some job information there.\n\nThe gap in the aggregation stage was even larger. For the same set of COSCUP assets and the same `--theme`\n\n, 2.5's summary was: \"This short video aims to showcase the vitality and diversity of the COSCUP open source community. From professional knowledge sharing and deep technical exchange to warm interaction and inclusion among community members\"—the whole thing stayed at an abstract level. 3.7 recognized the full event name \"COSCUP x UbuCon Asia,\" booth names like \"FOSS for All\" and \"Kubernetes,\" and even described a photo as \"Creative semiconductor chip snacks distributed on-site.\" These details weren't in my prompt; they all came from the text and objects in the photos.\n\nFor an application where \"asset understanding quality directly determines editing quality,\" the benefit of switching models was greater than I expected. The editing suggestions improved because it actually understood more, not because the prompt was written better.\n\nBy the way, 3.7's `note`\n\nstyle also changed to a \"Short Label: Detailed Description\" format. This change later broke all my subtitles, as discussed below.\n\nBackground music is generated using [Lyria 3](https://ai.google.dev/gemini-api/docs/music-generation). There are two models: `lyria-3-clip-preview`\n\nfor 30-second clips, and `lyria-3-pro-preview`\n\nfor full songs. My output is about 20 seconds, so the clip version is perfect.\n\nIt doesn't require a separate Vertex AI application or allowlisting; the same Gemini API key works. However, the calling method is completely different from `generate_content`\n\n, using `client.interactions.create()`\n\n:\n\n```\ninteraction = client.interactions.create(\n    model=\"lyria-3-clip-preview\",\n    input=\"An instrumental background music track for a short social-media video, \"\n          \"about 20 seconds long. Mood: Professional, Joyful, Community Cohesion, Happy. \"\n          \"No vocals, no lyrics, loopable.\",\n)\naudio_bytes = base64.b64decode(interaction.output_audio.data)\n```\n\nSeveral things were different from what I imagined.\n\nIt has no structured parameters. Length, BPM, genre, and mood must all be written in the natural language prompt, rather than passing a field like `bpm=120`\n\n. So the `generate_score(mood_tags, duration_sec)`\n\nfunction's job is actually to concatenate mood tags and seconds into an English sentence. Mood tags are aggregated from asset analysis results during the `plan`\n\nstage, and `poc render --mood \"Happy, Joyful, Celebration\"`\n\ncan further overlay desired directions.\n\nIt is single-turn generation and cannot be iteratively modified. Unlike Omni Flash's video editing, once the music is generated, it's set; if you're not satisfied, you have to submit a new prompt. All generated audio includes a SynthID watermark.\n\nWhen the music is shorter than the video, you have to handle it yourself. The clip version is max 30 seconds, but the video might be longer. So during mixing, I use `-stream_loop -1`\n\nto loop the audio infinitely and `-shortest`\n\nto trim it to the video length:\n\n```\ncmd.extend([\"-stream_loop\", \"-1\", \"-i\", str(audio_path)])\n# ... filter_complex, map video ...\ncmd.extend([\"-map\", f\"{audio_index}:a\", \"-c:a\", \"aac\", \"-b:a\", \"128k\", \"-shortest\"])\n```\n\nMusic generation failure (quota, network, safety filters) won't crash the entire render; it prints a warning and falls back to silent output. This principle was later added to the project's `CLAUDE.md`\n\n: any value-added feature calling an external generative API must degrade gracefully and not let the main process die because of a secondary feature.\n\nThe render stage uses ffmpeg's `xfade`\n\nfilter to connect clips. Each `xfade`\n\nrequires an `offset`\n\nparameter, which is \"at which second in the output timeline to start this transition.\" The logic for accumulation is: the sum of all previous clip lengths minus the seconds overlapped by each transition.\n\nAfter writing the first version, unit tests were all green, and real assets produced normal videos. Then review identified two scenarios where ffmpeg returns exit code 0, but the output file is wrong.\n\nScenario one: The transition is longer than the clip, causing the clip to be silently swallowed. For two 1-second clips with `transitions: \"crossfade 2s\"`\n\n, the calculated offset is `-1.000`\n\n. ffmpeg accepts this negative number, doesn't report an error, and finishes normally. The output is a 1-second video containing only the first clip; the second one disappears entirely. Since `EDL.transitions`\n\nis a free-text field, it's entirely possible for me to type `3s`\n\ninstead of `0.3s`\n\nwhen manually editing the YAML, and it won't tell me in any way.\n\nScenario two: `out`\n\nexceeds the actual asset length, causing everything following it to be truncated. For a 10-second video, if the EDL says `in: 8.0`\n\n/ `out: 15.0`\n\n, only 2 seconds can actually be taken. If a 1.5-second photo follows, the offset is calculated as 6.700, which falls after the end of the first stream. The result is a 2-second output where the photo is completely missing, and the exit code is still 0. This scenario is even more important to prevent because the EDL is generated by an LLM, and hallucinating an out-of-bounds end time is quite natural.\n\nI added explicit checks for both: if a negative offset is calculated, a `ValueError`\n\nis thrown specifying which clip and transition length; before rendering, `ffprobe`\n\nis used to read the actual length of each video asset, and if `out`\n\nexceeds it, an error is reported clearly stating the requested vs. actual duration.\n\nI care so much because a \"successful\" but incorrect output is much worse than a crash. If it crashes, I know to fix it immediately. With exit code 0 and a seemingly normal mp4, I might not notice until I watch the whole video and think \"wait, a segment is missing,\" and then have no idea where to start investigating.\n\nThe source for subtitles is the `note`\n\nfor each clip in the EDL—the editing reason written by Gemini. Since it already wrote a description for each segment, using it as an on-screen title is perfect.\n\nThe implementation doesn't use `drawtext`\n\n; instead, it generates an SRT file and burns it in using libass's `subtitles`\n\nfilter. The reason is that `drawtext`\n\nrequires manual handling of Chinese font paths and escaping characters; colons, commas, and single quotes all clash with filtergraph syntax. SRT with `force_style`\n\nis much cleaner, and specifying `FontName=Noto Sans TC`\n\nlets fontconfig find the Chinese font.\n\nThe first issue was two subtitles appearing on screen simultaneously. In the first version, each subtitle's display interval was just the clip's own start and end times. But with a 0.3s crossfade overlap between adjacent clips, those 0.3 seconds would have two lines of white text on a black background stacked together, which looked ugly. The fix was to change each subtitle's end time to \"when the next clip starts\" rather than its own end time, ensuring at most one subtitle is visible at any moment. Unit tests couldn't catch this because the SRT was perfectly valid and ffmpeg burned it successfully; I only found it by looking at the frames.\n\nThe second issue was subtitles all trailing with an ellipsis. The `note`\n\nis a full sentence description, which would fill the screen if burned directly, so it's truncated into a short title: cut at the first comma or period, or use a character limit if no punctuation is found, adding \"...\" if truncated.\n\nAfter switching to Gemini 3.7 Flash, this rule fell apart. 3.7 tends to write notes in a \"Opening shot: Showing the 2024 COSCUP x UbuCon Asia main visual backdrop\" format—a \"Short Label: Detailed Description\" style. Since colons weren't in my sentence-breaking character list, the whole sentence fell into the character-limit truncation path, and all eight subtitles ended with \"...\".\n\nHard truncation had a second flaw: it ignored word boundaries. \"Presenting the female speaker sharing presentation content about ChatGPT and Antigravity\" cut at the 20th character resulted in \"...and An...\", a halved English word.\n\nI fixed both: colons are now treated as label separators, and the label itself is used as the full title without an ellipsis; when hard truncation is necessary, if the cut point falls in the middle of a continuous string of English letters/numbers, it backtracks to before that string started, discarding the whole segment rather than cutting it in half. I also relaxed the character limit from 20 to 24.\n\nAfter re-burning, the eight subtitles became clean short titles like \"Opening Shot,\" \"Session Hall Live,\" \"Technical Sharing Close-up,\" \"Venue Easter Egg,\" and \"Community Booth Interaction,\" without a single ellipsis.\n\n`files.upload()`\n\nreturning doesn't mean the file is ready. This was caught by digging into the SDK source code during review, and it would crash on the real API while never showing up in tests. `client.files.upload()`\n\nreturns as soon as the bytes are transferred, without waiting for server-side processing. After a video is uploaded, it stays in a `PROCESSING`\n\nstate for several seconds; trying to use it for `generate_content`\n\nduring this time results in a 400 `FAILED_PRECONDITION`\n\n.\n\nWorse, my original retry loop made things worse: `analyze_file`\n\nwas wrapped in a retry, so each retry re-uploaded the entire video and immediately failed again, with only about 3 seconds of backoff across three tries. After three tries, the asset went into `_errors.json`\n\n, and the plan stage didn't read that file at the time, so the asset silently disappeared from the final product. The fix was adding a `wait_for_active()`\n\n, polling `client.files.get()`\n\nafter upload until the state is `ACTIVE`\n\nbefore proceeding, and moving the upload out of the retry loop.\n\n`_errors.json`\n\nwas written but not read. As mentioned, `analyze`\n\ndiligently recorded failed assets, but `plan`\n\ndidn't read them, and `summary.md`\n\nwouldn't mention them. The only way a user could notice was by counting the segments in the final product. Now `plan`\n\nattaches the failure list to the end of the summary, explicitly stating which assets were not included.\n\nRe-running ingest can bite you with old analysis results. This was encountered during actual use, not review. I changed the contents of the asset folder, adding new photos and deleting old ones, then re-ran `poc ingest`\n\n. `catalog.json`\n\nwas updated, but analysis results for deleted files were still sitting in `analysis/`\n\n. When `plan`\n\nread the analysis results, it didn't cross-reference them with the current catalog, so it fed outdated assets to Gemini. The model reasonably picked a segment from them, but since the file no longer existed, the whole plan failed. Now `load_analyses()`\n\nfilters by the catalog and prints which outdated records are ignored.\n\nTimestamp precision. `format_timestamp`\n\ninitially used `:04.1f`\n\n, keeping only one decimal place. Every time an EDL went in and out of YAML, it lost up to 0.05 seconds, which is about 1.5 frames at 30fps, causing edit points to drift. I changed it to `:06.3f`\n\nto keep millisecond precision.\n\nLooking back, these problems fall into two categories. `files.upload`\n\nand ffmpeg silent errors were caught by reading the code line-by-line during review. Subtitle overlapping, ellipsis issues, and stale analysis results only surfaced by actually running the code, playing the videos, and trying different sets of assets. When the tests were all green, those three issues were still lurking in the code.\n\nWhat ReelCraft does now is simple: a folder of videos and photos goes in, and a 9:16 short video with music and subtitles comes out, with a YAML file in the middle that I can manually edit.\n\nArchitecturally, what really makes this work is the \"per-file understanding, text aggregation\" split. It was conceived to bypass Omni Flash's lack of multi-video reasoning, but it ended up solving timestamp precision and asset count limits as well. After switching to Gemini 3.7 Flash, the granularity of asset understanding significantly increased, and the editing suggestions improved accordingly—the gains here were greater than what I got from tuning prompts.\n\nTwo areas remain untouched: Omni Flash's single-clip generative touch-up has an empty `touch_up_clip`\n\ninterface, and subtitles are currently derived automatically from `note`\n\n, with the `text_overlays`\n\nfield still empty. Neither music nor subtitles are cached; they are re-generated every time `render`\n\nis run.", "url": "https://wpnews.pro/news/dev-log-python-create-short-videos-from-photos-and-clips-with-gemini-3-7-flash", "canonical_source": "https://dev.to/gde/dev-logpython-create-short-videos-from-photos-and-clips-with-gemini-37-flash-reelcraft-1gc6", "published_at": "2026-08-14 16:55:48+00:00", "updated_at": "2026-08-14 17:05:41.337925+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "developer-tools"], "entities": ["Gemini 3.7 Flash", "Gemini Omni Flash", "ReelCraft", "ffmpeg", "Lyria 3", "Google"], "alternates": {"html": "https://wpnews.pro/news/dev-log-python-create-short-videos-from-photos-and-clips-with-gemini-3-7-flash", "markdown": "https://wpnews.pro/news/dev-log-python-create-short-videos-from-photos-and-clips-with-gemini-3-7-flash.md", "text": "https://wpnews.pro/news/dev-log-python-create-short-videos-from-photos-and-clips-with-gemini-3-7-flash.txt", "jsonld": "https://wpnews.pro/news/dev-log-python-create-short-videos-from-photos-and-clips-with-gemini-3-7-flash.jsonld"}}