{"slug": "ai-in-action-gemini-agentic-video-4-hidden-prerequisites-and-my-line-bot-process", "title": "[AI in Action] Gemini Agentic Video: 4 Hidden Prerequisites and My LINE Bot Integration Process", "summary": "A developer integrated Gemini's agentic video mode into a LINE Bot running on Cloud Run and Vertex AI, enabling timestamped question-answering over long videos such as a two-hour Google I/O '25 keynote. Testing revealed four silent prerequisites — api_version=\"v1beta1\", media_processing=\"AGENTIC\", a supported model, and a thinking_level setting — that must all be present or the API quietly falls back to static video processing while still returning 200 responses. Google's announcement cited 88% fewer tokens, 66% lower cost, and 7% higher accuracy for long-form video.", "body_md": "I have a LINE Bot that I use every day, [linebot-helper-python](https://github.com/kkdai/linebot-helper-python). If you send it a URL, it returns a summary and social media copy for four platforms; if you send a YouTube link, it returns a video summary. It also handles bookmarks, location queries, voice assistants, and more. It runs on Cloud Run and uses Vertex AI.\n\nIn late August, Google published a post [Introducing agentic video in Gemini](https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-agentic-video-in-gemini/). After reading it, my first thought was: this could allow my bot to do something it currently can't—**ask questions about a video**, rather than just providing a summary.\n\nAs it turned out, the parts not mentioned in the announcement were more worth documenting than the parts that were.\n\nOriginally, Gemini's video reading was \"static\": no matter what you asked, it would cram every frame and every audio segment into the context at a fixed sampling rate. For a two-hour video, the video itself could take up hundreds of thousands of tokens, even if you just wanted to ask, \"Did he mention pricing?\"\n\nThe Agentic mode flips this: it lets the model decide which segments to load. It first scans the transcript, determines that the answer might be around 1:24, and then only pulls in the frames for that specific segment.\n\nThe official announcement cited figures for long-form video scenarios: 88% fewer tokens, 66% lower costs, and 7% higher accuracy. To enable it, you just add a parameter to the Part:\n\n```\nvideo_part = types.Part(\n    file_data=types.FileData(file_uri=..., mime_type=\"video/mp4\"),\n    media_processing=\"AGENTIC\", # or \"STATIC\"\n)\n```\n\nThree models are supported: `gemini-3.7-flash`, `gemini-3.6-flash`, and `gemini-3.5-flash-lite`. Input sources can be YouTube URLs, Cloud Storage URIs, or embedded base64.\n\nI tested it with the two-hour Google I/O ‘25 keynote, asking \"Did he mention pricing? If so, please give me the timestamp.\" The response was:\n\nThe video mentions information related to pricing and subscription plans. It mainly appears around **1:24:40 to 1:26:10** in the video...\n\nIt then accurately described the differences between the Google AI Pro and Google AI Ultra plans. When I asked where the Android XR glasses segment was, it replied `1:36:31 to 1:50:30`, and the content matched perfectly.\n\nThis ability to precisely locate information within a two-hour video is where I think the real value of this feature lies. Anyone can do summaries, but \"when in this long video did he talk about X\" is something few tools currently do well.\n\nThis was the most expensive lesson this time, so I'm putting it first.\n\nTo make agentic video actually work on Vertex AI, **four things must be true simultaneously**:\n\n| # | Condition | What happens if missing | \n|---|---|---|\n| 1 | `api_version=\"v1beta1\"` | Agentic is only available in v1beta1; using `v1` silently falls back to STATIC | \n| 2 | `media_processing=\"AGENTIC\"` | Defaults to STATIC; if not set, it's not enabled | \n| 3 | Model is on the supported list | Unsupported models silently downgrade to STATIC | \n| 4 | `thinking_level` is set | See the next section; this is the trickiest one | \n\nThe keyword is **silent**. If any of these are missing, the API will return a 200, the answer will still be generated, and everything will look perfectly normal—except for the bill. No error message will tell you that agentic mode didn't take effect.\n\nThere are also two environmental prerequisites:\n\n`media_processing` field was added in `google-genai` My approach was to write a test for each of these four items, asserting the parameters sent to the SDK:\n\n``` python\ndef test_thinking_level_is_low(captured):\n    youtube_tool.summarize_youtube_video(VIDEO_URL)\n    config = captured[\"call_kwargs\"][\"config\"]\n    assert config.thinking_config is not None\n    assert str(config.thinking_config.thinking_level).upper().endswith(\"LOW\")\n```\n\nThe only reason these four tests exist is that \"if these four things are wrong, the program still runs, the answer still comes out, but the cost silently changes.\" This kind of bug cannot be caught by manual review.\n\nMy original design was this: the user pastes a link to get a summary, clicks \"Ask about this video\" to enter Q&A mode, and then every subsequent follow-up question uses the context from the previous round, avoiding the need to re-process the video.\n\nThe official documentation indeed says this: the response will carry `tool_call` / `tool_response` parts; put them back into the history as-is, and the next round won't need to re-process the video.\n\nIn practice, I couldn't get them.\n\nThe parts returned by Vertex only contained a bare `thought_signature`, with no `tool_call` or `tool_response`. Passing the history back resulted in:\n\n```\n400 INVALID_ARGUMENT: Invalid thought signature.\n```\n\nI tried three different serialization methods, all failed. Then I tried **not serializing at all and passing the response object back directly**, which also failed. So the problem wasn't my serialization; it was the platform.\n\nAfter setting `thinking_level` to `LOW`, the error stopped, but the `tool_use_prompt_token_count` for the second round was **2,163**, exactly the same as the **2,163** in the first round. The video was completely re-processed; no context was preserved.\n\n**Cause and Solution**: That multi-turn mechanism was written for the Gemini Developer API. Since it can't be preserved on Vertex, don't pretend it can: switch to a stateless re-query approach where every question is an independent call.\n\nThis decision actually made the implementation simpler: no need to serialize history, no need to handle thought signatures, and the session only needs to store three fields: \"which video is this user currently asking about.\" The complex version I originally planned was scrapped.\n\nBy the way, I was originally worried that \"history serialization might exceed the Firestore single-document 1 MiB limit.\" In reality, it was only 1.5–7 KiB, so **it was never an issue**. The things you spend time worrying about are often not where the real problems occur.\n\nI had to correct my causal inference three times to get this one right, and the process was more interesting than the conclusion.\n\nInitial spike, same 10-minute video:\n\n| Mode | in | out (incl. thinking) | Cost | \n|---|---|---|---|\n| STATIC | 54,546 | 291 | $0.0171 | \n| AGENTIC, no thinking_level set | 2,216 | 37,574 | $0.0946 | \n| AGENTIC + `thinking_level=\"LOW\"` | 2,216 | 638 | $0.0023 | \n\nIt seemed clear: agentic mode removed the video from the input (54,546 → 2,216), but the model used \"thinking\" to navigate, and **thinking tokens are billed as output** ($2.50/M, which is 8.3x the input price). So, not setting `thinking_level` was 5.5x more expensive than static, but setting it made it 7.4x cheaper.\n\nThen I ran a two-hour video, also with `LOW` set, and the thinking tokens were **359,961**, costing $0.91 for a single call.\n\nMy conclusion: \"On long videos, `thinking_level` is ignored.\"\n\nDuring the implementation phase, the agent responsible for that task ran a validation as I requested and hit a threshold, stopping to report: for the same video and the same settings, the thinking tokens for five consecutive runs were:\n\n```\n0, 0, 34911, 35122, 37410\n```\n\nIt even pulled the DEBUG logs to confirm that every request sent indeed carried `v1beta1` + `AGENTIC` + `thinking_level=LOW`, so it wasn't a client-side omission.\n\nSo it wasn't the video length—**I had run each setting only once and mistaken sampling noise for causality**.\n\nThe agent proposed a hypothesis: it might be related to prompt complexity, because the previous batch used a one-sentence short prompt, while the problematic batch used the official long prompt. Reasonable, so I had it re-test with the official prompt.\n\n27 calls, 6 times for each of the three settings plus 3 times for the Q&A path:\n\n| Setting | thinking peak | tool_use>0 | Average Cost | \n|---|---|---|---|\n| `thinking_level=\"LOW\"` | 0/6 | ✓ | $0.00146 | \n| `thinking_budget=0` | 0/6 | ✓ | $0.00137 | \n| Not set at all | 0/6 | ✓ | $0.00137 | \n| `STATIC` | 0/6 | **✗ (=0)** | $0.01659 | \n\nAll zero peaks. But the previous batch using the **same official prompt** had peaks in 3 out of 4 runs.\n\nSo prompt complexity wasn't it either. The behavior of the three client-side settings was identical; the only difference was **when they were called**.\n\n**Cause and Solution**: This is server-side non-determinism, and the client has no leverage to control it. I also confirmed two things: `thinking_budget` and `thinking_level` cannot be used together (the server returns a 400); `STATIC` is the only deterministic option, but its `tool_use_tokens` is 0. That's not \"the same feature in a stable mode,\" it's turning off agentic mode entirely.\n\nMy decision was to keep `thinking_level=\"LOW\"` and add a warning log:\n\n```\n# In testing, thinking tokens only fall into two groups: ~0 or ~35,000-37,000, with no values in between.\n# The cause is server-side non-determinism, which no client setting can prevent.\nTHINKING_TOKENS_WARN_THRESHOLD = 5000\n```\n\nI kept LOW not because it's more stable (it isn't), but because there was no measurable difference between the three settings, and changing it would mean swapping verified behavior for unverified behavior. The truly valuable reinforcement was turning that 60x cost event from something invisible into something searchable in Cloud Run logs. If you can't stop it, at least make it visible.\n\nSo the honest cost picture is: **normally about $0.0014 per call, with unpredictable and unpreventable peaks about 60x higher**. The \"66% cost reduction\" mentioned in the announcement holds true when there's no peak, but flips when there is.\n\nThe `thinking_level` parameter in my implementation is **required, with no default value**:\n\n``` php\ndef _generate_video(youtube_url: str, prompt: str, *, thinking_level: str) -> dict:\n```\n\nGiving it a default value is just an invitation for someone to omit it, and omitting it leaves no trace.\n\nThe flow is as follows:\n\n```\nUser pastes a YouTube link\n  → Receives summary + social media copy (existing feature)\n  → A new button appears: \"🎬 Ask about this video\"\n  → After clicking, the user types a question: \"Did he mention pricing?\"\n  → Answer comes back with timestamps\n  → User pastes a new URL → Automatically exits video mode\n```\n\nThree new modules, each independently testable:\n\n`tools/youtube_tool.py` (modified) — The only place communicating with the Gemini video API, with the four required parameters centralized in one function.`services/video_qa.py` (new) — Remembers \"User → Video\" mapping, with a 30-minute TTL.`services/usage_meter.py` (new) — Records tokens and costs for each call.\n`main.py` only adds two integration points: a message interceptor and a postback branch.\n\nWhen a user types in video mode, the decision order is:\n\n```\nExit condition check → Quota check → Call Gemini\n```\n\n**The exit check must come before the quota check.** If a user pastes a new URL to change the topic, they shouldn't be blocked because their video Q&A quota is exhausted. Those are two unrelated things. If the order is reversed, a person could get stuck in a mode where every message is rejected and they can't get out.\n\nThis rule has its own dedicated test because it's a contract, not an implementation detail.\n\nTo determine if a user wants to leave video mode, I used the simplest method: if the message contains a URL or starts with `/`, exit.\n\nI deliberately avoided using an LLM to judge intent because that would require an extra Gemini call for every message, and the cost of a false positive is low (the user just asks again). It's not worth the cost and latency.\n\nThere's a detail worth noting in the implementation: this check directly calls the same `find_url()` used by the main path. Initially, I wrote my own regex, but during review, someone suggested broadening it to support URLs without schemes (`www.youtube.com/...`). I checked what the main path actually used and found its regex couldn't catch those either: **broadening it unilaterally would only create a discrepancy**: video mode would exit, but the main path wouldn't treat it as a URL, leaving the user with a useless chat reply instead of a useless video reply.\n\nBy switching to the shared function, the two are always consistent; if `find_url` is broadened in the future, video mode will automatically follow suit.\n\nFor this implementation, I used Claude Code's subagents: a plan split into 8 tasks, with each task assigned to a fresh agent for implementation, followed by another agent for review. I only acted as the coordinator and arbitrator.\n\nThe benefit of this arrangement is that **the reviewer doesn't have the implementer's attachment**. An agent that just wrote the code is likely to think it's correct; a reviewer agent only sees the diff, the requirements, and the report, without the baggage of \"I thought about this for a long time.\"\n\nThe things they caught were far more valuable than I expected.\n\nAs mentioned in the thinking token section, the first correction happened when the implementation agent hit the threshold I set and stopped to report (I wrote \"stop if you see over 30,000, don't just commit silently\" in the instructions, and it followed them). The second was when it proposed the prompt complexity hypothesis and designed an experiment to rule it out.\n\nThe third was the most interesting. While updating the documentation, I replaced the debunked theory with \"`gemini-3.5-flash-lite` is the only model with stable costs.\" The reviewer agent pointed out that the 27 non-deterministic experiments **were run on 3.5-flash-lite**, and the peaks were measured on it, so that sentence directly contradicted the paragraph it was citing.\n\nI had replaced one baseless claim with another baseless claim, and I did it three times (in the spec, the config file comments, and the test docstrings).\n\nThe final branch review caught one thing: **the entry button for the entire feature wouldn't even show up**.\n\nThe button was attached to a carousel message, but four text messages were appended afterward. LINE only renders the quickReply of the **last message**, so the button was buried. A user pasting a YouTube link would receive the summary and four pieces of copy, but see no button.\n\nEight rounds of task reviews had passed it because each round only saw \"the button is correctly attached to the carousel\"; no one saw that four more messages would be added later. The tests missed it too—those tests called the handler directly, skipping the message assembly part.\n\n**This is something a single-task review is structurally unable to see**; it only becomes visible when the entire path is laid out.\n\nAfter fixing it, I verified it again with mutation testing: moving the attachment back to the original position caused only the new multi-URL test to fail, while the original single-URL test still passed, which is exactly why it couldn't catch the bug.\n\nNo matter how good the process is, it can't stop me from writing the wrong things in the instructions:\n\n`UnboundLocalError` where a variable was only assigned in one branch. The implementation agent corrected it.\nIt's a bit embarrassing to write down, but these are exactly the real outputs of this process: **every single error was caught before merging**.\n\n|  | Figures | \n|---|---|\n| Commits | 18 | \n| Tests | 192 → 275 passed | \n| Converged model literals | 28 places | \n| Fixed online failures | 1 (Smart Dialog 404) | \n| Actual cost | Approx. $6–8 (mostly on three rounds of cost measurement) | \n\nI also completed an item that had been on the roadmap for a long time: token and cost logging. This was originally \"do it when there's time,\" but after the thinking token pitfall, it became a necessary component: since peaks can't be prevented, you must at least be able to see that they happened after the fact.\n\n**Check constraints first, then design.** This time, almost every design decision was forced by constraints: Vertex not preserving context forced stateless queries, uncontrollable thinking tokens forced warning logs, and the inability to use `thinking_budget` and `thinking_level` together scrapped an entire option. It's much easier to check constraints thoroughly before starting than to design first and hit a wall later.\n\n**A single observation is not a conclusion.** My most expensive mistake was taking one observation, making a causal judgment, and then writing that judgment into the design document, where it spread to code comments, environment variable descriptions, and decision records. By the time I realized it was wrong, it took three rounds just to clean up the residue—and I even managed to generate new incorrect theories in the middle. Overturning a conclusion is much more work than establishing one.\n\n**Tests guard bug classes, not strings.** When the `gemini-3-pro-preview` deprecation broke a flagship feature, if I had just changed the model name, it would have happened again next time. By changing it to \"no preview models allowed,\" that entire class of problem was truly blocked.\n\n**Silent failures deserve dedicated tests.** For those four required parameters, if any one is wrong, the program still runs and the answer still comes out. This won't show up in error logs or be caught by code reviews; only the bill will tell you—and by the time the bill tells you, a month has usually passed.\n\nThe code is at [kkdai/linebot-helper-python](https://github.com/kkdai/linebot-helper-python), and this implementation is in PR #20. The official documentation is [Video understanding](https://ai.google.dev/gemini-api/docs/video-understanding), but that version is for the Gemini Developer API; if you are using Vertex AI like me, treat the multi-turn conversation section as a reference, not a guarantee.", "url": "https://wpnews.pro/news/ai-in-action-gemini-agentic-video-4-hidden-prerequisites-and-my-line-bot-process", "canonical_source": "https://dev.to/gde/ai-in-action-gemini-agentic-video-4-hidden-prerequisites-and-my-line-bot-integration-process-582f", "published_at": "2026-09-26 17:26:14+00:00", "updated_at": "2026-09-26 17:58:54.015162+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "generative-ai", "ai-tools"], "entities": ["Google", "Gemini", "Vertex AI", "Cloud Run", "LINE", "google-genai", "Google I/O"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-in-action-gemini-agentic-video-4-hidden-prerequisites-and-my-line-bot-process", "markdown": "https://wpnews.pro/news/ai-in-action-gemini-agentic-video-4-hidden-prerequisites-and-my-line-bot-process.md", "text": "https://wpnews.pro/news/ai-in-action-gemini-agentic-video-4-hidden-prerequisites-and-my-line-bot-process.txt", "jsonld": "https://wpnews.pro/news/ai-in-action-gemini-agentic-video-4-hidden-prerequisites-and-my-line-bot-process.jsonld"}}