[AI in Action] Gemini Agentic Video: 4 Hidden Prerequisites and My LINE Bot Integration Process

A developer integrated Gemini's agentic video mode into a LINE Bot running on Cloud Run and Vertex AI, enabling timestamped question-answering over long videos such as a two-hour Google I/O '25 keynote. Testing revealed four silent prerequisites — api_version="v1beta1", media_processing="AGENTIC", a supported model, and a thinking_level setting — that must all be present or the API quietly falls back to static video processing while still returning 200 responses. Google's announcement cited 88% fewer tokens, 66% lower cost, and 7% higher accuracy for long-form video.

I have a LINE Bot that I use every day, linebot-helper-python https://github.com/kkdai/linebot-helper-python . If you send it a URL, it returns a summary and social media copy for four platforms; if you send a YouTube link, it returns a video summary. It also handles bookmarks, location queries, voice assistants, and more. It runs on Cloud Run and uses Vertex AI. In late August, Google published a post Introducing agentic video in Gemini https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-agentic-video-in-gemini/ . After reading it, my first thought was: this could allow my bot to do something it currently can't— ask questions about a video , rather than just providing a summary. As it turned out, the parts not mentioned in the announcement were more worth documenting than the parts that were. Originally, Gemini's video reading was "static": no matter what you asked, it would cram every frame and every audio segment into the context at a fixed sampling rate. For a two-hour video, the video itself could take up hundreds of thousands of tokens, even if you just wanted to ask, "Did he mention pricing?" The Agentic mode flips this: it lets the model decide which segments to load. It first scans the transcript, determines that the answer might be around 1:24, and then only pulls in the frames for that specific segment. The official announcement cited figures for long-form video scenarios: 88% fewer tokens, 66% lower costs, and 7% higher accuracy. To enable it, you just add a parameter to the Part: video part = types.Part file data=types.FileData file uri=..., mime type="video/mp4" , media processing="AGENTIC", or "STATIC" Three models are supported: gemini-3.7-flash , gemini-3.6-flash , and gemini-3.5-flash-lite . Input sources can be YouTube URLs, Cloud Storage URIs, or embedded base64. I tested it with the two-hour Google I/O ‘25 keynote, asking "Did he mention pricing? If so, please give me the timestamp." The response was: The video mentions information related to pricing and subscription plans. It mainly appears around 1:24:40 to 1:26:10 in the video... It then accurately described the differences between the Google AI Pro and Google AI Ultra plans. When I asked where the Android XR glasses segment was, it replied 1:36:31 to 1:50:30 , and the content matched perfectly. This ability to precisely locate information within a two-hour video is where I think the real value of this feature lies. Anyone can do summaries, but "when in this long video did he talk about X" is something few tools currently do well. This was the most expensive lesson this time, so I'm putting it first. To make agentic video actually work on Vertex AI, four things must be true simultaneously : | | Condition | What happens if missing | |---|---|---| | 1 | api version="v1beta1" | Agentic is only available in v1beta1; using v1 silently falls back to STATIC | | 2 | media processing="AGENTIC" | Defaults to STATIC; if not set, it's not enabled | | 3 | Model is on the supported list | Unsupported models silently downgrade to STATIC | | 4 | thinking level is set | See the next section; this is the trickiest one | The keyword is silent . If any of these are missing, the API will return a 200, the answer will still be generated, and everything will look perfectly normal—except for the bill. No error message will tell you that agentic mode didn't take effect. There are also two environmental prerequisites: media processing field was added in google-genai My approach was to write a test for each of these four items, asserting the parameters sent to the SDK: python def test thinking level is low captured : youtube tool.summarize youtube video VIDEO URL config = captured "call kwargs" "config" assert config.thinking config is not None assert str config.thinking config.thinking level .upper .endswith "LOW" The only reason these four tests exist is that "if these four things are wrong, the program still runs, the answer still comes out, but the cost silently changes." This kind of bug cannot be caught by manual review. My original design was this: the user pastes a link to get a summary, clicks "Ask about this video" to enter Q&A mode, and then every subsequent follow-up question uses the context from the previous round, avoiding the need to re-process the video. The official documentation indeed says this: the response will carry tool call / tool response parts; put them back into the history as-is, and the next round won't need to re-process the video. In practice, I couldn't get them. The parts returned by Vertex only contained a bare thought signature , with no tool call or tool response . Passing the history back resulted in: 400 INVALID ARGUMENT: Invalid thought signature. I tried three different serialization methods, all failed. Then I tried not serializing at all and passing the response object back directly , which also failed. So the problem wasn't my serialization; it was the platform. After setting thinking level to LOW , the error stopped, but the tool use prompt token count for the second round was 2,163 , exactly the same as the 2,163 in the first round. The video was completely re-processed; no context was preserved. Cause and Solution : That multi-turn mechanism was written for the Gemini Developer API. Since it can't be preserved on Vertex, don't pretend it can: switch to a stateless re-query approach where every question is an independent call. This decision actually made the implementation simpler: no need to serialize history, no need to handle thought signatures, and the session only needs to store three fields: "which video is this user currently asking about." The complex version I originally planned was scrapped. By the way, I was originally worried that "history serialization might exceed the Firestore single-document 1 MiB limit." In reality, it was only 1.5–7 KiB, so it was never an issue . The things you spend time worrying about are often not where the real problems occur. I had to correct my causal inference three times to get this one right, and the process was more interesting than the conclusion. Initial spike, same 10-minute video: | Mode | in | out incl. thinking | Cost | |---|---|---|---| | STATIC | 54,546 | 291 | $0.0171 | | AGENTIC, no thinking level set | 2,216 | 37,574 | $0.0946 | | AGENTIC + thinking level="LOW" | 2,216 | 638 | $0.0023 | It seemed clear: agentic mode removed the video from the input 54,546 → 2,216 , but the model used "thinking" to navigate, and thinking tokens are billed as output $2.50/M, which is 8.3x the input price . So, not setting thinking level was 5.5x more expensive than static, but setting it made it 7.4x cheaper. Then I ran a two-hour video, also with LOW set, and the thinking tokens were 359,961 , costing $0.91 for a single call. My conclusion: "On long videos, thinking level is ignored." During the implementation phase, the agent responsible for that task ran a validation as I requested and hit a threshold, stopping to report: for the same video and the same settings, the thinking tokens for five consecutive runs were: 0, 0, 34911, 35122, 37410 It even pulled the DEBUG logs to confirm that every request sent indeed carried v1beta1 + AGENTIC + thinking level=LOW , so it wasn't a client-side omission. So it wasn't the video length— I had run each setting only once and mistaken sampling noise for causality . The agent proposed a hypothesis: it might be related to prompt complexity, because the previous batch used a one-sentence short prompt, while the problematic batch used the official long prompt. Reasonable, so I had it re-test with the official prompt. 27 calls, 6 times for each of the three settings plus 3 times for the Q&A path: | Setting | thinking peak | tool use 0 | Average Cost | |---|---|---|---| | thinking level="LOW" | 0/6 | ✓ | $0.00146 | | thinking budget=0 | 0/6 | ✓ | $0.00137 | | Not set at all | 0/6 | ✓ | $0.00137 | | STATIC | 0/6 | ✗ =0 | $0.01659 | All zero peaks. But the previous batch using the same official prompt had peaks in 3 out of 4 runs. So prompt complexity wasn't it either. The behavior of the three client-side settings was identical; the only difference was when they were called . Cause and Solution : This is server-side non-determinism, and the client has no leverage to control it. I also confirmed two things: thinking budget and thinking level cannot be used together the server returns a 400 ; STATIC is the only deterministic option, but its tool use tokens is 0. That's not "the same feature in a stable mode," it's turning off agentic mode entirely. My decision was to keep thinking level="LOW" and add a warning log: In testing, thinking tokens only fall into two groups: ~0 or ~35,000-37,000, with no values in between. The cause is server-side non-determinism, which no client setting can prevent. THINKING TOKENS WARN THRESHOLD = 5000 I kept LOW not because it's more stable it isn't , but because there was no measurable difference between the three settings, and changing it would mean swapping verified behavior for unverified behavior. The truly valuable reinforcement was turning that 60x cost event from something invisible into something searchable in Cloud Run logs. If you can't stop it, at least make it visible. So the honest cost picture is: normally about $0.0014 per call, with unpredictable and unpreventable peaks about 60x higher . The "66% cost reduction" mentioned in the announcement holds true when there's no peak, but flips when there is. The thinking level parameter in my implementation is required, with no default value : php def generate video youtube url: str, prompt: str, , thinking level: str - dict: Giving it a default value is just an invitation for someone to omit it, and omitting it leaves no trace. The flow is as follows: User pastes a YouTube link → Receives summary + social media copy existing feature → A new button appears: "🎬 Ask about this video" → After clicking, the user types a question: "Did he mention pricing?" → Answer comes back with timestamps → User pastes a new URL → Automatically exits video mode Three new modules, each independently testable: tools/youtube tool.py modified — The only place communicating with the Gemini video API, with the four required parameters centralized in one function. services/video qa.py new — Remembers "User → Video" mapping, with a 30-minute TTL. services/usage meter.py new — Records tokens and costs for each call. main.py only adds two integration points: a message interceptor and a postback branch. When a user types in video mode, the decision order is: Exit condition check → Quota check → Call Gemini The exit check must come before the quota check. If a user pastes a new URL to change the topic, they shouldn't be blocked because their video Q&A quota is exhausted. Those are two unrelated things. If the order is reversed, a person could get stuck in a mode where every message is rejected and they can't get out. This rule has its own dedicated test because it's a contract, not an implementation detail. To determine if a user wants to leave video mode, I used the simplest method: if the message contains a URL or starts with / , exit. I deliberately avoided using an LLM to judge intent because that would require an extra Gemini call for every message, and the cost of a false positive is low the user just asks again . It's not worth the cost and latency. There's a detail worth noting in the implementation: this check directly calls the same find url used by the main path. Initially, I wrote my own regex, but during review, someone suggested broadening it to support URLs without schemes www.youtube.com/... . I checked what the main path actually used and found its regex couldn't catch those either: broadening it unilaterally would only create a discrepancy : video mode would exit, but the main path wouldn't treat it as a URL, leaving the user with a useless chat reply instead of a useless video reply. By switching to the shared function, the two are always consistent; if find url is broadened in the future, video mode will automatically follow suit. For this implementation, I used Claude Code's subagents: a plan split into 8 tasks, with each task assigned to a fresh agent for implementation, followed by another agent for review. I only acted as the coordinator and arbitrator. The benefit of this arrangement is that the reviewer doesn't have the implementer's attachment . An agent that just wrote the code is likely to think it's correct; a reviewer agent only sees the diff, the requirements, and the report, without the baggage of "I thought about this for a long time." The things they caught were far more valuable than I expected. As mentioned in the thinking token section, the first correction happened when the implementation agent hit the threshold I set and stopped to report I wrote "stop if you see over 30,000, don't just commit silently" in the instructions, and it followed them . The second was when it proposed the prompt complexity hypothesis and designed an experiment to rule it out. The third was the most interesting. While updating the documentation, I replaced the debunked theory with " gemini-3.5-flash-lite is the only model with stable costs." The reviewer agent pointed out that the 27 non-deterministic experiments were run on 3.5-flash-lite , and the peaks were measured on it, so that sentence directly contradicted the paragraph it was citing. I had replaced one baseless claim with another baseless claim, and I did it three times in the spec, the config file comments, and the test docstrings . The final branch review caught one thing: the entry button for the entire feature wouldn't even show up . The button was attached to a carousel message, but four text messages were appended afterward. LINE only renders the quickReply of the last message , so the button was buried. A user pasting a YouTube link would receive the summary and four pieces of copy, but see no button. Eight rounds of task reviews had passed it because each round only saw "the button is correctly attached to the carousel"; no one saw that four more messages would be added later. The tests missed it too—those tests called the handler directly, skipping the message assembly part. This is something a single-task review is structurally unable to see ; it only becomes visible when the entire path is laid out. After fixing it, I verified it again with mutation testing: moving the attachment back to the original position caused only the new multi-URL test to fail, while the original single-URL test still passed, which is exactly why it couldn't catch the bug. No matter how good the process is, it can't stop me from writing the wrong things in the instructions: UnboundLocalError where a variable was only assigned in one branch. The implementation agent corrected it. It's a bit embarrassing to write down, but these are exactly the real outputs of this process: every single error was caught before merging . | | Figures | |---|---| | Commits | 18 | | Tests | 192 → 275 passed | | Converged model literals | 28 places | | Fixed online failures | 1 Smart Dialog 404 | | Actual cost | Approx. $6–8 mostly on three rounds of cost measurement | I also completed an item that had been on the roadmap for a long time: token and cost logging. This was originally "do it when there's time," but after the thinking token pitfall, it became a necessary component: since peaks can't be prevented, you must at least be able to see that they happened after the fact. Check constraints first, then design. This time, almost every design decision was forced by constraints: Vertex not preserving context forced stateless queries, uncontrollable thinking tokens forced warning logs, and the inability to use thinking budget and thinking level together scrapped an entire option. It's much easier to check constraints thoroughly before starting than to design first and hit a wall later. A single observation is not a conclusion. My most expensive mistake was taking one observation, making a causal judgment, and then writing that judgment into the design document, where it spread to code comments, environment variable descriptions, and decision records. By the time I realized it was wrong, it took three rounds just to clean up the residue—and I even managed to generate new incorrect theories in the middle. Overturning a conclusion is much more work than establishing one. Tests guard bug classes, not strings. When the gemini-3-pro-preview deprecation broke a flagship feature, if I had just changed the model name, it would have happened again next time. By changing it to "no preview models allowed," that entire class of problem was truly blocked. Silent failures deserve dedicated tests. For those four required parameters, if any one is wrong, the program still runs and the answer still comes out. This won't show up in error logs or be caught by code reviews; only the bill will tell you—and by the time the bill tells you, a month has usually passed. The code is at kkdai/linebot-helper-python https://github.com/kkdai/linebot-helper-python , and this implementation is in PR 20. The official documentation is Video understanding https://ai.google.dev/gemini-api/docs/video-understanding , but that version is for the Gemini Developer API; if you are using Vertex AI like me, treat the multi-turn conversation section as a reference, not a guarantee.