{"slug": "do-i-still-need-a-monkey-patch-for-gemini-live", "title": "Do I Still Need a Monkey Patch for Gemini Live?", "summary": "A developer has removed a 187-line monkey patch from a biometric security scanner after upgrading to Google ADK 2.6.3, which natively supports Gemini 3.1 Flash Live's realtime media API. The patch was previously required to translate deprecated media_chunks fields, but the framework now handles routing automatically. The project, available on GitHub, streams webcam and microphone data over WebSocket to Gemini for finger-counting and gesture detection.", "body_md": "No. And deleting 187 lines of it was the single biggest benefit of moving to ADK 2.x — but it was not the only one, and it was not the last thing that needed fixing.\n\nThe project is a biometric security scanner, built to exercise the parts of the Gemini Live API that a text chatbot never touches. A browser captures webcam and microphone, streams both to a FastAPI backend over a single WebSocket, and the backend forwards them to Gemini 3.1 Flash Live through the Agent Development Kit. The model watches the video feed, counts the fingers being held up, and calls a tool.\n\nThree tools are registered:\n\n`report_digit(count)`\n\n— the detected finger count, which drives the UI`trigger_system_error()`\n\n— fired on an offensive gesture, which terminates the session`trigger_heavy_metal_mode()`\n\n— fired on the \"Devil's Horns\", a secret overrideThe transport is deliberately plain. Binary WebSocket frames carry a 1-byte type prefix — `1`\n\nfor audio, `2`\n\nfor JPEG — with 16 kHz PCM going up and 24 kHz PCM coming back, played through an AudioWorklet so the main thread stays free. Everything runs locally with `make run`\n\n, or on Cloud Run behind `make deploy`\n\n.\n\n```\ngit clone https://github.com/xbill9/way-back-home\ncd way-back-home/level_3_new\n```\n\nTwo versions live side by side in that repo: `level_3`\n\n, the original design, and `level_3_new`\n\n, the current one. Most of this article is the diff between them.\n\nThe original build ran on `google-adk`\n\n1.27.2, and it worked — but only because a file named `patch_adk.py`\n\nsat next to it, 187 lines long, applied at import time before anything else could run.\n\nThe problem it solved was real. Gemini 3.1 deprecated `media_chunks`\n\n, the field a 1.x ADK used to send realtime media. A 1.x ADK talking to a 3.1 Live model would send the deprecated shape and get nothing useful back. The patch monkey-patched three separate call sites to translate:\n\n```\n# level_3_gemini/backend/app/patch_adk.py\nif hasattr(rt_input, \"media_chunks\") and rt_input.media_chunks:\n    logger.info(\"[PATCH] Unrolling 'media_chunks' from realtime_input.\")\n    for chunk in rt_input.media_chunks:\n        ...\n        await self.send_realtime_input(audio=chunk)\n        ...\n        await self.send_realtime_input(video=chunk)\n```\n\nThe three targets were `live.AsyncSession.send_realtime_input`\n\n, which unrolled `media_chunks`\n\ninto the new typed keywords; `GeminiLlmConnection.send_realtime`\n\n, which routed each blob to `audio=`\n\n, `video=`\n\nor `text=`\n\nby mime type; and `AudioCacheManager.cache_audio`\n\n, which was guarded against a `NoneType`\n\nblob that would otherwise raise.\n\nIt worked, and it was a liability. Monkey patching a framework means every upgrade is a gamble — the patch either becomes redundant, becomes wrong, or silently stops applying because the method it wraps was renamed. The closing recommendation in the original write-up was to delete it the moment the ADK supported the model natively.\n\nThat moment arrived with `google-adk`\n\n2.6.3, and `patch_adk.py`\n\nwas deleted outright. The framework now does the routing itself, detecting the model generation and dispatching on it:\n\n```\n# google/adk/models/gemini_llm_connection.py, send_realtime()\nif isinstance(input, types.Blob):\n  if self._is_gemini_3_x_live or self._is_gemini_3_5_live_translate:\n    if input.mime_type and input.mime_type.startswith('audio/'):\n      await self._gemini_session.send_realtime_input(audio=input)\n    elif input.mime_type and input.mime_type.startswith('image/'):\n      await self._gemini_session.send_realtime_input(video=input)\n    else:\n      logger.warning(\n          'Blob not sent. Unknown or empty mime type for'\n          ' send_realtime_input: %s',\n          input.mime_type,\n      )\n  else:\n    await self._gemini_session.send_realtime_input(media=input)\n```\n\nNote the third branch. Audio and image mime types are dispatched explicitly, and anything else is dropped with a warning rather than guessed at — so a blob sent with a missing or unexpected mime type goes nowhere, and the only evidence is a log line.\n\nText is handled the same way. A single-part text `Content`\n\nis routed to `send_realtime_input(text=...)`\n\nfor 3.x models rather than going out as client content, which matches the Live API's own guidance that `send_client_content`\n\nis only for seeding history.\n\nThat is the whole first patch target and the whole second one, upstream, maintained, and tested by someone else. The third — the `NoneType`\n\nguard on `cache_audio`\n\n— was not carried over, because upstream still calls `len(audio_blob.data)`\n\nunguarded. No path in this application produces a blob with `data=None`\n\n, so it stays deleted rather than being reintroduced as a precaution.\n\nThe patch was hiding a bug in the calling code, and deleting it exposed the bug rather than causing it.\n\n`LiveRequestQueue.send_realtime()`\n\naccepts `types.Blob`\n\nand nothing else. The old patch had used `model_construct`\n\ninternally, which skips Pydantic validation, so passing a bare string worked by accident. Without the patch it raises a `ValidationError`\n\n.\n\nThe call site that matters is the keepalive. This project sends a text stimulus every ten seconds when the client goes quiet, and under 1.x that stimulus was a string handed straight to `send_realtime()`\n\n. Text has to go through `send_content()`\n\ninstead:\n\n``` php\ndef send_text_stimulus(live_request_queue: LiveRequestQueue, text: str) -> None:\n    live_request_queue.send_content(\n        types.Content(role=\"user\", parts=[types.Part(text=text)])\n    )\n```\n\nThis is the removal most likely to take an agent off the air quietly. It does not fail at startup. It fails the first time the keepalive fires, ten seconds into a session that otherwise looks healthy. Anyone migrating a Live agent off 1.x should check that call site before touching anything else.\n\nThe migration was the headline, but it was not the end of the work. Reading the Live API documentation with the source open beside it turned up several things that had been wrong the whole time, none of which any build, test or lint run had ever objected to.\n\n**Video was running at twice the documented maximum.** The capabilities guide is specific:\n\nVideo frames are sent as individual images (e.g., JPEG or PNG) at a specific frame rate (max 1 frame per second).\n\nThe project ran at 2 FPS and permitted up to 5 through an environment variable. Nothing rejects the surplus frames, which is why it went unnoticed — but they are billed, and they consume the session budget twice as fast. `VIDEO_FPS`\n\nnow defaults to 1.0 and is hard-clamped there, so `VIDEO_FPS=3`\n\nyields 1.0 rather than being honoured. A documented limit that is not enforced is how the 2 FPS crept in to begin with.\n\n**Audio-plus-video sessions cap at two minutes.** From the session management guide:\n\naudio-only sessions are limited to 15 minutes, and audio-video sessions are limited to 2 minutes\n\nContext window compression removes the cap entirely. `RunConfig.context_window_compression`\n\ndefaults to `None`\n\n, so it has to be asked for:\n\n```\ncontext_window_compression=types.ContextWindowCompressionConfig(\n    sliding_window=types.SlidingWindow(),\n),\n```\n\nThis application streams both continuously, so it had been on the two-minute clock since the first version. Short test sessions never reached it. A demo where someone works through five gestures does.\n\n**Interruptions were documented and unhandled.** When a user talks over the model, the model stops generating — but the audio it already sent is sitting in the client's ring buffer and keeps playing. The Live API guidance is to stop playback and clear the queue on interruption. ADK surfaces `interrupted`\n\non the event, and because the backend forwards whole events as JSON, the flag was already arriving in the browser with nothing reading it. The clearing machinery already existed too. Three lines connected them.\n\nBoth input and output audio transcription were enabled in `RunConfig`\n\nfrom the very first version of this project. Neither ever produced a line of output.\n\n```\ninput_transcription = getattr(event, \"input_audio_transcription\", None)\nif input_transcription and input_transcription.final_transcript:\n    logger.info(f\"USER TRANSCRIPT: {input_transcription.final_transcript}\")\n```\n\nTwo mistakes are stacked here. `input_audio_transcription`\n\nis the `RunConfig`\n\nfield that *enables* transcription — it is not the field on the event that transcription produces. And `final_transcript`\n\nis not a member of `types.Transcription`\n\nat all; the fields are `text`\n\n, `finished`\n\n, `language_code`\n\n, `speaker_label`\n\nand `words`\n\n.\n\nEither mistake alone raises `AttributeError`\n\nand gets fixed in minutes. Together, behind the default on `getattr`\n\n, they produce silence. The condition evaluates to `None and ...`\n\n, which is falsy, forever.\n\nThe correct field names:\n\n```\ninput_transcription = getattr(event, \"input_transcription\", None)\nif input_transcription and input_transcription.finished:\n    logger.info(f\"USER TRANSCRIPT: {input_transcription.text}\")\n```\n\nGating on `finished`\n\nis deliberate. ADK emits partial transcription events with `finished=False`\n\nand one accumulated event with `finished=True`\n\n, so this produces one clean line per turn instead of one per fragment. The `run_live()`\n\ndocstring is the reference: partial and non-partial events are both yielded to the caller, but only non-partial ones are saved to the session.\n\nThe fix produces this on connect, which is the exact opening line the agent instruction specifies:\n\n```\nINFO - GEMINI TRANSCRIPT: Scanner Online.\n```\n\nThe original design captured frames on a timer:\n\n``` js\nintervalRef.current = setInterval(() => { /* capture, send */ }, 500);\n```\n\nThe rewrite replaced that with `requestAnimationFrame`\n\nand a manual elapsed-time check. On paper it is the better primitive — frame-aligned, idle when the compositor has nothing to do, and paired with `toBlob`\n\ninstead of `toDataURL`\n\nit keeps JPEG encoding off the main thread.\n\nIn a backgrounded tab, `requestAnimationFrame`\n\nis throttled to zero.\n\nThe microphone is not. It runs in an AudioWorklet on the audio thread, which browsers keep alive so capture and playback survive a tab switch. The result is an asymmetric, silent failure: switch tabs and video stops completely while audio streams on. The WebSocket stays open, the session stays billed, and finger detection — the entire point of the application — stops with no error on either side. A 65-second session logged 8,050 audio packets and zero video frames.\n\nTimers are throttled in background tabs as well, but to roughly one second rather than to zero, so the fix is a self-rescheduling timeout:\n\n``` js\nconst captureFrame = () => {\n    if (ws.current?.readyState === WebSocket.OPEN) { /* capture, send */ }\n    if (intervalRef.current !== null) {\n        intervalRef.current = setTimeout(captureFrame, frameIntervalRef.current);\n    }\n};\n```\n\nDegrading to roughly 1 FPS beats stopping. Re-reading the interval on each tick also keeps the server's `config`\n\nframe authoritative, which the rAF version did and a plain `setInterval`\n\nwould not.\n\nFour things were deleted outright, for ninety lines removed and one added.\n\nThe **base64 JSON media path** decoded `type: \"audio\"`\n\nand `type: \"image\"`\n\npayloads out of JSON. It was not speculative — it was the original design's entire wire protocol, orphaned when media moved to binary frames with a type prefix. Its two tests went with it; they were pinning an implementation, not a contract anyone relied on.\n\nThe ** proactivity and affective_dialog query parameters** were declared on the WebSocket endpoint, documented in the docstring, and read by nothing. In the original design they were real, feeding a conditional\n\n`RunConfig`\n\n. Gemini 3.1 Flash Live then shipped without support for either, the config was removed, and the parameters outlived it. The Live API reference lists both under limitations — \"Proactive audio — Not yet supported in Gemini 3.1 Flash Live\" and the same line for affective dialogue — each followed by an instruction to remove any configuration for the feature.A **function-response scan** walked `server_content.model_turn.parts`\n\nlooking for `function_response`\n\n. `model_turn`\n\ncarries model output; a `functionResponse`\n\nis something a client sends. The list was structurally always empty.\n\nA **second notification channel**, `lastMessage`\n\n, was set on every match, system error and heavy-metal trigger, exported from the frontend hook, and read by no component. The callbacks drive the UI.\n\nADK 2.0 moved agents onto a graph engine, and the new constraints fail quietly rather than loudly. Four are worth knowing before writing anything:\n\n`Agent`\n\n.`BaseNode`\n\nsubclasses and `_run_async_impl()`\n\nor `generate_content()`\n\noverrides are `run_live()`\n\nowns the session.`except`\n\ninside a node masks that, and catching `BaseException`\n\nbreaks pausing outright.`run_live(session=...)`\n\nis deprecated.`user_id`\n\nand `session_id`\n\n.This project satisfies all four, which is why the upgrade was uneventful: a plain `Agent`\n\n, `InMemorySessionService`\n\n, no hand-built events, and its broad handlers sitting in the transport layer rather than inside the graph.\n\nOne 2.x addition is worth adopting. `Runner`\n\ngained `auto_create_session`\n\n, and `run_live()`\n\nalready calls its internal get-or-create helper — without the flag a missing session is a `ValueError`\n\n, which is why the code hand-rolled get-then-create:\n\n```\nrunner = Runner(\n    app_name=APP_NAME,\n    agent=root_agent,\n    session_service=session_service,\n    auto_create_session=True,\n)\n```\n\nOne is worth skipping. `Runner(app=App(...))`\n\nis now described as the recommended construction, but `Runner(agent=..., app_name=...)`\n\nis still supported and gets wrapped into an `App`\n\ninternally, with no deprecation warning. \"Recommended\" and \"required\" are different words.\n\nWhether any of this is still true after the next ADK release is checkable rather than a matter of reading changelogs:\n\n```\npython -W error::DeprecationWarning -m pytest -q\n```\n\nZero ADK warnings today. That command is the check after any bump.\n\n**ADK's own docstring names a field that does not exist.** `run_live()`\n\nrefers to `RunConfig.save_live_model_audio_to_session`\n\n. In 2.6.3 the real fields are `save_live_blob`\n\nand `save_live_audio`\n\n. Where documentation and installed source disagree, the source wins — it is the thing that runs.\n\n**A green test suite proves less than it looks like.** The suites stub `run_live()`\n\n, which is what makes them hermetic: no API key, no network, no charge. It also means session creation, resumption and teardown are never exercised, and the suite passes whether or not they work. The two changes most capable of breaking every session — `auto_create_session`\n\nand `context_window_compression`\n\n— were verified against the real API with a throwaway WebSocket client instead.\n\nThe wire protocol, the agent instruction, the tool definitions and the deployment path all came through the migration untouched: binary frames with a 1-byte prefix, 16 kHz PCM in and 24 kHz out, the three server-side tools, Secret Manager for the API key, and the model-id fallback to `gemini-2.5-flash`\n\nunder `adk run`\n\n, since the Live preview model still 404s on `generateContent`\n\n.\n\nA major-version framework upgrade that touches only the compatibility layer is the outcome to want. It is also the argument for keeping shims isolated in one file with a name that says what it is.\n\n`patch_adk.py`\n\nis deleted`send_content()`\n\n`send_realtime()`\n\ntakes `types.Blob`\n\nonly. The keepalive is the call site that catches people out.`RunConfig`\n\nfield name off the event since the original design.`requestAnimationFrame`\n\nstops dead in a background tab while the microphone does not.The Agent Development Kit 2.x release removed the need for a compatibility patch that had been carried since the original build, and deleting it was almost entirely subtractive — the pleasant kind of upgrade. The one removal that bites is `send_realtime()`\n\nrejecting anything that is not a `types.Blob`\n\n, which surfaces as a keepalive failure ten seconds into an otherwise healthy session rather than as a crash at startup.\n\nThe wider lesson came after the migration. A framework upgrade tells you what stopped compiling; it says nothing about what still compiles and is wrong. A dead branch compiles. A no-op log line runs. A throttled callback returns cleanly. An undocumented frame rate is accepted by the server. Every one of these survived a migration, a test suite, a lint config and a demo that visibly worked, and each was found by reading the documentation next to the code rather than by running anything.\n\nFor anyone running a Live agent of their own, three checks cost about an hour between them: confirm the transcript handler has ever produced output, confirm video keeps flowing with the tab hidden, and read the session-limit page next to your `RunConfig`\n\n.", "url": "https://wpnews.pro/news/do-i-still-need-a-monkey-patch-for-gemini-live", "canonical_source": "https://dev.to/gde/do-i-still-need-a-monkey-patch-for-gemini-live-4c3e", "published_at": "2026-08-13 01:41:40+00:00", "updated_at": "2026-08-13 02:15:15.465124+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "large-language-models"], "entities": ["Google", "Gemini 3.1 Flash Live", "Agent Development Kit (ADK)", "FastAPI", "WebSocket", "AudioWorklet", "Cloud Run", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/do-i-still-need-a-monkey-patch-for-gemini-live", "markdown": "https://wpnews.pro/news/do-i-still-need-a-monkey-patch-for-gemini-live.md", "text": "https://wpnews.pro/news/do-i-still-need-a-monkey-patch-for-gemini-live.txt", "jsonld": "https://wpnews.pro/news/do-i-still-need-a-monkey-patch-for-gemini-live.jsonld"}}