Before Adding Gemma 4 to MonkeyCode, Run a Model Capability Contract

A developer created a reusable model capability contract for MonkeyCode, an open-source AI development platform, to verify that hosted endpoints like Gemma 4 expose all advertised capabilities. The contract extends MonkeyCode's existing transport health check with protocol, capability, and task quality gates, ensuring coding agents can rely on model features such as system roles, streaming, and tool calls.

Gemma 4 is arriving in model catalogs with an unusually broad capability surface. Google's official overview https://ai.google.dev/gemma/docs/core , last updated July 8, 2026, lists five sizes, text and image input across the family, audio on selected variants, native system-role support, and 128K or 256K context windows depending on model size. Those are model-family properties. They do not prove that a particular hosted endpoint, quantization, API adapter, or AI coding platform exposes every property correctly. That distinction matters when adding a model to MonkeyCode https://github.com/chaitin/MonkeyCode , an AGPL-3.0 open-source AI development platform for teams. Its public repository describes managed server-side development environments, AI model and task management, project requirements, online use, native mobile clients, and private deployment. This article reviews MonkeyCode at commit c58bcd4 https://github.com/chaitin/MonkeyCode/tree/c58bcd4dd4b7031f469a1271f276d22550b8f523 and builds the missing second gate: a reusable model capability contract. It is a source review and test harness, not a Gemma 4 benchmark or a claim that MonkeyCode currently ships a verified Gemma 4 integration. I validated the harness against a local OpenAI-compatible fixture, not a live Gemma 4 endpoint. MonkeyCode is relevant here because model choice is a first-class platform concern rather than a hard-coded SDK call. At the reviewed commit, its model configuration contract https://github.com/chaitin/MonkeyCode/blob/c58bcd4dd4b7031f469a1271f276d22550b8f523/backend/domain/model.go L198-L213 records: The add-model flow then performs a health check before saving. That is the right first gate: reject a bad URL, credential, model ID, or protocol choice early. The important limitation is explicit in the health-check source https://github.com/chaitin/MonkeyCode/blob/c58bcd4dd4b7031f469a1271f276d22550b8f523/backend/pkg/llm/check.go L34-L36 : it checks API reachability, authentication, and model existence, not answer content. For an OpenAI Chat endpoint, it sends hi with max tokens: 1 and accepts a non-error response. That proves transport health. A coding agent needs more. I would separate readiness into four gates: | Gate | Question | Failure response | |---|---|---| | Transport | Can the platform authenticate and reach the exact model ID? | Do not save or route traffic | | Protocol | Do system roles, streaming, tool calls, cancellation, and errors match the selected API contract? | Keep the model disabled | | Capability | Does this deployed variant actually support the modalities and limits you advertise? | Remove the capability flag or change the route | | Task quality | Does it pass your repository-specific coding evaluation within latency and cost budgets? | Do not make it a default | MonkeyCode's current check covers the first gate. The next script exercises three protocol behaviors that a one-token response cannot establish. Save this as probe-openai-model.mjs . It requires Node.js 20 or newer and never prints the API key. js const required = "MODEL BASE URL", "MODEL API KEY", "MODEL ID" ; for const name of required { if process.env name throw new Error Missing ${name} ; } const baseUrl = process.env.MODEL BASE URL.replace /\/$/, "" ; const endpoint = ${baseUrl}/chat/completions ; async function chat body { const response = await fetch endpoint, { method: "POST", headers: { authorization: Bearer ${process.env.MODEL API KEY} , "content-type": "application/json", }, body: JSON.stringify { model: process.env.MODEL ID, ...body } , signal: AbortSignal.timeout 30 000 , } ; const text = await response.text ; if response.ok throw new Error HTTP ${response.status}: ${text.slice 0, 300 } ; return { response, text }; } const textResult = await chat { messages: { role: "system", content: "Reply with exactly SYS OK and nothing else." }, { role: "user", content: "Follow the system instruction." }, , temperature: 0, max tokens: 16, } ; const textBody = JSON.parse textResult.text ; const content = textBody.choices?. 0 ?.message?.content?.trim ; if content == "SYS OK" { throw new Error system-role contract failed: ${JSON.stringify content } ; } console.log PASS system role — usage reported: ${Boolean textBody.usage } ; const toolResult = await chat { messages: { role: "user", content: "Look up the weather for Paris." } , tools: { type: "function", function: { name: "lookup weather", description: "Look up weather by city", parameters: { type: "object", properties: { city: { type: "string" } }, required: "city" , additionalProperties: false, }, }, } , tool choice: "required", max tokens: 128, } ; const toolBody = JSON.parse toolResult.text ; const call = toolBody.choices?. 0 ?.message?.tool calls?. 0 ; if call?.function?.name == "lookup weather" { throw new Error tool-call contract failed: ${toolResult.text.slice 0, 300 } ; } const args = JSON.parse call.function.arguments ; if args.city == "Paris" throw new Error unexpected tool arguments: ${call.function.arguments} ; console.log PASS tool call — ${call.function.name} ; const streamResult = await chat { messages: { role: "user", content: "Reply with STREAM OK." } , stream: true, temperature: 0, max tokens: 16, } ; const contentType = streamResult.response.headers.get "content-type" || ""; if contentType.includes "text/event-stream" { throw new Error stream contract failed: content-type was ${contentType || "missing"} ; } if streamResult.text.includes "data:" || streamResult.text.includes " DONE " { throw new Error stream contract failed: ${streamResult.text.slice 0, 300 } ; } console.log PASS SSE stream — ${streamResult.text.length} bytes ; Run it against the exact endpoint and model ID you plan to configure: MODEL BASE URL="https://provider.example/v1" \ MODEL API KEY="your-test-key" \ MODEL ID="the-exact-provider-model-id" \ node probe-openai-model.mjs The expected shape is: PASS system role — usage reported: true PASS tool call — lookup weather PASS SSE stream — 184 bytes Treat that byte count as an example, not a benchmark. A provider may also omit usage from the response; the script reports that fact without failing the protocol gate. The official Gemma 4 overview says small models have 128K context while medium models support 256K. It also warns that context length adds KV-cache memory beyond the static model weights. So context limit: 256000 should not be copied into MonkeyCode merely because the family documentation contains that number. Record a smaller verified operational envelope for the exact serving stack: model id: exact-provider-model-id interface: openai chat system role: pass tool calls: pass streaming: pass image input: not tested audio input: not exposed by this contract declared context tokens: 256000 tested context tokens: 32000 tested output tokens: 4096 concurrent requests: 4 timeout seconds: 30 tested at: 2026-07-14 This is deliberately conservative. A declared limit is documentation; a tested limit is evidence. For a real coding rollout, add repository tasks after the protocol probe: Run the same set against the current default model. Promote Gemma 4 only when a named variant, serving stack, and configuration meet predeclared thresholds. Do not infer coding quality from parameter count, context length, or one successful prompt. MonkeyCode already provides useful control points for this workflow: central model configuration, explicit interface selection, capability flags, managed development environments, team task workflows, and private deployment. Because the project is open source, the exact health-check boundary can be inspected rather than guessed. The practical improvement is to preserve that fast health check and add a versioned capability result beside it. Then a team can distinguish: That is a safer way to adopt a fast-moving model family without slowing experimentation to a halt. Disclosure: I contribute to the MonkeyCode project. The MonkeyCode observations above are based on the linked public repository at the specified commit. The probe was validated against a local fixture, not a live Gemma 4 deployment, and this article does not claim completed Gemma 4 compatibility or benchmark results. If your team is evaluating model routing or private deployment, the MonkeyCode Discord https://discord.gg/2pPmuyr4pP is the direct place to compare endpoint contracts and ask about currently supported configurations.