{"slug": "before-adding-gemma-4-to-monkeycode-run-a-model-capability-contract", "title": "Before Adding Gemma 4 to MonkeyCode, Run a Model Capability Contract", "summary": "A developer created a reusable model capability contract for MonkeyCode, an open-source AI development platform, to verify that hosted endpoints like Gemma 4 expose all advertised capabilities. The contract extends MonkeyCode's existing transport health check with protocol, capability, and task quality gates, ensuring coding agents can rely on model features such as system roles, streaming, and tool calls.", "body_md": "Gemma 4 is arriving in model catalogs with an unusually broad capability surface. Google's [official overview](https://ai.google.dev/gemma/docs/core), last updated July 8, 2026, lists five sizes, text and image input across the family, audio on selected variants, native system-role support, and 128K or 256K context windows depending on model size.\n\nThose are model-family properties. They do **not** prove that a particular hosted endpoint, quantization, API adapter, or AI coding platform exposes every property correctly.\n\nThat distinction matters when adding a model to [MonkeyCode](https://github.com/chaitin/MonkeyCode), an AGPL-3.0 open-source AI development platform for teams. Its public repository describes managed server-side development environments, AI model and task management, project requirements, online use, native mobile clients, and private deployment.\n\nThis article reviews MonkeyCode at commit [ c58bcd4](https://github.com/chaitin/MonkeyCode/tree/c58bcd4dd4b7031f469a1271f276d22550b8f523) and builds the missing second gate: a reusable model capability contract.\n\nIt is a source review and test harness, **not** a Gemma 4 benchmark or a claim that MonkeyCode currently ships a verified Gemma 4 integration. I validated the harness against a local OpenAI-compatible fixture, not a live Gemma 4 endpoint.\n\nMonkeyCode is relevant here because model choice is a first-class platform concern rather than a hard-coded SDK call.\n\nAt the reviewed commit, its [model configuration contract](https://github.com/chaitin/MonkeyCode/blob/c58bcd4dd4b7031f469a1271f276d22550b8f523/backend/domain/model.go#L198-L213) records:\n\nThe add-model flow then performs a health check before saving. That is the right first gate: reject a bad URL, credential, model ID, or protocol choice early.\n\nThe important limitation is explicit in [the health-check source](https://github.com/chaitin/MonkeyCode/blob/c58bcd4dd4b7031f469a1271f276d22550b8f523/backend/pkg/llm/check.go#L34-L36): it checks API reachability, authentication, and model existence, not answer content. For an OpenAI Chat endpoint, it sends `hi`\n\nwith `max_tokens: 1`\n\nand accepts a non-error response.\n\nThat proves transport health. A coding agent needs more.\n\nI would separate readiness into four gates:\n\n| Gate | Question | Failure response |\n|---|---|---|\n| Transport | Can the platform authenticate and reach the exact model ID? | Do not save or route traffic |\n| Protocol | Do system roles, streaming, tool calls, cancellation, and errors match the selected API contract? | Keep the model disabled |\n| Capability | Does this deployed variant actually support the modalities and limits you advertise? | Remove the capability flag or change the route |\n| Task quality | Does it pass your repository-specific coding evaluation within latency and cost budgets? | Do not make it a default |\n\nMonkeyCode's current check covers the first gate. The next script exercises three protocol behaviors that a one-token response cannot establish.\n\nSave this as `probe-openai-model.mjs`\n\n. It requires Node.js 20 or newer and never prints the API key.\n\n``` js\nconst required = [\"MODEL_BASE_URL\", \"MODEL_API_KEY\", \"MODEL_ID\"];\nfor (const name of required) {\n  if (!process.env[name]) throw new Error(`Missing ${name}`);\n}\n\nconst baseUrl = process.env.MODEL_BASE_URL.replace(/\\/$/, \"\");\nconst endpoint = `${baseUrl}/chat/completions`;\n\nasync function chat(body) {\n  const response = await fetch(endpoint, {\n    method: \"POST\",\n    headers: {\n      authorization: `Bearer ${process.env.MODEL_API_KEY}`,\n      \"content-type\": \"application/json\",\n    },\n    body: JSON.stringify({ model: process.env.MODEL_ID, ...body }),\n    signal: AbortSignal.timeout(30_000),\n  });\n  const text = await response.text();\n  if (!response.ok) throw new Error(`HTTP ${response.status}: ${text.slice(0, 300)}`);\n  return { response, text };\n}\n\nconst textResult = await chat({\n  messages: [\n    { role: \"system\", content: \"Reply with exactly SYS_OK and nothing else.\" },\n    { role: \"user\", content: \"Follow the system instruction.\" },\n  ],\n  temperature: 0,\n  max_tokens: 16,\n});\nconst textBody = JSON.parse(textResult.text);\nconst content = textBody.choices?.[0]?.message?.content?.trim();\nif (content !== \"SYS_OK\") {\n  throw new Error(`system-role contract failed: ${JSON.stringify(content)}`);\n}\nconsole.log(`PASS system role — usage reported: ${Boolean(textBody.usage)}`);\n\nconst toolResult = await chat({\n  messages: [{ role: \"user\", content: \"Look up the weather for Paris.\" }],\n  tools: [{\n    type: \"function\",\n    function: {\n      name: \"lookup_weather\",\n      description: \"Look up weather by city\",\n      parameters: {\n        type: \"object\",\n        properties: { city: { type: \"string\" } },\n        required: [\"city\"],\n        additionalProperties: false,\n      },\n    },\n  }],\n  tool_choice: \"required\",\n  max_tokens: 128,\n});\nconst toolBody = JSON.parse(toolResult.text);\nconst call = toolBody.choices?.[0]?.message?.tool_calls?.[0];\nif (call?.function?.name !== \"lookup_weather\") {\n  throw new Error(`tool-call contract failed: ${toolResult.text.slice(0, 300)}`);\n}\nconst args = JSON.parse(call.function.arguments);\nif (args.city !== \"Paris\") throw new Error(`unexpected tool arguments: ${call.function.arguments}`);\nconsole.log(`PASS tool call — ${call.function.name}`);\n\nconst streamResult = await chat({\n  messages: [{ role: \"user\", content: \"Reply with STREAM_OK.\" }],\n  stream: true,\n  temperature: 0,\n  max_tokens: 16,\n});\nconst contentType = streamResult.response.headers.get(\"content-type\") || \"\";\nif (!contentType.includes(\"text/event-stream\")) {\n  throw new Error(`stream contract failed: content-type was ${contentType || \"missing\"}`);\n}\nif (!streamResult.text.includes(\"data:\") || !streamResult.text.includes(\"[DONE]\")) {\n  throw new Error(`stream contract failed: ${streamResult.text.slice(0, 300)}`);\n}\nconsole.log(`PASS SSE stream — ${streamResult.text.length} bytes`);\n```\n\nRun it against the **exact endpoint and model ID** you plan to configure:\n\n```\nMODEL_BASE_URL=\"https://provider.example/v1\" \\\nMODEL_API_KEY=\"your-test-key\" \\\nMODEL_ID=\"the-exact-provider-model-id\" \\\nnode probe-openai-model.mjs\n```\n\nThe expected shape is:\n\n```\nPASS system role — usage reported: true\nPASS tool call — lookup_weather\nPASS SSE stream — 184 bytes\n```\n\nTreat that byte count as an example, not a benchmark. A provider may also omit usage from the response; the script reports that fact without failing the protocol gate.\n\nThe official Gemma 4 overview says small models have 128K context while medium models support 256K. It also warns that context length adds KV-cache memory beyond the static model weights.\n\nSo `context_limit: 256000`\n\nshould not be copied into MonkeyCode merely because the family documentation contains that number. Record a smaller verified operational envelope for the exact serving stack:\n\n```\nmodel_id: exact-provider-model-id\ninterface: openai_chat\nsystem_role: pass\ntool_calls: pass\nstreaming: pass\nimage_input: not_tested\naudio_input: not_exposed_by_this_contract\ndeclared_context_tokens: 256000\ntested_context_tokens: 32000\ntested_output_tokens: 4096\nconcurrent_requests: 4\ntimeout_seconds: 30\ntested_at: 2026-07-14\n```\n\nThis is deliberately conservative. A declared limit is documentation; a tested limit is evidence.\n\nFor a real coding rollout, add repository tasks after the protocol probe:\n\nRun the same set against the current default model. Promote Gemma 4 only when a named variant, serving stack, and configuration meet predeclared thresholds. Do not infer coding quality from parameter count, context length, or one successful prompt.\n\nMonkeyCode already provides useful control points for this workflow: central model configuration, explicit interface selection, capability flags, managed development environments, team task workflows, and private deployment. Because the project is open source, the exact health-check boundary can be inspected rather than guessed.\n\nThe practical improvement is to preserve that fast health check and add a versioned capability result beside it. Then a team can distinguish:\n\nThat is a safer way to adopt a fast-moving model family without slowing experimentation to a halt.\n\nDisclosure: I contribute to the MonkeyCode project. The MonkeyCode observations above are based on the linked public repository at the specified commit. The probe was validated against a local fixture, not a live Gemma 4 deployment, and this article does not claim completed Gemma 4 compatibility or benchmark results.\n\nIf your team is evaluating model routing or private deployment, the [MonkeyCode Discord](https://discord.gg/2pPmuyr4pP) is the direct place to compare endpoint contracts and ask about currently supported configurations.", "url": "https://wpnews.pro/news/before-adding-gemma-4-to-monkeycode-run-a-model-capability-contract", "canonical_source": "https://dev.to/kongkong1/before-adding-gemma-4-to-monkeycode-run-a-model-capability-contract-4d07", "published_at": "2026-07-14 04:22:25+00:00", "updated_at": "2026-07-14 04:30:46.554816+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["MonkeyCode", "Gemma 4", "Google", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/before-adding-gemma-4-to-monkeycode-run-a-model-capability-contract", "markdown": "https://wpnews.pro/news/before-adding-gemma-4-to-monkeycode-run-a-model-capability-contract.md", "text": "https://wpnews.pro/news/before-adding-gemma-4-to-monkeycode-run-a-model-capability-contract.txt", "jsonld": "https://wpnews.pro/news/before-adding-gemma-4-to-monkeycode-run-a-model-capability-contract.jsonld"}}