Multimodal Gemma 4 Visual Regression & Patch Agent devchallenge gemmachallenge gemma ai Gemma 4 Challenge: Build With Gemma 4 Submission This is a submission for the Gemma 4 Challenge: Build with Gemma 4 What I Built Multimodal Gemma 4 Visual Regression & Patch Agent The Multimodal Gemma 4 Visual Regression & Patch Agent (Contextual Code Review Visual Patch Agent) is a production-grade multimodal code analysis and visual repair tool powered by Google's native multimodal Gemma 4 models. It bridges the gap between front-end UI bugs and back-end source code by cross-referencing visual screenshots directly with stylesheets, DOM selectors, or components to diagnose root causes, generate patches, and validate them through a closed-loop pipeline. Mermaid Flow Core Features Multimodal Visual & Logical Analysis: Ingests code files (CSS, JS, JSX, TS, TSX, HTML, Python, etc.) alongside UI screenshots of visual regressions or layouts to trace layout bugs directly back to specific CSS selectors or JS component rendering logic. Closed-Loop Safety Validation Pipeline: To ensure generated code is production-safe: PatchApplicabilityChecker: Runs a dry-run git apply --check in an ephemeral in-memory repository to guarantee conflict-free application. ASTValidator: Uses ast.parse for Python files and a custom token-matching parenthesis/bracket balance scanner for JS/TS/JSX to ensure zero syntax errors. FileGroundingValidator: Verifies that diff headers correspond strictly to uploaded file scopes, eliminating AI hallucinations. PatchValidator: Screens changes against dangerous operations (rm -rf, eval/exec, malicious package imports). Interactive Visual Verification Loop: Scrub Split Slider: Compare buggy screenshots with expected fixes side-by-side using an interactive slider. Pixel-Diff Heatmap Overlay: Computes visual color channel changes in-browser using HTML5 Canvas getImageData to overlay changed regions and compute a visual alignment score. "Simulate Fix" Canvas: Shift layout slices and preview the corrected layout on the client side instantly. Automated Benchmark Framework: Built-in test harness with 10 pre-configured CSS, JavaScript, and Python bug cases that evaluates root-cause accuracy, git apply rates, and AST validity. 📊 We validated the agent against a robust suite of 10 distinct frontend and backend bugs (overflow limits, z-index overlays, flex layouts, None pointer checks, circular dependencies, DOM element mismatches). The agent achieved 100% correctness across all engineering tests: Overall Agent Success Rate: 100.0% (10/10 cases resolved) UI Bug Localization Accuracy: 100.0% (correct CSS/JS selector mapping) Git Apply applicability: 100.0% (clean, zero-hunk conflict applying) AST / Syntax validity: 100.0% (100% syntactically correct patches) Average Analysis Latency: 0.90s Average Patch Line Accuracy: 100.0% (identical alignment with human-engineered fixes) Benchmark Table Case ID Test Case Name Language / Type Latency (s) Localization Git Apply AST Valid Patch Accuracy Status 1 CSS Overflow Bug CSS 1.25s PASSED PASSED PASSED 100.0% ✅ SUCCESS 2 Z-Index Stacking Context CSS 1.03s PASSED PASSED PASSED 100.0% ✅ SUCCESS 3 Flexbox Alignment Mismatch CSS 0.60s PASSED PASSED PASSED 100.0% ✅ SUCCESS 4 Python AttributeError (None check) Python 0.67s PASSED PASSED PASSED 100.0% ✅ SUCCESS 5 JS Click Event Selector Mismatch JS 0.96s PASSED PASSED PASSED 100.0% ✅ SUCCESS 6 CSS Low Contrast Contrast Bug CSS 0.82s PASSED PASSED PASSED 100.0% ✅ SUCCESS 7 CSS Sidebar Mobile Breakpoint CSS 0.54s PASSED PASSED PASSED 100.0% ✅ SUCCESS 8 Python Circular Dependency Import Python 0.61s PASSED PASSED PASSED 100.0% ✅ SUCCESS 9 Python SQL Injection / Validation Python 1.42s PASSED PASSED PASSED 100.0% ✅ SUCCESS 10 JS DOM Element querySelector Mismatch JS 1.14s PASSED PASSED PASSED 100.0% ✅ SUCCESS Demo Live URL: https://multimodal-visual-regression-patch-agent.vercel.app Video Demo: https://youtu.be/gvarF7T1C5E See the Gemma 4 Visual Regression & Patch Agent in action, illustrating drag-and-drop file ingestion, screenshot visual overlays, patch generation, and real-time validation badges: Screenshots Patch interface Visual display of the interactive Regression Loop application interface Split slider Interactive Split slider Side-by-side view Visual verification loop Side-by-Side view Pixel Diff Heatmap Pixel-diff heatmap visualization Visual Match Interactive visual match simulation with related code snippets Try It Yourself (Local Reproduction / Setup) You can run the entire agentic system and its benchmark suite locally in seconds using Mock Mode (no API keys required)!
git clone https://github.com/kanyingidickson-dev/Multimodal-Visual-Regression-Patch-Agent.git
cd Multimodal-Visual-Regression-Patch-Agent
python3 -m venv venv source venv/bin/activate pip install -r backend/requirements.txt cd frontend npm install npm run build cd .. python3 backend/benchmark.py python3 backend/app.py Open http://127.0.0.1:5000 to interact with the premium dark glassmorphic review dashboard! You can click Load Example on Model settings for a quick demo launch and review.
For Testing Without API Key:
echo "MOCK_MODE=true" >> .env
python backend/app.py Code Repository: https://github.com/kanyingidickson-dev/Multimodal-Visual-Regression-Patch-Agent Directory Layout: . ├── backend/ │ ├── app.py # FastAPI server & route handlers │ ├── benchmark.py # Automated benchmark suite runner │ ├── code_reviewer.py # Multi-stage review orchestration │ ├── file_parser.py # File ingestion & truncation utilities │ ├── gemma_client.py # API client for OpenRouter & Hugging Face │ ├── patch_utils.py # Security scanners, AST, & git validators │ ├── requirements.txt # Backend dependencies │ └── demo.py # Command-line testing entry ├── frontend/ # React dashboard codebase │ ├── src/ # Source directory │ │ ├── App.jsx # Core dashboard and Visual Verification UI │ │ ├── App.css # Stylesheets │ │ ├── index.css # Color design tokens and layout classes │ │ └── api.js # API client connection methods │ ├── dist/ # Built production frontend bundles │ ├── package.json # npm configuration │ └── vite.config.js # Vite settings ├── examples/ # Demo assets │ ├── benchmark-cases/ # Built-in 10 benchmark test directories │ ├── broken-app/ # Example buggy application │ ├── sample-output.json # Standard review structure file │ └── sample-screenshot.png # Base testing image ├── prompts/ # Custom agent instructions │ ├── system_prompt.md # Architectural guidance rules │ └── user_prompt.md # Multimodal instruction format ├── Dockerfile # Production Docker image blueprint ├── docker-compose.yml # Container coordinator ├── README.md # Project documentation └── LICENSE # MIT License Key Directory Structure backend/app.py — FastAPI web server supporting dynamic parameters and multipart file/screenshot ingestion. backend/benchmark.py — Automated test case generator and benchmark runner. backend/code_reviewer.py — Core orchestrator wrapping OpenRouter/HuggingFace API calls in multimodal content blocks. backend/gemma_client.py — Client supporting dense model choices and contextual, high-fidelity mock review generations. backend/patch_utils.py — Closed-loop safety validators (Git apply check, AST parsers, and file grounding). frontend/src/App.jsx — React interface with interactive before/after split scrub sliders, pixel difference canvases, and patch validation panels. How I Used Gemma 4 Native Multimodality: Native pixel integration enables excellent spatial mapping from image regions to matching stylesheets. 256K Context Window: Essential for ingesting multiple visual assets alongside dense code modules. Accurate Code Generation: Ensures precise unified git diff syntaxes that compile and apply flawlessly.
For OpenRouter and Hugging Face, images are mapped to base64 data payloads. We structure the prompt to pass visual tokens first, as prepending pixels optimizes the native layout spatial grounding before digesting text source code:
if images:
user_content = []
for img_data in images:
user_content.append({
"type": "image_url",
"image_url": {"url": img_data}
})
user_content.append({
"type": "text",
"text": user_prompt
})
JSON Output Constraints: To enable programmatic extraction of findings and patches, the system instructs Gemma 4 to respond in structured JSON. The output is parsed automatically, feeding the diff highlights and safety validators:
{
"summary": "...",
"root_cause": "...",
"fix_plan": ["...", "..."],
"patch": "diff --git a/filename b/filename...",
"assumptions": ["...", "..."],
"confidence": "high | medium | low"
}
Safety Layer To protect developers, all generated patches are validated before rendering: Block matches on destructive shell scripts (e.g. rm -rf, /dev/null). Warns if insecure libraries are imported (e.g. pickle, subprocess in unsafe parameters). Checks code validation errors using compilation. 🚀 Future Vision & Roadmap Headless visual regression (CI/CD): Incorporate Playwright automation tasks to apply patches in temporary containers, launch the application, capture screenshots, and complete the visual loop automatically in the cloud. Bi-directional IDE Sync: Allow developers to highlight visual elements in a browser extension and instantly jump to the corresponding code line inside VS Code or Cursor. Support for Figma Files: Integrate Figma design files directly to compare pixel-perfect implementations automatically. Built for the Gemma 4 Challenge:- demonstrating how open, multimodal models can empower developers with intelligent, visual-aware coding tools. Top comments (1) pic Add to the discussion tahosin profile image S M Tahosin • May 24 Taking visual regression testing from "here is a failed diff" to "here is the patch to fix the UI" is a massive workflow upgrade! It’s amazing to see Gemma 4 being used in a production-grade multimodal capacity like this. Did you find the model struggled with highly subtle pixel shifts (like font anti-aliasing), or did it confidently distinguish them from actual layout breaks? Great project! 1 like Like Reply Code of Conduct • Report abuse profile Bright Data Promoted Image of Bright Data and n8n Challenge SOC-CERT: Automated Threat Intelligence System with n8n & AI