{"slug": "gemini-4-neon-walkthrough-why-the-same-model-swings-from-39-to-12-8", "title": "Gemini 4 Neon walkthrough: why the same model swings from 39% to 12.8%", "summary": "A developer re-ran a coding evaluation for Gemini 4 Neon and found the model's score collapses from 39% to 12.8% when a generic function-calling shim replaces its native tool-calling format. The shim rejects the model's parallel tool calls, and each rejected call is counted as a failed test case, with raw logs showing parsing failures lining up one-to-one with the score drop. Confirming the delta took roughly six hours of GPU time across both configurations.", "body_md": "I got tired of seeing the same model quoted at 39% and 12.8% with no explanation, so I recorded a full walkthrough breaking down exactly where those numbers come from. It started when I re-ran a coding eval and got a score nearly a third of what the leaderboard claimed — same weights, same prompts, different harness.\n\nThe video goes through the two evaluation setups side by side: one uses the model's native tool-calling format, the other wraps everything in a generic function-calling shim. That shim alone drops the score from 39% to 12.8% because the model keeps emitting parallel tool calls that the generic parser rejects, and every rejected call counts as a failed test case. I show the raw logs where you can see the parsing failures line up one-to-one with the score drop.\n\nIt cost me about six hours of GPU time re-running both configurations to confirm the delta wasn't noise, plus the editing time to make the logs readable on screen. The walkthrough is up at [https://www.youtube.com/watch?v=gBSZXtzBSfE](https://www.youtube.com/watch?v=gBSZXtzBSfE) if you want to see the exact log lines that explain the gap.", "url": "https://wpnews.pro/news/gemini-4-neon-walkthrough-why-the-same-model-swings-from-39-to-12-8", "canonical_source": "https://dev.to/imapphelp/gemini-4-neon-walkthrough-why-the-same-model-swings-from-39-to-128-2k0e", "published_at": "2026-11-21 04:04:14+00:00", "updated_at": "2026-11-21 04:11:55.837558+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-agents"], "entities": ["Gemini 4 Neon", "Google"], "also_reported_by": [], "takeaway": "A generic function-calling shim dropped Gemini 4 Neon's coding eval score from 39% to 12.8% because it rejected the model's parallel tool calls and counted each rejection as a failed test case.", "recommendation": {"advice": "Run coding evals through the model's native tool-calling format rather than a generic function-calling shim, since rejected parallel tool calls can cut scores from 39% to 12.8%.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/gemini-4-neon-walkthrough-why-the-same-model-swings-from-39-to-12-8", "markdown": "https://wpnews.pro/news/gemini-4-neon-walkthrough-why-the-same-model-swings-from-39-to-12-8.md", "text": "https://wpnews.pro/news/gemini-4-neon-walkthrough-why-the-same-model-swings-from-39-to-12-8.txt", "jsonld": "https://wpnews.pro/news/gemini-4-neon-walkthrough-why-the-same-model-swings-from-39-to-12-8.jsonld"}}