cd /news/large-language-models/gemini-4-neon-walkthrough-why-the-sa… · home › topics › large-language-models › article
[ARTICLE · art-173627] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Gemini 4 Neon walkthrough: why the same model swings from 39% to 12.8%

A developer re-ran a coding evaluation for Gemini 4 Neon and found the model's score collapses from 39% to 12.8% when a generic function-calling shim replaces its native tool-calling format. The shim rejects the model's parallel tool calls, and each rejected call is counted as a failed test case, with raw logs showing parsing failures lining up one-to-one with the score drop. Confirming the delta took roughly six hours of GPU time across both configurations.

recommendation

Run coding evals through the model's native tool-calling format rather than a generic function-calling shim, since rejected parallel tool calls can cut scores from 39% to 12.8%.

more in playbook: Evals and testing for LLM apps
by read1 min views4 publishedNov 21, 2026

I got tired of seeing the same model quoted at 39% and 12.8% with no explanation, so I recorded a full walkthrough breaking down exactly where those numbers come from. It started when I re-ran a coding eval and got a score nearly a third of what the leaderboard claimed — same weights, same prompts, different harness.

The video goes through the two evaluation setups side by side: one uses the model's native tool-calling format, the other wraps everything in a generic function-calling shim. That shim alone drops the score from 39% to 12.8% because the model keeps emitting parallel tool calls that the generic parser rejects, and every rejected call counts as a failed test case. I show the raw logs where you can see the parsing failures line up one-to-one with the score drop.

It cost me about six hours of GPU time re-running both configurations to confirm the delta wasn't noise, plus the editing time to make the logs readable on screen. The walkthrough is up at https://www.youtube.com/watch?v=gBSZXtzBSfE if you want to see the exact log lines that explain the gap.

── more in #large-language-models 4 stories · sorted by recency
── more on @gemini 4 neon 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/gemini-4-neon-walkth…] indexed:0 read:1min 2026-11-21 · —