cd /news/large-language-models/gemma-got-10-12-answers-right-but-no… · home › topics › large-language-models › article
[ARTICLE · art-170527] src=discuss.huggingface.co ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Gemma got 10/12 answers right, but none of its raw replies parsed as JSON

A local test of two models run through Ollama on an RTX 4090 found that Qwen (qwen3.8:latest) returned 12/12 correct answers that parsed directly as JSON, while Gemma (gemma4:26b) wrapped all 12 replies in Markdown code fences and scored 0/12 on raw parsing, according to a Project Driftflare write-up posted to the Hugging Face forums. Stripping the fences recovered 10 of Gemma's 12 answers, with the same ordering task missed at both seeds 42 and 43, which commenter nootxlm characterized as a content failure rather than a formatting one. The author called for a rerun with Ollama's structured output enabled to determine whether the constraint fixes only formatting or also changes the ordering result.

read1 min views3 publishedNov 16, 2026
Gemma got 10/12 answers right, but none of its raw replies parsed as JSON
Image: Discuss (auto-discovered)

I ran gemma4:26b and qwen3.8:latest locally with Ollama on an RTX 4090. I wanted to answer a practical question for Project Driftflare: when a workflow asks for JSON, can the next step use the reply as it comes back?

I used six tasks with fixed answer keys, repeated at seeds 42 and 43. The prompt asked for JSON, with no schema enforcement. Qwen returned 12/12 correct answers that parsed directly. Gemma wrapped all 12 replies in Markdown code fences. Removing the fences recovered 10 correct answers. It missed the same ordering task at both seeds.

I’d be interested in a rerun with Ollama’s structured output enabled. Does it fix the formatting alone, or does the ordering result change too? If you try it, please share your exact model tag, hardware, and both scores. Project Driftflare

nootxlm 2

Worth reporting three numbers separately, because they answer different questions: how often the raw reply parses as JSON exactly as returned, how often the content is right once the fences are stripped, and how often the same task fails at both seeds. On your run Gemma is 0/12 on the first and 10/12 on the second, and the ordering miss belongs to the third — it fails at both seeds, so it is a content failure, not a formatting one. Run the structured-output arm on the same six tasks and seeds: if parsing goes to 12/12 while ordering still misses twice, structured output fixed formatting only; if ordering changes, the constraint changed the answer itself, which is worth knowing before the format is trusted in a workflow.

── more in #large-language-models 4 stories · sorted by recency
── more on @gemma 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/gemma-got-10-12-answ…] indexed:0 read:1min 2026-11-16 · —