{"slug": "i-asked-the-same-question-to-7-local-llms-speed-and-intelligence-didn-t-line-up", "title": "I Asked the Same Question to 7 Local LLMs — Speed and Intelligence Didn't Line Up: DGX Spark Benchmarks", "summary": "A Microsoft MVP based in Japan benchmarked seven local large language models on an NVIDIA DGX Spark, running identical Japanese business questions to compare speed, format adherence, and answer quality. Results showed no clean correlation between model size and performance: a fast model gave risky answers, a 120B model returned an empty response, and Nemotron 3 Super's 87GB weights did not consistently justify their size. The benchmark highlights practical trade-offs for deploying local AI in business contexts.", "body_md": "*Originally published on my Substack. I'm a Microsoft MVP based in Japan, writing in English about the AI agent systems I actually run in production.*\n\nLocal AI models keep multiplying. But comparing numbers on model cards alone doesn't tell you which one to actually use.\n\nDoes a higher parameter count mean smarter? Does MoE mean faster? If a model is popular on AI Arena, is it good for my own work? Each question offers a partial clue, but in the end you can't decide without running the same task through the models yourself.\n\nSo this time, I ran the exact same Japanese question through 7 major models running on a single NVIDIA DGX Spark. That includes NVIDIA's Nemotron 3 Super 120B-A12B, for which I actually deployed the large quantized version.\n\nWhat I compared wasn't just tokens/s.\n\nWhether the answer finished in the specified format\n\nWhether it made claims that would be risky in real business use\n\nWhat caveats it raised\n\nWhether any model couldn't answer without changing its settings\n\nThe results didn't shake out into a clean \"bigger model = better\" story. One model was fast but gave an answer too risky to show a customer as-is. Another had excellent timing but dropped the instruction to provide 3 items. And one 120B model, after thinking as hard as it could, ended up with a completely empty answer field.\n\nSo which model struck the best balance for short business-style answers? And did Nemotron 3 Super prove that its 87GB of weights are worth it compared to a 23GB-class model? Below I lay out the raw measurements and every full answer, unedited.\n\n🤖✍️ This article was co-written with AI — AI agents (Claude Code / Codex) generated the draft based on actual collaborative work with Ebisuda and real measurement data, and Ebisuda reviewed and edited the content before publishing.\n\nTo keep observing while only swapping out the model, you need to keep the previous conditions, raw logs, and the reasoning behind each decision in one place. I use \"Ebi Workspace,\" a plugin compatible with Claude Code / Codex, to manage the project's current state and measurement records in Obsidian.\n\n👉 [Claude Code / Codex That Never Forgets, Never Gets Lost — Ebi Workspace](https://note.com/ebibibi/n/n300772aeac93)\n\nI handle model deployment, measurement, and article writing from Discord through the open-source \"Ebi Agent Chat Relay.\" It's a conversation and coordination framework that splits multiple AI sessions into separate threads, letting them run while avoiding conflicts over the same work targets.\n\n👉 [Ebi Agent Chat Relay (GitHub)](https://github.com/ebibibi/ebi-agent-chat-relay)\n\nI used the GB10 in a DGX Spark. The unified memory available from the OS is about 121GiB, and Ollama is version 0.32.9. I fixed the comparison conditions as follows.\n\ntemperature: 0\n\nseed: 42\n\ncontext: 8192\n\nMax generation budget: 320 tokens\n\nthinking: OFF\n\nOne cold run after unloading each model, followed by 3 warm runs\n\nPrimary comparison: average response time across the 3 warm runs\n\nOptimizing each model individually toward its own recommended settings would blend configuration differences together with model differences. So I kept the primary comparison on common conditions, and only ran separate diagnostics to check the cause for any model that couldn't answer under those common conditions.\n\nThe question posed a scenario where a small or midsize business wants to adopt generative AI without sending customer data to an external cloud AI. I asked the models to list 3 tasks suited to local AI, with each item written as a single line in the form \"Task name: reason / caveat,\" followed by a short adoption verdict at the end. I also prohibited inventing product names or law names.\n\nThe warm averages ranked as follows.\n\nQwen3.5 35B — 1.49s, 76.27 tokens/s, followed the format\n\nGLM-4.7-Flash — 1.94s, 64.64 tokens/s, broke the format\n\nQwen3.6 35B-A3B — 2.12s, 44.72 tokens/s, followed the format\n\nGPT-OSS 120B — 8.98s, displayed answer was empty\n\nNemotron 3 Super 120B-A12B — 9.09s, 20.36 tokens/s, followed the format\n\nGemma 4 31B — 9.81s, 10.41 tokens/s, followed the format\n\nQwen2.5 72B — 21.68s, 4.42 tokens/s, followed the format\n\nFor short-form responses, Qwen3.5 is overwhelmingly the fastest. But looking at the actual answer shows why deciding on speed alone is dangerous.\n\nHere's Qwen3.5 35B's answer.\n\nCustomer support log analysis: zero risk of confidential data leakage / need to check the balance between processing speed and accuracy\n\nInternal document summarization and drafting: no need to send materials externally even when they contain personal information / depends on the quality of the training data\n\nBusiness rule matching system: can reference confidential internal regulations with high security / incurs initial setup costs\n\nAdoption verdict: when handling highly confidential customer data, local AI is a valid option.\n\nThe format is clean. But \"zero risk of leakage\" is an overstatement. Even with local deployment, risks remain from device compromise, permission settings, logs, backups, and data being carried out. The fastest number on the clock and an answer quality you could adopt for business as-is didn't line up.\n\nHere's Qwen3.6 35B-A3B's answer.\n\nUsing internal data: avoids the risk of customer information leakage / requires specialized knowledge to set up the environment\n\nInternal document summarization: drafting meeting minutes where confidentiality is essential / improving accuracy requires curating training data\n\nHandling business FAQs: responding to inquiries that include personal information / building a verification process to prevent incorrect answers\n\nAdoption verdict: carefully weigh the balance between securing safety and operational load, and roll it out in stages.\n\nIt raised distinct caveats — environment setup, training data, and a verification process for incorrect answers — and kept its final verdict to a staged rollout. It followed the format in 2.12 seconds.\n\nThis is purely my editorial judgment on this one question, but for short, structured Japanese business answers, Qwen3.6 would be my first candidate. It wasn't the fastest, but it struck the best balance between speed and answer quality.\n\nNemotron 3 Super, added this time, is a 120B total / 12B active MoE. Its Q4_K_M quantization in Ollama measured 86.83GB. On the DGX Spark, all the weights fit onto the GPU side, and the warm average was 9.09 seconds.\n\nAuto-generating FAQs for customer support: can learn internal knowledge and respond instantly / operate only within a scope that excludes confidential information\n\nDrafting sales materials: generates templates from past proposals and deal records / exclude personal information from the input\n\nSummarizing and translating internal manuals: restructures long documents into forms appropriate for each department / don't feed in documents where alteration is prohibited\n\nAdoption verdict: it's appropriate to start with a staged pilot operation, within a scope that protects data confidentiality while achieving operational efficiency.\n\nAmong the 7 models, this was the longest and most careful answer. On the other hand, it's somewhat conservative — for instance, demanding the exclusion of personal information even though local operation is the whole premise. On raw short-answer speed alone, it loses to the 23GB-class Qwen3.6.\n\nThat said, Nemotron 3 Super's real strength lies in agentic processing, code, function calling, and long-form reasoning. You can't conclude from this one question that \"loading 87GB isn't worth it.\" It didn't show a speed advantage in the short-answer sprint — the result points to measuring it on longer work next.\n\nGPT-OSS 120B used up the shared 320-token budget entirely on internal thinking, leaving the displayed answer empty. This isn't an API error. It's the result of all 320 generated tokens going into the thinking side.\n\nEven raising the max generation budget to 1000 tokens, it cut off partway through the second item. Raising it to 4000 tokens finally produced a complete answer, at a warm time of 25.89 seconds and 1087 output tokens.\n\nHandling customer inquiries: can be trained using only internal data, reducing the risk of information leakage / model updates require regular data maintenance\n\nAutomatic summarization of internal documents: can extract key points without sending confidential information externally / summarization accuracy requires tuning for business terminology\n\nInventory optimization simulation: runs demand forecasting and ordering plans locally / needs to periodically retrain on seasonal variation in the data\n\nAdoption verdict: if confidentiality is the top priority, start with local AI and consider expanding in stages.\n\nEven with the same model, not knowing the generation budget turns it into \"a model that answers nothing.\" Model configuration is part of its performance.\n\nSummarizing and drafting internal documents: since it contains confidential information, data leakage risk can be avoided. / If there's uncertainty about accuracy due to insufficient recognition, training adjustments on the local model are needed.\n\nGenerating FAQs from customer support history: quality is high because it can reflect internal-specific terminology and know-how.\n\nAdoption verdict:\n\nIf data protection is the priority, it's best suited for document creation and QA tasks.\n\n1.94 seconds is attractive, but against the requirement of 3 items it effectively delivered only 2, and the one-line-under-80-characters constraint broke down too. By speed measurement alone it ranks 2nd; by the standard of completing the task, it's a failure.\n\nLooking up internal regulations: handles highly confidential material such as employment rules, to prevent external leakage / answer accuracy\n\nDrafting customer responses: creates drafts without hiding personal information, for efficiency / securing compute resources\n\nSummarizing technical documents: to safely analyze proprietary design documents and know-how / model optimization\n\nAdoption verdict: recommended when confidentiality is the top priority and the operating cost is acceptable.\n\nIt followed the format. But the caveats end in bare nouns like \"answer accuracy\" and \"securing compute resources,\" leaving it weak on what to actually check. \"Without hiding personal information\" is also a phrase that shouldn't be adopted without permissions and auditing in place first.\n\nCustomer information management: strengthens data protection and privacy compliance / high initial cost\n\nDocument creation support: can handle highly confidential information / requires learning up-to-date information\n\nAutomating quality inspection: improves productivity through fast processing / requires specialized training data\n\nAdoption verdict: suited to work where security and privacy are prioritized.\n\nIt followed the format, but at 21.68 seconds it was the slowest among the models that completed their answer this time. \"Privacy compliance\" can't be guaranteed by localization alone, and the answer overall stays abstract.\n\nI drew four conclusions from this one question.\n\nThe fastest model doesn't necessarily return the safest answer\n\nEven with high tokens/s, dropping the specified format means the task isn't complete\n\nThinking models are put at a disadvantage just by matching the generation budget, so separate the common-condition test from the recovery diagnostic\n\nFor short structured answers, Qwen3.6 strikes a good balance. Nemotron 3 Super should be re-evaluated on longer agentic work\n\nThis isn't a comprehensive ranking. There was only one question, and I didn't measure factuality, code, images, long context, or parallel load. The common conditions of temperature=0 and thinking OFF also aren't settings that draw out each model's maximum capability.\n\nThat's exactly why I kept the raw answers. Rather than stopping at \"it took 9.09 seconds,\" I compared what it actually said in those 9.09 seconds. From there, you can pick the model that fits your own work.\n\nNext, I'll hand the same agentic task to Nemotron 3 Super, Qwen3.6, and GPT-OSS, and compare completion rate, tool calls, and time taken. After Qwen wins the sprint, does the 87GB Nemotron come back in the long distance? That's the next thing to observe.\n\nAs of August 12, 2026, NVIDIA's official Hugging Face model card points to the NVIDIA Nemotron Open Model License. On the other hand, the embedded label on **nemotron-3-super:120b-a12b-q4_K_M** as pulled from Ollama showed the NVIDIA Software and Model Evaluation License.\n\nSince my use here is evaluation, I went ahead with the measurements, but before moving the Ollama-distributed build into production or commercial use, I'll double-check whether this labeling discrepancy has been resolved. I'm not drawing a legal conclusion here.\n\nSo, what did you think? You might have been surprised that even a smaller local LLM can return answers this fast and this good. They keep getting smaller and smarter. That feeling of the same hardware getting more and more capable over time is something you can only experience in an era like this one. It's genuinely fascinating.", "url": "https://wpnews.pro/news/i-asked-the-same-question-to-7-local-llms-speed-and-intelligence-didn-t-line-up", "canonical_source": "https://dev.to/ebibibi/i-asked-the-same-question-to-7-local-llms-speed-and-intelligence-didnt-line-up-dgx-spark-4mg1", "published_at": "2026-08-16 01:01:37+00:00", "updated_at": "2026-08-16 01:11:07.761450+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-products", "ai-tools"], "entities": ["NVIDIA", "DGX Spark", "Nemotron 3 Super", "Qwen3.5", "GLM-4.7-Flash", "Qwen3.6", "GPT-OSS", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/i-asked-the-same-question-to-7-local-llms-speed-and-intelligence-didn-t-line-up", "markdown": "https://wpnews.pro/news/i-asked-the-same-question-to-7-local-llms-speed-and-intelligence-didn-t-line-up.md", "text": "https://wpnews.pro/news/i-asked-the-same-question-to-7-local-llms-speed-and-intelligence-didn-t-line-up.txt", "jsonld": "https://wpnews.pro/news/i-asked-the-same-question-to-7-local-llms-speed-and-intelligence-didn-t-line-up.jsonld"}}