{"slug": "qwen-3-8-27b-benchmarked-agentic-index-vision-and-reasoning-tests", "title": "Qwen 3.8 27B Benchmarked: Agentic Index, Vision, and Reasoning Tests", "summary": "Alibaba's Qwen team released Qwen 3.8 27B, a dense 27 billion parameter multimodal model that scored 51 on the Artificial Analysis agentic index, trailing only Kimi K2, a 2.8 trillion parameter model. In hands-on tests, the model demonstrated strong object counting and bounding box drawing, and its intelligence level was close to Claude 4.8 on max settings. Running locally on a dual NVIDIA DGX Spark cluster with NVFP4 quantization, it achieved 15-20 tokens per second on single-thread inference and 60-70 tokens per second under concurrency.", "body_md": "# Qwen 3.8 27B Benchmarked: Agentic Index, Vision, and Reasoning Tests\n\nHands-on benchmarks of Qwen 3.8 27B covering its agentic index score, image counting, bounding box drawing, and reasoning effort output quality.\n\n## What is Qwen 3.8 27B and why does it matter?\n\nQwen 3.8 27B is a dense 27 billion parameter model from Alibaba’s Qwen team, positioned as one of the strongest local models you can currently run on consumer or prosumer hardware. It’s a multimodal model, meaning it handles both text and images, and it supports adjustable reasoning effort for tuning output quality against compute cost. What makes it notable isn’t just raw benchmark performance but that its real-world behavior, tested through actual coding and vision tasks, largely holds up to those numbers.\n\n## TL;DR\n\n- Qwen 3.8 27B scored\n**51 on the artificial analysis agentic index**, putting it just behind Kimi K2, a much larger 2.8 trillion parameter model, which is a striking result for a 27B dense model. - Its\n**intelligence level lands close to Claude 4.8 on max settings**, a model that was state-of-the-art only a few months prior, suggesting real gains beyond benchmark optimization. - The model shows surprisingly strong\n**object counting and bounding box drawing** on images, dividing pictures into patches and pointing at objects without relying on external detection tools. **Reasoning effort settings**(off, low, medium, extra high) dramatically change output quality and token usage, with medium sometimes burning more tokens than extra high depending on the task.- Running locally on hardware like a dual\n**NVIDIA DGX Spark cluster** with the NVFP4 quantized version produced roughly 15 to 20 tokens per second on single-thread inference, scaling to 60 to 70 tokens per second under concurrency. - The model pairs well with agentic coding harnesses like\n**DeepSeek Harness**, which surged past 60,000 GitHub stars within a week of release and offers full trajectory auditing of every tool call and reasoning step. - Cranking reasoning effort too high can backfire, sometimes consuming the entire completion budget on thinking and leaving little or nothing for actual output.\n\n## Remy is new. The platform isn't.\n\nRemy is the latest expression of years of platform work. Not a hastily wrapped LLM.\n\n## How does Qwen 3.8 27B perform on agentic benchmarks?\n\nThe most cited number here is the artificial analysis agentic index, a benchmark meant to capture how well a model performs on task-oriented, tool-using workflows rather than static question answering. Qwen 3.8 27B scored 51 on this index, trailing only Kimi K2, a 2.8 trillion parameter model that dwarfs Qwen’s 27 billion parameters by roughly a hundredfold.\n\nThat gap in scale versus performance is the headline story. A dense model this size competing with models orders of magnitude larger raises the obvious question of benchmark optimization, sometimes called “benchmaxing,” where a model is tuned specifically to score well on known test suites rather than generalize. That’s a fair concern with any model that overperforms its parameter count. But the value of this particular result is that it was paired with hands-on testing in coding and vision tasks, and the model’s real-world output, from website generation to image analysis, matched the benchmark’s implied competence rather than collapsing under practical use.\n\nComparatively, the model’s general intelligence level tracks close to Claude 4.8 running at max reasoning settings, a model that held state-of-the-art status only months earlier. For a locally runnable, open dense model, that’s a meaningful benchmark to clear.\n\n## What can Qwen 3.8 27B do with images?\n\nQwen 3.8 27B is a vision language model, so it accepts images alongside text prompts. Basic image description works as expected: fed a reference photo, it produces detailed, accurate descriptions of scene content. Asked to go further and analyze an image using OpenCV, it independently wrote its own Python analysis pipeline, including edge detection and corner overlays to identify shapes and features in the image, all running locally without external vision APIs.\n\nThe more surprising result involves counting. Vision language models are generally known to struggle with counting objects in images, often producing plausible-sounding but wrong totals. Given an image and asked how many cars appeared in it, Qwen counted 26, then explained its reasoning by walking through the image row by row: a top row with a specific count, a main lot with another count, and so on. The final number was close to accurate, and the visible chain of thought showed the model dividing the image into patches of variable size to count objects section by section rather than guessing a total outright.\n\nAsked a follow-up about how many white cars appeared, the model repeated the same patch-based counting process and arrived at eight, when the actual image contained closer to five. The method was sound even if the final number overshot.\n\n## Can Qwen 3.8 27B draw bounding boxes on objects?\n\n## Remy doesn't write the code. It manages the agents who do.\n\nRemy runs the project. The specialists do the work. You work with the PM, not the implementers.\n\nYes, and this is arguably the most unusual capability demonstrated. Asked to draw bounding boxes around objects in an image, the model wrote its own Python program and supplied the actual pixel coordinates for each box directly from its own visual analysis, not from an external object detection model. The results were largely accurate: most bounding boxes correctly framed objects like cars and a pickup truck, though a handful were oversized or loosely fit.\n\nThis kind of grounded visual pointing, where a language model directly outputs spatial coordinates tied to what it perceives in an image, has been described in research from teams like DeepSeek’s vision papers, but seeing it work reliably in a locally run open model is less common. The lack of dependency on a separate object detector makes the result more notable, even accounting for imperfect boxes on some objects.\n\n## How does reasoning effort affect output quality?\n\nQwen 3.8 27B exposes four reasoning effort levels: completely off, low, medium, and extra high. There’s no “high” tier between medium and extra high. This setting controls how much internal reasoning, or “thinking,” the model performs before producing a final answer, and it has an outsized effect on output quality.\n\nIn a test asking the model to research itself using web search and generate a self-descriptive website, the differences were stark:\n\n**Reasoning off** produced roughly 20,000 output tokens but included fabricated, made-up information, the kind of low-quality output typical of small local models running without deliberation.**Low reasoning** produced a noticeably better website with clean visuals, arguably competitive with early releases of larger commercial models.**Medium reasoning** added animations and further polish, though token consumption jumped substantially, using around 860,000 input tokens and 38,000 output tokens across roughly 63 minutes.**Extra high reasoning** delivered the most refined result, with animations similar to medium but different color choices, consuming around 500,000 input tokens and 60,000 output tokens over about 86 minutes.\n\nThe key risk: setting reasoning too high doesn’t guarantee proportionally better output, and it can sometimes consume the entire token budget on internal thinking, leaving nothing for the final answer. Token usage doesn’t scale linearly or predictably with reasoning level either, since medium reasoning in this test used more total tokens than extra high. Anyone deploying this model should treat reasoning effort as a variable that needs task-specific tuning, not a dial to simply max out.\n\n## Is Qwen 3.8 27B worth running locally?\n\nFor anyone with capable local hardware, yes, with caveats. On a dual NVIDIA DGX Spark cluster running the NVFP4 quantized version, single-thread inference produced 15 to 20 tokens per second, a modest rate given that DGX Sparks aren’t purpose-built for dense 27B models. Under concurrent requests, throughput rose to 60 to 70 tokens per second, which is workable for agentic or batch workloads.\n\nFor platform compatibility, Apple Silicon users are best served by the MLX-formatted version of the model, while Linux and Windows users have options like vLLM and SGLang for serving. The model is available on Hugging Face and integrates with agentic coding harnesses, notably DeepSeek Harness, an open-source, plugin-based agent framework that gained rapid adoption (reportedly crossing 60,000 GitHub stars within about a week) despite not accepting external code contributions. That harness includes a trajectory feature that logs every system prompt, tool call, payload, and timing detail for full auditability, useful for debugging where an agent’s tool use or reasoning breaks down.\n\n### Built like a system. Not vibe-coded.\n\nRemy manages the project — every layer architected, not stitched together at the last second.\n\nThe tradeoff is compute cost at higher reasoning settings. Extra high reasoning tasks can take well over an hour and consume hundreds of thousands of tokens, so matching reasoning effort to the task’s actual complexity matters for anyone running this on limited hardware.\n\n## Frequently Asked Questions\n\n### What is the agentic index score for Qwen 3.8 27B?\n\nIt scored 51 on the artificial analysis agentic index, just behind Kimi K2, a 2.8 trillion parameter model, despite Qwen having a fraction of the parameters.\n\n### Does Qwen 3.8 27B support image analysis and vision tasks?\n\nYes. It can describe images in detail, perform object counting by dividing images into patches, and draw bounding boxes around detected objects by generating coordinate data directly, without relying on a separate object detection model.\n\n### How accurate is Qwen 3.8 27B at counting objects in images?\n\nIt performs noticeably better than typical vision language models at this task, using a patch-based counting approach visible in its reasoning trace, though counts can still overshoot the true number on more complex images.\n\n### What reasoning effort setting should I use with Qwen 3.8 27B?\n\nIt depends on the task. Low reasoning already produces solid results for many generation tasks, while medium and extra high improve polish but significantly increase token usage and runtime, sometimes without proportional quality gains.\n\n### What hardware do I need to run Qwen 3.8 27B locally?\n\nIt runs on consumer and prosumer hardware. Apple Silicon users should use the MLX version, while Linux and Windows users can serve it through vLLM or SGLang. Performance scales with hardware, with multi-GPU or clustered setups like dual DGX Sparks improving concurrent throughput.", "url": "https://wpnews.pro/news/qwen-3-8-27b-benchmarked-agentic-index-vision-and-reasoning-tests", "canonical_source": "https://www.mindstudio.ai/blog/qwen-3-27b-local-benchmark/", "published_at": "2026-08-20 00:00:00+00:00", "updated_at": "2026-08-20 18:13:40.110022+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "computer-vision", "ai-research", "ai-products"], "entities": ["Alibaba", "Qwen", "Qwen 3.8 27B", "Kimi K2", "Claude 4.8", "NVIDIA DGX Spark", "DeepSeek Harness", "Artificial Analysis"], "alternates": {"html": "https://wpnews.pro/news/qwen-3-8-27b-benchmarked-agentic-index-vision-and-reasoning-tests", "markdown": "https://wpnews.pro/news/qwen-3-8-27b-benchmarked-agentic-index-vision-and-reasoning-tests.md", "text": "https://wpnews.pro/news/qwen-3-8-27b-benchmarked-agentic-index-vision-and-reasoning-tests.txt", "jsonld": "https://wpnews.pro/news/qwen-3-8-27b-benchmarked-agentic-index-vision-and-reasoning-tests.jsonld"}}