A 27B Model, Shrunk to Fit Consumer GPUs #
Underdog Saluki 27B 1.0 has accomplished a significant feat: compressing a 27-billion-parameter language model to a mere 7.89 GB. This contrasts sharply with the full-precision Qwen 27B, which occupies roughly 54–55 GB, and even standard 4-bit quantized versions, typically around 16.5 GB. This extreme reduction makes a large LLM accessible on consumer hardware.
This size reduction is achieved through aggressive quantization, a process that shrinks the numerical representation of a model’s internal weights. Saluki represents each weight at approximately 2.35 bits, a dramatic decrease from the 16 bits used in the original full-precision model. While this inevitably sacrifices some fidelity, it drastically cuts memory footprint.
For local deployment, Saluki 27B is designed for ease of use. It functions with stock llama.cpp and Ollama, eliminating the need for custom forks. A GPU with 12GB of VRAM may run the model, but performance—particularly context length and generation speed—will depend on available memory and the extent of GPU offload.
The Surprising Win: Tool Calls, Not Smarts #
Saluki’s true appeal emerges in its agentic capabilities, not raw intelligence. Testing revealed compelling tool-calling behavior: it correctly selected a weather tool for Salt Lake City, Dubai, and Saigon, supplying structured arguments. Crucially, the model also knew when not to use a tool, answering directly when appropriate.
This specialized tuning yielded surprising benchmark results. On Underdog’s tool-calling benchmark, Saluki scored 88 out of 120. This surpassed the underlying ISTA-DASLab quant’s 76 and even the full 54 GB Qwen 27B’s reported 84.
Design logic explains this apparent paradox. Underdog prioritized tool choice and schema-following over preserving every capability equally, a deliberate trade-off for agentic workflows. By focusing on where the model allocates its compressed "knowledge," developers could enhance specific functionalities.
For instance, while excelling at tool calls, Saluki sacrifices in other domains. Its AIME 2025 math score dropped to 79.2 from the original model’s 94, and its SWE-bench verified score fell from 33 to 30. This highlights a targeted quantization strategy rather than a general performance uplift.
Full training details for Saluki have not been published, leaving specifics of the calibration pass via importance matrix (imatrix) targeting tool use and schema following undisclosed. This makes it difficult to ascertain the exact mechanisms behind its impressive, albeit specialized, performance gains.
The Bill Comes Due in Math—and Maybe Speed #
The extreme compression comes with a cost: a significant drop in general intelligence benchmarks. Saluki’s AIME 2025 math score plummeted from the original model’s 96.7 to 79. Similarly, its SWE-bench Verified performance fell from 33/50 to 30/50. This trade-off reveals a model optimized for a narrow, but critical, capability.
Tool-calling quality itself isn't uniformly high across all conditions. While Saluki scores 88/120 on Underdog’s own tool-calling benchmark, its parallel-call score varies from 42 with a strict parser to 55 with a lenient one. Most reported results, including the 88/120 score, originate from the developer’s benchmarks. For further details and the model card, consult ConwayResearch/Underdog-Saluki-27B-1.0 - Hugging Face.
Critically, a smaller model size does not guarantee faster execution. An independent agent-loop test found Saluki to be the slowest among tested models, processing approximately 10 steps per minute. The primary bottleneck identified was prefill, the stage where the model processes input context before generating a response. Smaller weights reduce VRAM footprint, but they do not automatically accelerate overall task completion in an agentic workflow.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
Who Should Try Saluki—and Who Should Skip It #
Saluki’s extreme quantization creates a niche. Developers with 12–16GB of available VRAM, who prioritize local agent experimentation and structured tool calling, will find Saluki a compelling option. Its 7.89GB footprint and robust llama.cpp compatibility simplify local deployment.
However, if your workflows demand strong mathematical reasoning, coding proficiency, or broad general intelligence, a standard 4-bit Qwen version remains the better choice. These models typically require around 24GB of VRAM but preserve more of the original 27B model’s core capabilities, as evidenced by higher AIME 2025 and SWE-bench Verified scores.
Approach the 88/120 tool calling score with practical caution. Underdog’s benchmarks, while promising, use a specific test harness, and independent evaluations remain limited. Always consult the model card for guidance on optimal performance, such as disabling “thinking” and setting temperature to zero for tool-specific calls.
Ultimately, Saluki demonstrates a fascinating trade-off: unparalleled access to a 27B model on consumer hardware, but with a narrow, specialized focus. Understand its limitations to leverage its strengths effectively.
Frequently Asked Questions #
What is Underdog Saluki 27B?
It is a heavily quantized Qwen 27B model, about 7.9GB in size, tuned to prioritize tool calling.
Can Saluki 27B run on a 12GB GPU?
It can fit on some 12GB GPUs with an appropriate runtime and settings, but available memory, context length, and GPU offload affect performance.
Is Saluki better than the original Qwen 27B?
It scored higher on one reported tool-calling benchmark, but performed worse on math and some coding tests. Results also need independent verification.
Does a smaller model file mean faster agent runs?
No. Smaller weights reduce memory needs, but prefill and other agent-loop bottlenecks can still make execution slow.