{"slug": "running-glm-5-2-locally-with-rondine-and-pi", "title": "Running GLM-5.2 Locally with Rondine and Pi", "summary": "Rondine, a tool that detects hardware and configures inference engines, enables running GLM-5.2, a 744B-parameter Mixture-of-Experts coding model requiring approximately 245GB of memory, locally on a 256GB Mac Studio-class machine. The setup uses Rondine with llama.cpp for inference and Pi for the coding-agent loop, with a practical 32K coding context. Rondine automates decisions like quantization selection, memory allocation, and context length, making the complex deployment reproducible.", "body_md": "# Running GLM-5.2 Locally with Rondine and Pi\n\nIn my previous article, [The Local Agent Trio: cmux + Pi + Unsloth Studio](/2026/cmux-pi-unsloth-local-glm-setup/), I split the local coding stack into three replaceable layers:\n\n**cmux** for the terminal workspace**Unsloth Studio** for inference**Pi** for the coding-agent loop\n\nThat architecture still makes sense. But there is another way to handle the inference layer—especially when you want the configuration to come from the hardware rather than a model picker.\n\nThat is what [Rondine](https://github.com/antonellof/rondine) does.\n\nRondine detects your machine, checks whether a model fits, selects an inference engine and quantization, applies hardware-specific settings, downloads the weights, and starts an OpenAI-compatible server.\n\nFor this setup, the stack becomes:\n\n| Layer | Tool | Purpose |\n|---|---|---|\n| Workspace | cmux, optionally | Terminal panes and agent notifications |\n| Inference | Rondine + llama.cpp | Hardware planning, model download, and serving |\n| Agent | Pi | Repository tools and coding loop |\n\nThe interesting target is **GLM-5.2**: a frontier-scale Mixture-of-Experts coding model that requires approximately 245GB of memory even at 2-bit quantization.\n\n## The GLM-5.2 hardware reality\n\nRondine’s catalog describes GLM-5.2 as a 744B-parameter MoE model with approximately 40B active parameters per token and support for a context window of up to one million tokens.\n\nThe active parameter count reduces computation, but it does not eliminate the need to store the complete model.\n\nRondine currently provides these configurations:\n\n| Variant | Approximate size | Intended hardware |\n|---|---|---|\n`UD-IQ1_S` |\n223GB | Aggressive compression |\n`UD-IQ2_M` |\n239GB | Recommended 2-bit balance |\n| Official BF16 | ~1.5TB | Large multi-GPU or multi-node system |\n\nFor a single machine, Rondine prefers the Unsloth `UD-IQ2_M`\n\nGGUF and requires at least 245GB of usable memory.\n\nIn practice, this means a **256GB Mac Studio-class machine**. Even then, context length matters. A model may fit at a small context and fail once the KV cache, batching, and operating-system memory are included.\n\nFor that reason, Rondine uses a practical **32K coding context** instead of blindly enabling the model’s theoretical maximum.\n\n## Why use Rondine?\n\nA raw GLM-5.2 llama.cpp launch requires several decisions:\n\n- Which quantization should I download?\n- Does it fit in unified memory?\n- How much memory should remain available to macOS?\n- What context length is realistic?\n- Should the KV cache use F16, Q8, or something smaller?\n- Should all layers be offloaded to Metal?\n- Which batch and micro-batch sizes should I use?\n- Should flash attention be enabled?\n- How many parallel slots should the server expose?\n\nRondine turns those decisions into a reproducible plan.\n\nIt is not another inference engine. It installs and drives llama.cpp, MLX-LM, or vLLM while keeping the generated command visible.\n\n## Install Rondine\n\nRondine requires Python 3.11+ and `uv`\n\n:\n\n```\ngit clone https://github.com/antonellof/rondine.git\ncd rondine\n\nuv tool install .\n```\n\nStart by inspecting the machine:\n\n```\nrondine doctor\n```\n\nOn a compatible high-memory Mac, the output should identify Apple Silicon, unified memory, and available engines.\n\nYou can then ask Rondine to plan GLM-5.2 directly:\n\n```\nrondine plan glm-5.2 \\\n  --profile coding \\\n  --save-as glm-coding\n```\n\nGLM-5.2 is marked as opt-in because of its size. Explicitly naming it opts into planning it, but Rondine still rejects the plan if the detected hardware cannot satisfy the memory estimate.\n\nBefore downloading hundreds of gigabytes, inspect the generated launch:\n\n```\nrondine serve --preset glm-coding --dry-run\n```\n\nThis prints the resolved llama.cpp command without starting it.\n\n## What the coding configuration contains\n\nFor GLM-5.2, Rondine’s coding profile applies:\n\n```\ncontext:           32768\ntemperature:       1.0\ntop_p:             0.95\nmin_p:             0.01\nthinking:          enabled\nreasoning effort:  max\nparallel slots:    1\n```\n\nOn Apple Silicon, the llama.cpp template also enables:\n\n- Full Metal offload with\n`-ngl 99`\n\n- Flash attention\n- Continuous batching\n- Q8 KV caches\n- Large batch and micro-batch sizes where memory permits\n`mlock`\n\nto reduce unwanted swapping- A single coding slot so context is not divided among concurrent clients\n\nThe effective command remains visible through the dry run, which is useful when experimenting or reporting a problem upstream.\n\n## Download GLM-5.2\n\nOnce the plan looks correct:\n\n```\nrondine setup --engine llama.cpp\nrondine pull glm-5.2\n```\n\nThe preferred quantization is downloaded from:\n\n```\nunsloth/GLM-5.2-GGUF\n```\n\nwith the pattern:\n\n```\n*UD-IQ2_M*\n```\n\nThis is a multi-hundred-gigabyte download. Plan for enough disk space beyond the model itself, use a stable connection, and remember that loading it will also require memory for context, cache, and the operating system.\n\nIf 239GB is too large, `UD-IQ1_S`\n\nreduces the weights to approximately 223GB, but that is a more aggressive quality compromise.\n\n## Start the OpenAI-compatible server\n\nLaunch the saved preset:\n\n```\nrondine serve \\\n  --preset glm-coding \\\n  --name glm\n```\n\nRondine starts llama.cpp on:\n\n```\nhttp://127.0.0.1:8080/v1\n```\n\nThe model alias exposed to clients is:\n\n```\nrondine/glm-5.2\n```\n\nConfirm it through the models endpoint:\n\n```\ncurl http://127.0.0.1:8080/v1/models\n```\n\nThen run Rondine’s coding verification:\n\n```\nrondine verify \\\n  --name glm \\\n  --profile coding\n```\n\nThe verification checks:\n\n- Server readiness\n- The\n`/v1/models`\n\nendpoint - A short code-generation request\n- A best-effort OpenAI-format tool-call request\n\nThis distinction matters: a process listening on port 8080 does not necessarily mean the model, chat template, and tool-calling path are working correctly.\n\n## Test GLM-5.2 directly\n\nBefore adding Pi, make one direct request:\n\n```\ncurl http://127.0.0.1:8080/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"rondine/glm-5.2\",\n    \"messages\": [\n      {\n        \"role\": \"user\",\n        \"content\": \"Write a typed Python binary search function. Return code only.\"\n      }\n    ],\n    \"max_tokens\": 512\n  }'\n```\n\nIf this works, the inference layer is ready. Pi becomes a client of the same endpoint.\n\n## Install Pi\n\nPi is a minimal coding agent with four core tools:\n\n`read`\n\n`write`\n\n`edit`\n\n`bash`\n\nInstall it with:\n\n```\nnpm install -g --ignore-scripts \\\n  @earendil-works/pi-coding-agent\n```\n\nPi can connect to custom OpenAI-compatible providers through:\n\n```\n~/.pi/agent/models.json\n```\n\nAdd Rondine as a provider:\n\n```\n{\n  \"providers\": {\n    \"rondine\": {\n      \"baseUrl\": \"http://127.0.0.1:8080/v1\",\n      \"api\": \"openai-completions\",\n      \"apiKey\": \"rondine\",\n      \"models\": [\n        {\n          \"id\": \"rondine/glm-5.2\",\n          \"name\": \"GLM-5.2 local via Rondine\",\n          \"contextWindow\": 32768\n        }\n      ]\n    }\n  }\n}\n```\n\nMost local llama.cpp servers do not validate the API key, but Pi expects a non-empty value.\n\nUse the exact model identifier returned by:\n\n```\ncurl http://127.0.0.1:8080/v1/models\n```\n\n## Start coding with Pi\n\nMove into a repository and launch Pi:\n\n```\ncd ~/Projects/my-application\npi\n```\n\nInside Pi, open the model selector:\n\n```\n/model\n```\n\nChoose:\n\n```\nGLM-5.2 local via Rondine\n```\n\nPi now supplies the agent loop while Rondine owns the inference lifecycle.\n\nA useful first prompt is something constrained and verifiable:\n\n```\nRead the project structure and test configuration.\n\nFind one untested error path in the HTTP client, add a focused test,\nrun only the relevant test suite, and show me the resulting diff.\nDo not modify production code unless the test reveals a real defect.\n```\n\nThis exercises reading, editing, shell execution, and multi-step reasoning without giving the model permission to refactor the entire repository.\n\n## Add cmux if you want the complete stack\n\ncmux remains optional, but it makes the workflow easier to observe.\n\nA practical layout is:\n\n**Pane 1:** Rondine server logs**Pane 2:** Pi running in the target repository**Pane 3:** Shell for health checks, Git, and memory monitoring\n\nStart Rondine in the foreground if you want logs attached to a pane:\n\n```\nrondine serve \\\n  --preset glm-coding \\\n  --name glm \\\n  --foreground\n```\n\nStart Pi in another:\n\n```\ncd ~/Projects/my-application\npi\n```\n\nThe architecture remains modular:\n\n```\ncmux\n  ├── Rondine\n  │     └── llama.cpp\n  │           └── GLM-5.2 UD-IQ2_M\n  └── Pi\n        └── http://127.0.0.1:8080/v1\n```\n\nSwap Pi for another OpenAI-compatible coding client and the inference layer stays unchanged. Swap GLM-5.2 for a smaller model and Pi’s workflow stays unchanged.\n\n## If GLM-5.2 does not fit\n\nMost developers do not have 256GB of unified memory. Rondine should reject GLM-5.2 on those machines—that is better than discovering the limitation after a 239GB download.\n\nUse the same workflow with a smaller model:\n\n```\nrondine suggest --profile coding\n```\n\nTypical alternatives include:\n\n- Qwen3.6 27B for 24–48GB Macs\n- Qwen3.6 35B-A3B for larger Macs or 24GB NVIDIA GPUs\n- Gemma 4 12B for smaller systems\n- DeepSeek-V4-Flash at 3-bit for approximately 128GB systems\n\nFor a quick end-to-end test, Rondine also includes Qwen2.5-Coder 3B:\n\n```\nrondine plan qwen2.5-coder-3b \\\n  --context 4096 \\\n  --save-as small-coder\n\nrondine pull qwen2.5-coder-3b\nrondine serve --preset small-coder\nrondine verify --name small-coder\n```\n\nThe Pi configuration is identical apart from the model ID.\n\nThis is the main advantage of separating the agent from inference: the workflow does not depend on one model fitting forever.\n\n## Stopping and restarting\n\nStop the managed GLM server with:\n\n```\nrondine stop --name glm\n```\n\nRestart the saved configuration later:\n\n```\nrondine serve \\\n  --preset glm-coding \\\n  --name glm\n```\n\nInspect it at any time:\n\n```\nrondine preset show glm-coding\n```\n\nThe tuned command is no longer something you have to recover from shell history.\n\n## Honest limitations\n\nThis setup does not make GLM-5.2 small.\n\nThe recommended quant still occupies approximately 239GB, and a 256GB Mac leaves limited headroom. Large contexts may require reducing batches, using a smaller context, or moving to hardware with more memory.\n\nRondine currently provides a hardware-aware configuration, not a published GLM-5.2 performance guarantee. Its documented benchmark is a smaller Qwen2.5-Coder run on an M2 Pro; that validates the plan-to-API path but should not be presented as evidence of GLM-5.2 throughput.\n\nTool calling also depends on the model, llama.cpp version, chat template, and streaming compatibility. Run `rondine verify`\n\n, test Pi on a disposable branch, and inspect the server logs before trusting a long autonomous session.\n\n## Bottom line\n\nThe local coding stack works best when its layers remain replaceable:\n\n**Rondine** chooses, configures, downloads, and serves the model.**llama.cpp** performs inference.**Pi** reads, edits, and runs commands in the repository.**cmux**, optionally, keeps the sessions visible.\n\nGLM-5.2 is the demanding case that makes hardware-aware planning valuable. On a compatible 256GB machine, Rondine turns a 239GB sharded GGUF and a page of llama.cpp flags into a named, inspectable preset exposed at a standard OpenAI URL.\n\nPi does not need to know how any of that works. It only needs:\n\n```\nhttp://127.0.0.1:8080/v1\n```\n\nThat is the useful abstraction: a very large local model underneath, a very small coding agent on top, and a standard API boundary between them.\n\n**Links**", "url": "https://wpnews.pro/news/running-glm-5-2-locally-with-rondine-and-pi", "canonical_source": "https://www.fratepietro.com/2026/running-glm-5-2-locally-rondine-pi/", "published_at": "2026-07-21 22:00:00+00:00", "updated_at": "2026-08-05 10:56:56.151465+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["Rondine", "GLM-5.2", "llama.cpp", "Pi", "Unsloth Studio", "cmux", "Apple Silicon", "Mac Studio"], "alternates": {"html": "https://wpnews.pro/news/running-glm-5-2-locally-with-rondine-and-pi", "markdown": "https://wpnews.pro/news/running-glm-5-2-locally-with-rondine-and-pi.md", "text": "https://wpnews.pro/news/running-glm-5-2-locally-with-rondine-and-pi.txt", "jsonld": "https://wpnews.pro/news/running-glm-5-2-locally-with-rondine-and-pi.jsonld"}}