Local LLMs for Agentic Coding: A Practical Guide for Modest Hardware

In just 20 minutes, you can have your local AI engine and agent ready to serve.

This guide focuses on two highly efficient engines: llama.cpp and MTPLX. I'm not affiliated with either project. I discovered these through extensive experimentation with Ollama and oMLX. While Ollama and oMLX are excellent tools, I found llama.cpp and MTPLX to be more stable for my specific workflow. Feel free to pick whichever engine you prefer. This guide remains applicable regardless of your choice.

Llama.cpp offers broad hardware flexibility across macOS, Linux, and Windows, while MTPLX is optimized solely for Apple Silicon.

Note on testing methodology: I benchmarked all models on a MacBook M4 Pro with 48 GB of RAM. GPU figures are based on community research and documentation, not personal benchmarks. Speed figures are from my M4 Pro. NVIDIA/AMD owners should treat the speed table as indicative and re-tune quant/context for their VRAM; for smaller GPUs (e.g., 8 GB), you will need more aggressive quantization.

Choosing the right inference engine is critical for agentic workflows. Unlike standard chat interfaces where a single slow response is merely an inconvenience, an agent relies on a continuous loop of reasoning, planning, and execution. In this context, even slight fluctuations in stability or speed can disrupt the entire logic chain. This is why I settled on llama.cpp and MTPLX. They provide the consistent performance required to maintain "agentic momentum" during complex tasks.

The difference between llama.cpp and MTPLX.

Llama.cpp is a bit slower than MTPLX, but offers superior memory efficiency and cross-platform versatility. For example, it allows me to run a 5-bit quantized Qwen3.8–27B with a 70k context window.

In contrast, MTPLX is optimized specifically for Apple Silicon to deliver faster performance. But on MTPLX, the 4-bit quant of Qwen3.8–27B can consume the same amount of RAM as the 5-bit quant on llama.cpp.

Installation can be handled via the respective official sites for llama.cpp and MTPLX.

Models

When selecting a model, my primary strategy is to estimate its memory footprint. As a rule of thumb, the model size should be roughly half your available RAM. With 48 GB of RAM, I typically target ~20 GB models. While it is possible to push up to 30 GB models, you must limit the context window to under 32K tokens to maintain stability. This is critical for agentic workflows. The Pi coding agent can easily consume ~10K tokens on the first prompt due to skills and AGENTS.md processing. Context can increase by nearly twofold when using Claude Code.

If you are working with 16 GB or 24 GB of RAM, focus on 9B to 14B parameter models like Qwen3.5-9B and Qwen3-14B. During the development of DocScan (detailed in my recent article), I found that ~70K tokens per task was more than enough for this mid-range application.

For heavy-duty coding and complex logic, the Qwen and Gemma families are a top-tier choice.

Recommended Models

Speed

Model Architectures

Models come in different quants. The best analogy for a quant is a movie resolution: a 16-bit model is a 4K movie, 3-bit is 720p, 1-bit is 320p. You can still get the same emotional palette if you like cinema and have enough imagination.

To run an optimized server, use the following command as a baseline:

  llama serve \
    --models-preset ~/{your path}/models.ini \
    --flash-attn on \
    --parallel 2 \
    -ngl all \
    --threads 10 --threads-batch 12 \
    --cache-type-k q5_1 --cache-type-v q5_1 \
    --load-mode mmap+mlock \
    --jinja \
    --props --metrics \
    --chat-template-kwargs '{"preserve_thinking": true}' \
    --sleep-idle-seconds 300 \
    --models-max 1

🚀 Performance & Speed

🧠 Memory Management

⚙️ Model Behavior

Download model GGUFs from Hugging Face, directly or via the HF CLI.

Once you have downloaded your models, you need to register them in a models.ini file. This lets you switch between them quickly. You'll find temp, top-p, top-k, and other parameters on an appropriate model page.

Here is an example of how to structure your models.ini:

version = 1

[*]
c = 76800
spec-type = ngram-mod

[Qwen3.6-35B-A3B-MTP]  
model = {your model path}/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf  
mmproj = {your mmproj path}/mmproj-F16.gguf  
spec-type = draft-mtp  
spec-draft-n-max = 2  
rea = on  
reasoning-format = deepseek  
temp = 1.0  
top-p = 0.95  
top-k = 20  
min-p = 0.0  
presence-penalty = 1.5  
repeat-penalty = 1.0 

[Qwen3.8-27B]  
model = {your model path}/Qwen3.8-27B-UD-Q5_K_XL.gguf  
mmproj = {your mmproj path}/mmproj-F16.gguf  
rea = on  
reasoning-format = deepseek  
spec-type = ngram-mod  
spec-draft-n-max = 5  
spec-draft-p-min = 0.6  
temp = 1.0  
top-p = 0.95  
top-k = 20  
min-p = 0.0  
presence-penalty = 0.0  
repeat-penalty = 1.0

One note: don't be confused with reasoning-format = deepseek for qwen model. In llama.cpp, the reasoning-format controls how the server parses and returns the model’s chain-of-thought output. The name “deepseek” comes from the fact that the message.reasoning_content API field was first introduced by DeepSeek’s hosted API.

🚀 Speed & Optimization (Speculative Decoding)

🧠 Reasoning & Logic (Chain of Thought)

🎨 Creativity & Randomness (Sampling)

Backend: Use CUDA for NVIDIA (fastest) or ROCm for AMD. Vulkan is the best cross-vendor option.

VRAM & Sizing: To avoid slow PCIe transfers, ensure the model + KV cache fits entirely in VRAM.

Optimization: Use -ngl all (or 99) to offload all layers to the GPU. If the model spills into system RAM, speed will drop significantly. When in doubt, use a smaller quantization.

Speed vs. capacity: A high-end NVIDIA GPU such as the RTX 4090 is often faster for token generation when the model and its working memory fit in VRAM. Macs can accommodate much larger models in high-memory configurations because CPU and GPU share unified memory, but model capacity and generation speed are separate trade-offs.

MTPLX has a friendly GUI and step-by-step setup.

Two things I changed in the settings:

Decreasing the context window in the server settings doesn’t decrease the model capacity. So it doesn’t mean the model will be more unreliable when it reaches more than a half context boundary.

MTPLX includes an embedded model picker and supports importing models via a direct link. You can find models on Hugging Face that fit your configuration. Always check the file size by clicking “View on HF” to ensure it fits your hardware.

Rule: The model size should be roughly half your RAM.

I use Pi, but you can choose any orchestrator: OpenCode, Vibe from Mistral, Claude Code, Codex, etc. The key step is configuring the backend endpoint.

For Pi configuration: Set the baseUrl in ~/.pi/agent/models.json.

~/.pi/agent/models.json example:

{  
   "providers": {  
      "llamacpp": {  
         "api": "openai-completions",  
         "apiKey": "llama",  
         "baseUrl": "http://127.0.0.1:8080/v1",  
         "compat": {  
            "supportsDeveloperRole": true,  
            "supportsReasoningEffort": true  
         },  
         "models": [  
            {  
               "contextWindow": 76800,  
               "id": "Qwen3.6-35B-A3B-MTP",  
               "input": [  
                  "text",  
                  "image"  
               ],  
               "name": "Qwen3.6 35B-A3B MTP Thinking",  
               "reasoning": true  
            },  
            {  
               "contextWindow": 76800,  
               "id": "Qwen3.8-27B",  
               "input": [  
                  "text",  
                  "image"  
               ],  
               "name": "Qwen3.8 27B",  
               "reasoning": true  
            }
         ]  
      },  
      "mtplx": {  
         "api": "openai-completions",  
         "apiKey": "mtplx",  
         "authHeader": true,  
         "baseUrl": "http://127.0.0.1:8000/v1",  
         "models": [  
            {  
               "contextWindow": 76800,  
               "id": "Youssofal--Qwen3.8-27B-MTPLX-Optimized-Speed",  
               "input": [  
                  "text",  
                  "image"  
               ],  
               "name": "Qwen3.8-27B-MTPLX",  
               "reasoning": true  
            },  
            {  
               "contextWindow": 76800,  
               "id": "Youssofal--Qwen3.6-35B-A3B-MTPLX-Optimized-Speed",  
               "input": [  
                  "text",  
                  "image"  
               ],  
               "name": "Qwen3.6-35B-A3B-MTPLX",  
               "reasoning": true  
            }  
         ]  
      }  
   }  
}

Verify it's alive. Open a terminal and hit the server on port 8080 (llama.cpp) or 8000 (MTPLX):

curl http://127.0.0.1:8080/v1/models

If you see your model IDs in the JSON response, the backend is serving. You can also open http://127.0.0.1:8080 in your browser. The built-in WebUI gives you a quick chat window to sanity-check a response.

The setup is complete, and your system is ready.

Understand Latency Stages (Prefill vs. Decode): Be patient during the initial request. Every response has two phases: prefill and decode.

Respect the Memory Ceiling: Your context window must fit within your available RAM. For a 48 GB system, aim for roughly 75K tokens. For extremely long tasks, use an extension like plannotator or break the work into a structured plan.md file to be implemented across multiple sessions.

Monitor Resource Pressure: Keep a system monitor (Activity Monitor on macOS, htop on Linux, or Task Manager on Windows) open at all times. Watch for rising memory pressure, which is a leading indicator of performance degradation.

Minimize Prompt Complexity: Keep your "skills" targeted. Instead of one massive skill set, use specific ones (e.g., Code Search, Software Engineering). For domain-specific logic (Jetpack Compose, SwiftUI, Flutter, RN), move those instructions to the project level to keep the agent's context clean and modular.

Keep AGENTS.md Granular: Avoid bloated instruction files. Keep individual AGENTS.md files under ~100 lines. In Feature-First Modular Architectures, utilize separate AGENTS.md files within specific feature or layer directories to maintain precision.

Use RAG to Mitigate Hallucinations: Retrieval-Augmented Generation (RAG) is essential for providing the model with external data, but it is not a silver bullet. LLMs can still "hallucinate" plausible-looking methods, mismatched versions, or fused class names.

Adopt a Multi-Stage Workflow: For complex tasks, use a specialized three-stage approach: Planning (using a high-precision dense model), Coding (using a high-speed MoE model), and Review (returning to a dense model for verification).

I used local models to build the document detection project from my recent article. Now, I’ll add a new feature: exporting scan results to a file.

Goal: Add a file export feature.

Scope: A basic prompt to test what the current setup can handle.

make a plan
Feature: save scan result to a PDF file
ScanResult screen should show a "save to file" button when it has a scan result
Review whether it's possible to save a file without runtime permissions or any other unnecessary user interaction. Check if there's a convenient API.
Use web search for this.
put a plan to save-scan-plan.md

The Outcome: The feature works, but the architecture is slightly off. The PdfExporter.kt ended up in the presenter layer.

Next prompt:

1. New: presenter/filescan/PdfExporter.kt (or data/utils per layering)
it should be data/utils
make an updated plan in save-scan-plan-mod-1.md

The plan file has been regenerated to match the new scope, but this is mostly for educational purposes—you don’t need to commit a new plan version every time.

Why? I didn’t emphasize the clean-arch layers in the previous prompt. If I had specified something like "use the DocumentDetectorRepository data flow," the placement would have been correct from the start.

Last prompt:

make a note that there should be no direct dependency between data and presenter layers
it should work through domain
make an updated plan in save-scan-plan-mod-2.md

This task took about an hour: 30 minutes to implement and another 30 minutes to polish, double-check the agent, and test. While MTPLX would be significantly faster, the result at 10-15 tps (planning) and ~60 tps (implementation) is excellent. You’ve just gotten a local-storage feature with additions across all architecture layers and fallback compatibility for an hour of work.

What went well:

Areas for improvement:

...
val pdf = PdfDocument()
val page = pdf.startPage(PdfDocument.PageInfo.Builder(width, height, 1).create())
...
val bytes = ByteArrayOutputStream()
pdf.writeTo(bytes)
pdf.close()

The next prompt found all issues:

review Bitmap.toPdf unhandled errors

The fix took only 1 minute and 3 seconds.

Just one button at first glance. But under the hood, it is a complete feature: PDF export with its full architectural layers (IScanPdfRepository, ScanPdfRepositoryImpl, and PdfExporter), robust file saving with API fallback logic, comprehensive state handling, runtime permissions, Koin bindings, documentation updates, and unit tests. A task that would traditionally take 3-4 hours was completed in a fraction of the time.

At its core, running local LLMs means shifting the intelligence from a distant cloud server directly onto your own hardware. While hosted services are powerful, local solutions offer three critical advantages that cloud providers cannot match: Control, Privacy, and Flexibility.

The Potential is Limitless

Once you have established this local intelligence layer, the applications are vast. It can power seamless IDE integration, automate document summarization, or serve as the core engine for your Personal Knowledge Management system. From assisting with emails and browser tasks - such as tracking shopping items, analyzing real-estate listings, or curating daily news digests. The only limit is how you choose to implement it.