{"slug": "the-stack-under-the-model-omlx-llama-cpp-hermes-and-why-there-are-so-many", "title": "The Stack Under the Model: oMLX, llama.cpp, Hermes, and Why There Are So Many", "summary": "Retort, a framework for comparing coding stacks, reveals that a model's performance depends on the entire stack—language, weights format, serving engine, agent, context engine, sampling, and prompt—not just the model itself. The project maps the layers of the stack, including oMLX, llama.cpp, and Hermes, and introduces a metaharness that treats the agent layer as a variable to be measured.", "body_md": "[Retort](https://github.com/adrianco/retort) is a framework for comparing coding stacks, with results for versions of Claude and local tests on a 64GB M5Pro for many languages. It was developed with Claude, and I was getting confused by all the components of the harness and stack, so I asked Claude to explain what is out there and how they are being tested. That's what follows:\n\nMost benchmarks answer \"which *model* is best?\" Retort insists that's the wrong\n\nunit. A coding result is produced by a whole **stack** — and the model is only\n\none layer of it:\n\nlanguage × model × weights-format × serving engine × agent/harness × context engine × sampling × prompt\n\nChange any layer and the numbers move. TypeScript went from failing to 1.00 on\n\nthe local 80B by raising *one context-engine knob* — same model, same weights.\n\nSo before the results make sense, you need a map of the layers. This post is that\n\nmap: what each piece is, where it came from, what competes with what, how they\n\nstack, and why the zoo is so crowded. Then it explains the **metaharness** — the\n\npart of Retort that turns the harness *itself* into a variable you can measure.\n\nIf you've seen `oMLX`\n\n, `llama.cpp`\n\n, `Hermes`\n\n, `GGUF`\n\n, `omp`\n\n, `lcm`\n\n, or\n\n`OpenRouter`\n\nfly past in the other blogs and nodded along without quite knowing\n\nwhat they are, start here.\n\nRead this from the bottom up — each layer sits on the one below it:\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  PROMPT / METHODOLOGY   \"write tests first\" · BDD · terse    │  ← how you ask\n├──────────────────────────────────────────────────────────────┤\n│  AGENT / HARNESS        claude-code · Hermes · gemini · omp  │  ← the ReAct loop:\n│                         opencode                             │    read → edit → run → repeat\n├──────────────────────────────────────────────────────────────┤\n│  CONTEXT ENGINE         lcm (compaction) · sampling knobs    │  ← what the model \"sees\"\n├──────────────────────────────────────────────────────────────┤\n│  SERVING ENGINE         oMLX · llama.cpp · Ollama · cloud API│  ← turns weights into tokens\n├──────────────────────────────────────────────────────────────┤\n│  WEIGHTS + FORMAT       safetensors · GGUF · MLX  (a model   │  ← the numbers on disk\n│                         at some quantization, e.g. 4-bit)    │\n├──────────────────────────────────────────────────────────────┤\n│  HARDWARE               Apple Silicon · NVIDIA · a cloud GPU │  ← what runs the math\n└──────────────────────────────────────────────────────────────┘\n```\n\n\"Which model is best\" only asks about one band in the middle. Retort measures the\n\ncolumn. The rest of this post walks the layers where the confusing names live —\n\n**weights, serving, agent, context** — and then the metaharness, which makes the\n\n*agent* layer a factor you can sweep.\n\nA model is a big pile of numbers (weights). How those numbers are *stored on\ndisk* is a surprising source of fragmentation, because the file format is tied to\n\n`Q4_K_M`\n\n≈ 4-bit). Designed to run well on CPUs and non-CUDA GPUs.**Quantization** is the other axis here: the same model shipped at 4-bit is a\n\nquarter the size of 16-bit and runs far faster, at some accuracy cost. When you\n\nsee `Qwen3-Coder-Next-4bit`\n\n(≈42 GB) that's \"the 80B model, MLX format, 4-bit.\"\n\nWhy two local formats (GGUF and MLX) for the same models? Because each is glued\n\nto a different engine, which is the next layer.\n\nThe serving engine is the program that loads the weights and turns your prompt\n\ninto tokens, usually exposing an **OpenAI-compatible HTTP endpoint** (`POST`\n\n) so anything that speaks that protocol can drive it. This\n\n/v1/chat/completions\n\nis the layer with the most competitors, because it's where hardware, format, and\n\nperformance all collide.\n\n`llama-server`\n\n; Retort's `llamacpp`\n\nbackend drives it with\n`--jinja`\n\nso the model's own chat/tool template is applied.`ollama run qwen`\n\n). It made local models one-command easy and is how most\npeople first run one. Retort's earliest local attempt used it (see below) and\nmoved on.`127.0.0.1:8080`\n\n. It's the MLX-world counterpart to\nllama.cpp: same job (weights → tokens over HTTP), different format and a\nMetal/MLX-native path tuned for Macs. Retort restarts it at each experiment\nboundary and records the `provenance.json`\n\n.**What competes with what:** llama.cpp (GGUF) and oMLX (MLX) are the two serious\n\n*local* engines, split mostly by format and hardware heritage — GGUF/llama.cpp is\n\nthe cross-platform default, MLX/oMLX is the Apple-Silicon-native path. Ollama\n\ncompetes on *ease*, not capability (it *is* llama.cpp underneath). Retort now\n\nsupports **both** local engines via a `serving.backend: omlx | llamacpp`\n\nswitch,\n\nprecisely because neither dominates: some models ship only as MLX, some only as\n\nGGUF, and some new architectures land in one engine months before the other.\n\nThat \"which engine has this architecture yet?\" gap is a recurring reason a\n\npromising model can't be tested — it's a serving-layer problem, not a model one.\n\nA served model just answers messages. To *build software* it needs a loop that\n\nreads files, writes edits, runs the tests, reads the failures, and tries again —\n\nthe **agentic loop** (often \"ReAct\": reason → act → observe, repeat). That loop is\n\nthe **agent / harness**, and it's a real, swappable layer: the same model behaves\n\ndifferently under different harnesses because they differ in how they present\n\ntools, when they retry, and how they manage the conversation.\n\nThe harnesses Retort supports:\n\n`claude-code`\n\n`gemini`\n\n`gemini-*`\n\nmodels. The agent usually `Hermes`\n\n`lcm`\n\n, below) while oMLX serves the tokens. `opencode`\n\n`omp`\n\n**Why the agent is its own layer (and its own zoo):** each vendor ships the\n\nharness tuned for its own model, and open harnesses exist to run *any* model. They\n\ngenuinely differ — a weak model can pass under a forgiving harness and fail under\n\na strict one — which is exactly why \"which harness?\" deserves measurement rather\n\nthan assumption. That's what the metaharness (last section) is for.\n\nTwo smaller layers sit between the agent and the model, and they've caused more\n\nwrong conclusions in this project than any model choice.\n\n`lcm`\n\n)`lcm`\n\n, and its key\nknob is `context_threshold`\n\n— the fraction of the window at which it compacts.\nAt the default `0.35`\n\nit compacts at ~92K tokens and the 80B intermittently\n`0.9`\n\n(\"full context\") the same model runs Python, Go, `temperature`\n\n, `top_p`\n\n, `top_k`\n\n, `repetition_penalty`\n\n: how\nrandomly the next token is picked. Retort learned the hard way that the oMLX\n`temperature = 1.0`\n\nroughly halved local reliability, and that a\n`repetition_penalty ≠ 1.0`\n\n(even a value a model's own card recommends) can\nderail the multi-turn tool loop into stalls. These are recorded per run and held\nfixed on purpose.The lesson threaded through the whole project: **a set-but-unverified knob in one\nof these quiet layers produces confident, wrong results.** Retort's provenance\n\nThe zoo isn't accidental. Several independent forces each spawn alternatives:\n\n| Force | What it splits | Example |\n|---|---|---|\nHardware lineage |\nserving engine + format | Apple-Silicon/unified-memory → MLX/oMLX; cross-platform/CUDA → GGUF/llama.cpp |\nFormat lock-in |\nwhich engine can even load a model | a model published only as MLX can't run on llama.cpp, and vice-versa |\nNew architectures |\nwhich engine supports them first\n|\na brand-new model's arch may land in one engine months before the other — the usual reason a candidate is \"blocked\" |\nEase vs control |\nwrapper vs raw engine | Ollama (easy) wraps llama.cpp (control); one command vs every flag |\nVendor vs open |\nthe agent/harness | claude-code/gemini ship per-vendor; Hermes/opencode/omp run any model |\nCost & privacy |\nlocal vs cloud | a 64 GB Mac runs the 80B for \\$0 and offline; the cloud rents frontier capability per token |\nQuantization |\nsize vs accuracy | the same model at 4-bit / 6-bit / 8-bit trades RAM and speed against quality |\n\nNone of these layers *dominates*, so none of the others disappear. That's the\n\nwhole reason Retort measures the stack instead of the model: the interesting\n\nquestion is usually \"which *combination* is best for **my** language, task, and\n\nbudget?\" — and the answer moves as new engines, formats, and harnesses land.\n\n**Where Retort's featured stacks sit today:** the free local path is\n\n**Qwen3-Coder-Next 80B, MLX 4-bit, served by oMLX, driven by Hermes with lcm at\ncontext_threshold 0.9** on a 64 GB Mac — Python/Go/TypeScript at 1.00 for \\$0.\n\nEverything above treats the harness as a fixed choice per run. But the harness is\n\na bundle of *strategies* — and those strategies plausibly move results as much as\n\nthe model does. The layer\n\n`retort_metaharness`\n\nIt doesn't reinvent Retort — it composes the same design generator, ANOVA, and\n\nPareto engine — but it adds three new factors on top of the usual language/model:\n\n`harness_config`\n\n— the orchestration strategy (the headline factor)\n| level | what it does |\n|---|---|\nbase-ReAct |\nplain single-agent reason→act→observe loop. The control: no tricks. |\nself-consistency-N |\nsample N independent solutions and majority/judge-select the best. An accuracy lever that costs N× the tokens (default N=5). |\nrouted |\na cheap model drafts; a frontier model takes over only on low-confidence steps. A cost lever — aims for comparable reliability at lower \\$. |\n+agenticow-memory |\nReAct plus a copy-on-write memory that persists agent state across steps/replicates — tests whether memory changes the outcome. |\n+darwin-evolved-genome |\nReAct driven by a harness \"genome\" (prompt + tool policy) tuned by an evolutionary loop — tests whether evolution moves the needle. |\n\n`scaffold`\n\n— the reasoning structure wrapped around each attempt\n`none`\n\n· `plan-and-solve`\n\n(plan first, then execute) · `reflexion`\n\n(attempt,\n\nself-critique, retry).\n\n`model`\n\n— the raw model, spanning cheap→frontier\n`deepseek-v4-pro`\n\n· `glm-5.2`\n\n· `opus-4.8`\n\n· `gpt-5.2`\n\n, reached through\n\n**OpenRouter** (a unified API that routes one request format to many providers),\n\nchosen to spread from cheap to frontier so the analysis can *separate* a model\n\neffect from a harness effect.\n\nBecause a fractional-factorial design crosses **all** of these at once (rather than\n\nchanging one thing at a time), the ANOVA can attribute the variance in a metric to\n\n**model vs harness vs scaffold vs language + their interactions**, and report which\n\neffects are cleanly estimated versus confounded. Concretely, it answers questions\n\nthe model-only grid structurally can't:\n\n`+agenticow-memory`\n\n, how much is the memory branching versus\njust the underlying model being good?`routed`\n\nactually hold reliability while cutting cost — i.e., does it sit\non the accuracy-vs-\\$ Pareto front?`self-consistency-5`\n\nworth 5× the tokens, or a rounding error on a task the\nmodel already nails?`scaffold`\n\na real lever or a ritual — and does that depend on\nmodel strength?The honest caveats: it's **cloud-only** (OpenRouter, metered — the\n\n`self-consistency × frontier × replicates`\n\ncorner gets expensive) and the real\n\norchestration logic lives in an **external solver** the adapter shells out to, so\n\nwithout that solver only a \\$0 stub runs. That's why it's a documented\n\nside-branch rather than a headline result — it stays one until a first screening\n\nrun shows the harness variance is real enough to promote. The staged plan lives in\n\n[ docs/future-experiments.md](https://docs/future-experiments.md).\n\nThere are \"so many\" harnesses and engines because at least six independent forces\n\n— hardware, format, new architectures, ease-vs-control, vendor-vs-open, and\n\ncost-vs-privacy — each keep their own alternatives alive, and no layer has a\n\nwinner that retires the others. The practical consequence is the thesis of this\n\nwhole repo: **don't benchmark the model, benchmark the stack.** The model is one\n\nband in a tall column, and the quiet layers under and around it — the serving\n\nengine, the context threshold, the sampling defaults, and the orchestration\n\nstrategy — routinely decide the result. The metaharness is Retort turning the last\n\nof those, the orchestration layer, from an assumption into a measurement.", "url": "https://wpnews.pro/news/the-stack-under-the-model-omlx-llama-cpp-hermes-and-why-there-are-so-many", "canonical_source": "https://dev.to/adrianco_54/the-stack-under-the-model-omlx-llamacpp-hermes-and-why-there-are-so-many-k75", "published_at": "2026-07-22 22:49:51+00:00", "updated_at": "2026-07-22 23:29:55.646417+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Retort", "Claude", "oMLX", "llama.cpp", "Hermes", "GGUF", "Ollama", "OpenRouter"], "alternates": {"html": "https://wpnews.pro/news/the-stack-under-the-model-omlx-llama-cpp-hermes-and-why-there-are-so-many", "markdown": "https://wpnews.pro/news/the-stack-under-the-model-omlx-llama-cpp-hermes-and-why-there-are-so-many.md", "text": "https://wpnews.pro/news/the-stack-under-the-model-omlx-llama-cpp-hermes-and-why-there-are-so-many.txt", "jsonld": "https://wpnews.pro/news/the-stack-under-the-model-omlx-llama-cpp-hermes-and-why-there-are-so-many.jsonld"}}