{"slug": "running-laguna-s-2-1-locally-on-apple-silicon-52-tok-s-with-38-5-gb-peak-memory", "title": "Running Laguna S 2.1 locally on Apple Silicon: 52 tok/s with 38.5 GB peak memory", "summary": "The mlx-community/Laguna-S-2.1-oQ2e quantized model running in-process through mlx-vlm on a 128 GB Apple M5 Max achieved a perfect overall score of 1.000 across six tasks, with 40.85 generation tok/s and 37.77 GB peak MLX memory, finishing the suite in 87.38 seconds. The benchmark, sponsored by DWS LLC and published by developer tanishq-dubey, found oQ2e passed all 38 hidden assertions and all long-context retrieval tests through 256K tokens, outperforming the official NVFP4 MLX build and other tested paths.", "body_md": "This repository contains a reproducible local harness for comparing Laguna S 2.1 MLX quantizations on Apple Silicon. Each run records the raw output and task score along with token and prefill throughput, peak MLX memory, load time, package versions, model revision, and machine metadata.\n\nThis work was sponsored by [DWS LLC](https://dws.rip).\n\nOn the M5 Max used for these tests, the smallest quant, `mlx-community/Laguna-S-2.1-oQ2e`\n\n, running in-process through `mlx-vlm`\n\nwas the fastest option. It passed all six tasks and finished the suite faster than the tested llama.cpp, DFlash, official NVFP4 MLX, and serial oMLX paths. The measurements are in `BENCHMARK_RESULTS.md`\n\n.\n\nThese results were measured on a 128 GB Apple M5 Max using macOS 27.0, Python 3.13.12, MLX 0.32.0, and mlx-vlm 0.6.6. The score combines three generation tasks and three agentic coding tasks. Throughput is weighted across the generation tasks.\n\nThe left panel reproduces the Terminal-Bench 2.1 figures in [Poolside's model card](https://huggingface.co/poolside/Laguna-S-2.1#benchmark-results). The right panel comes from this repository's committed CSV. They use separate scales and are not the same evaluation. Regenerate the SVG with:\n\n```\nuv run --frozen laguna-bench chart\n```\n\n| Quant | Overall score | Generation | Agentic | Generation tok/s | Peak MLX GB | Suite time |\n|---|---|---|---|---|---|---|\n`mlx-community/Laguna-S-2.1-oQ2e` |\n1.000 | 1.000 | 1.000 | 40.85 | 37.77 | 87.38s |\n`mlx-community/Laguna-S-2.1-oQ3e` |\n0.625 | 0.417 | 0.833 | 48.58 | 50.69 | 84.79s |\n`poolside/Laguna-S-2.1-NVFP4-mlx` |\n0.875 | 0.750 | 1.000 | 7.25 | 73.47 | 301.77s |\n\nThe oQ2e conversion was the clear default for this machine. It passed all 38 hidden assertions while using about 38 GB of peak MLX memory. oQ3e decoded faster in the controlled profile, but its exact-format and implementation errors lowered its task score. The official NVFP4 MLX build was functional but much slower in this runtime.\n\nLong-context retrieval also passed at every tested size through 256K tokens on oQ2e:\n\n| Prompt tokens | Prefill tok/s | Decode tok/s | Peak MLX GB | Retrieval |\n|---|---|---|---|---|\n| 1,016 | 1411.42 | 60.86 | 37.23 | pass |\n| 16,376 | 1613.07 | 52.48 | 38.46 | pass |\n| 65,528 | 1088.75 | 32.03 | 41.19 | pass |\n| 131,064 | 771.80 | 23.48 | 45.09 | pass |\n| 262,136 | 566.31 | 12.16 | 52.87 | pass |\n\nFor a local coding setup, use oQ2e through in-process mlx-vlm with temperature 0, top-p 1, the default unquantized KV cache, and a prefill step of 2048. Staying at or below 64K gives a much better latency balance. The 128K and 256K cases fit and passed retrieval, but decode speed fell sharply.\n\nThe full [benchmark report](/tanishq-dubey/macos-laguna-s2.1/blob/main/BENCHMARK_RESULTS.md) includes the standardized quant profile, sampling search, KV cache and prefill search, engine bake-off, revisions, and compatibility failures. The machine-readable source is [the combined CSV](/tanishq-dubey/macos-laguna-s2.1/blob/main/results/laguna_s21_full_results.csv).\n\nThe helper creates the environment and uses the fastest quant from our tests by default:\n\n```\ngit clone https://github.com/tanishq-dubey/macos-laguna-s2.1.git laguna-s21-bench\ncd laguna-s21-bench\nscripts/laguna.sh download\nscripts/laguna.sh prompt 'Write a Python LRU cache with tests'\n```\n\nStart an interactive chat or a local OpenAI-compatible server:\n\n```\nscripts/laguna.sh chat\nscripts/laguna.sh server\n```\n\nThe server binds to `127.0.0.1:8080`\n\n. Set `LAGUNA_MODEL`\n\n, `LAGUNA_HOST`\n\n, or `LAGUNA_PORT`\n\nto change the defaults. For example:\n\n```\nLAGUNA_MODEL=mlx-community/Laguna-S-2.1-oQ3e scripts/laguna.sh server\n```\n\nThe suite has six fixed tasks:\n\n| Kind | Tier | What it measures |\n|---|---|---|\n| generation | small | exact structured-output instruction following |\n| generation | medium | single-function Python synthesis against hidden tests |\n| generation | large | larger algorithm/module synthesis against hidden tests |\n| agentic | small | inspect files, derive an answer, and write an artifact |\n| agentic | medium | diagnose and repair a tested Python bug |\n| agentic | large | implement a multi-file feature and satisfy tests |\n\nGeneration uses greedy decoding (`temperature=0`\n\n) with a fixed MLX seed. Each agent task starts in a fresh temporary workspace. The agent has access only to an allowlist of tools, and the harness records every turn. Task prompts and tests are versioned in source. Here, \"deterministic\" means that the decoding settings and fixtures are repeatable. Metal kernels and new model or runtime revisions can still change the results.\n\n`--agent-prompt-cache`\n\nenables experimental MLX KV reuse. It improves throughput within each turn, but mlx-vlm 0.6.6 changed a greedy Laguna trajectory during testing because Laguna mixes global caches with rotating sliding-window caches. The option stays off by default until cold and cached runs produce exactly the same output.\n\n```\nuv sync --extra dev --python 3.13 --locked\n```\n\nLaguna support currently comes from `mlx-vlm`\n\n, even though these are text-only models.\n\nList tasks and the curated quant ladder:\n\n```\nuv run --frozen laguna-bench list\n```\n\nRun the complete suite on the smallest quant:\n\n```\nuv run --frozen laguna-bench run \\\n  --model mlx-community/Laguna-S-2.1-oQ2e \\\n  --output results\n```\n\nRun a cheap smoke test first:\n\n```\nuv run --frozen laguna-bench run \\\n  --model mlx-community/Laguna-S-2.1-oQ2e \\\n  --task generation-small \\\n  --output results\n```\n\nEvery run creates `results/<timestamp>-<model>/run.json`\n\n, a readable `summary.md`\n\n, the raw generations, and the final agent workspaces. A failed task counts as a benchmark result and does not cause the CLI itself to fail. A model or runtime failure does.\n\nCompare the latest complete run for every tested model:\n\n```\nuv run --frozen laguna-bench compare --output results\n```\n\nRun the standardized short profile for a new quant or the complete context and hyperparameter matrix:\n\n```\nuv run --frozen laguna-bench sweep --model mlx-community/Laguna-S-2.1-oQ3e --profile quant\nscripts/laguna.sh community\n```\n\nThe full profile covers 256 to 262,144 input tokens, decodes from 64 to 1,024 tokens, three sampling configurations, uniform and TurboQuant KV cache options, a fixed KV window, and several prefill chunk sizes. The 256K case takes several minutes on the reference M5 Max. To export the catalog, task results, and performance records to one CSV, run:\n\n```\nuv run --frozen laguna-bench export --output results\n```\n\nThe output is `results/laguna_s21_full_results.csv`\n\n. Raw generations and error details remain in the adjacent JSON artifacts.\n\nThe harness can also benchmark a local `llama-server`\n\n(including Poolside's DFlash build) through its OpenAI-compatible endpoint:\n\n```\nuv run --frozen laguna-bench run \\\n  --engine openai \\\n  --base-url http://127.0.0.1:8080/v1 \\\n  --model laguna \\\n  --output results\n```\n\nSizes are repository payloads observed on 2026-07-21 and should be refreshed before downloading:\n\n`mlx-community/Laguna-S-2.1-oQ2e`\n\n: 33.74 GiB, calibrated 2.70 bpw`mlx-community/Laguna-S-2.1-oQ3e`\n\n: 45.86 GiB, faster decode but lower quality in this suite`poolside/Laguna-S-2.1-NVFP4-mlx`\n\n: 66.97 GiB, official testing-only build; functional but slow on this runtime`poolside/Laguna-S-2.1-GGUF`\n\nQ4_K_M: 70.01 GiB, functional through Poolside's llama.cpp branch\n\nThe Vontra and pipenetwork 4-bit MLX conversions both fail to load in mlx-vlm 0.6.6, each because of a different router or quantization incompatibility. We did not download their 6-bit derivatives after those failures. The 116.34 GiB 8-bit conversions leave too little headroom for weights, the KV cache, and macOS on a 128 GB machine. The CSV still includes every variant and records each failure or omission explicitly.\n\nResults from other Apple Silicon machines are welcome. The harness records the chip, memory, macOS version, model revision, and runtime versions, then merges your rows into the committed CSV without removing earlier community results.\n\nFirst, fork the repository on GitHub and clone your fork. Install [uv](https://docs.astral.sh/uv/) if needed, then create a branch and sync the locked environment:\n\n```\ngit clone https://github.com/<your-user>/macos-laguna-s2.1.git\ncd macos-laguna-s2.1\ngit switch -c results/<model>-<chip>\nuv sync --extra dev --python 3.13 --locked\n```\n\nThe reference oQ2e model is a 33.74 GiB download and peaked near 38 GB in the short suite. Make sure your Mac has enough unified memory and free disk space before starting. Download the default model and reproduce the six quality tasks plus the standardized quant profile:\n\n```\nscripts/laguna.sh download\nscripts/laguna.sh bench\n```\n\nTo test a different conversion, set `LAGUNA_MODEL`\n\nfor both commands:\n\n```\nexport LAGUNA_MODEL=mlx-community/Laguna-S-2.1-oQ3e\nscripts/laguna.sh download\nscripts/laguna.sh bench\n```\n\nFor the complete context, sampling, KV cache, and prefill matrix, run the community profile. It reaches 256K input tokens and can take several minutes per long case on the reference M5 Max:\n\n```\nLAGUNA_MODEL=mlx-community/Laguna-S-2.1-oQ2e scripts/laguna.sh community\n```\n\nReview the comparison, refresh the merged CSV, and run the tests before committing:\n\n```\nuv run --frozen laguna-bench compare --output results\nuv run --frozen laguna-bench export --output results\nuv run --frozen pytest -q\ngit diff -- results/laguna_s21_full_results.csv\n```\n\nRaw run directories stay local because they are large and can contain machine-specific paths. Add the combined CSV to your pull request. Update `BENCHMARK_RESULTS.md`\n\nonly when your run adds a finding that needs explanation.\n\n```\ngit add results/laguna_s21_full_results.csv\ngit commit -m \"Add <chip> results for <model>\"\ngit push -u origin HEAD\ngh pr create --fill\n```\n\nIn the pull request, briefly mention your Mac model, chip, unified memory, macOS version, and whether the run was plugged in and otherwise idle. Do not edit or delete existing CSV rows by hand. Load failures are useful results too: the harness records them as failure rows so compatibility gaps remain visible.\n\nThe agent can list, read, and write files only inside its disposable task workspace. It can also invoke the fixture's fixed pytest command. Generation graders execute model-produced Python in a temporary directory with a timeout, but this does not provide a hardened OS sandbox. Run only models you trust, and keep sensitive data on the machine backed up.", "url": "https://wpnews.pro/news/running-laguna-s-2-1-locally-on-apple-silicon-52-tok-s-with-38-5-gb-peak-memory", "canonical_source": "https://github.com/tanishq-dubey/macos-laguna-s2.1", "published_at": "2026-07-22 03:02:48+00:00", "updated_at": "2026-07-22 03:22:16.865639+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["DWS LLC", "Apple M5 Max", "mlx-community/Laguna-S-2.1-oQ2e", "mlx-vlm", "poolside/Laguna-S-2.1-NVFP4-mlx", "tanishq-dubey", "MLX", "Laguna S 2.1"], "alternates": {"html": "https://wpnews.pro/news/running-laguna-s-2-1-locally-on-apple-silicon-52-tok-s-with-38-5-gb-peak-memory", "markdown": "https://wpnews.pro/news/running-laguna-s-2-1-locally-on-apple-silicon-52-tok-s-with-38-5-gb-peak-memory.md", "text": "https://wpnews.pro/news/running-laguna-s-2-1-locally-on-apple-silicon-52-tok-s-with-38-5-gb-peak-memory.txt", "jsonld": "https://wpnews.pro/news/running-laguna-s-2-1-locally-on-apple-silicon-52-tok-s-with-38-5-gb-peak-memory.jsonld"}}