Samosa Chat: Run Qwen3.6-35B-A3B Locally on a 16 GB Apple Silicon Mac

Run Qwen3.6-35B-A3B locally on a 16 GB Apple Silicon Mac.

In your terminal, or in your browser · Runs on the CPU · No cloud account · No telemetry

Credit.Samosa Chat is built on[colibrì]by JustVugg. Its expert-streaming design, SIMD kernels, and core utility headers made this project possible. The model is the text part of[Qwen3.6-35B-A3B], created and released by the Qwen team. Samosa Chat is an independent, unofficial, Apache-2.0 project. It is not affiliated with or endorsed by either team.

What Samosa adds:its own Qwen3.6 inference engine in C — the 30 Gated DeltaNet layers, the 10 attention layers, and the routed-expert path — the group-32 quantization format and its converter, the byte-budgeted expert cache that fits 35B parameters into 16 GB, sealed conversations that resume exactly, a local server and browser app, an atomic installer that verifies and rolls back, and the tests around all of it.[The full list].

A real, unedited recording on the 16 GB M3 MacBook Air — a question in, an answer out, no cloud:

Real time, played at normal speed. The before the answer is the model ; after that it writes at about 5–9 tokens per second.

curl -fsSL https://huggingface.co/deepanwa/Samosa-Chat-Qwen3.6-35B-A3B-group32/resolve/main/install.sh | sh

Then open a new terminal and ask it something:

samosa "explain how DNS works"

You need an Apple Silicon Mac, 16 GB of RAM, Apple's Command Line Tools (for the C compiler), and about 30 GB of free disk. The download is about 24 GB. The installer resumes interrupted downloads, checks the SHA-256 of every file, compiles the C engine on your machine, and smoke-tests it before switching the new release live. A corrupt or interrupted upgrade leaves your existing install untouched. It does not need administrator rights.

Everything lives under ~/.samosa

, and nothing is installed system-wide:

path	what it is
~/.samosa/bin/samosa
the samosa command itself
~/.samosa/current
symlink to the active release
~/.samosa/releases/
verified releases, kept so an upgrade can roll back
~/.samosa/chats/
your saved conversations

The installer adds ~/.samosa/bin

to your PATH

by appending one line to your shell's rc file (~/.zshrc

for zsh, ~/.bashrc

for bash, otherwise ~/.profile

). That only affects terminals you open afterwards — which is why the step above says to open a new one. If samosa

still is not found:

export PATH="$HOME/.samosa/bin:$PATH"

~/.samosa/bin/samosa "how are you"

samosa doctor

reports which release is active and whether the model, engine, and tokenizer are healthy.

To uninstall, delete ~/.samosa

and remove that one line from your rc file.

The model lives at deepanwa/Samosa-Chat-Qwen3.6-35B-A3B-group32.

Both come from the same install. Pick based on what you want:

| |---|

Web app

samosa "your question"

samosa app

the normal way to use Samosaa demoIf you just want to chat with the model, use the terminal. It starts instantly, does everything the model can do, and is the interface that gets used day to day.

The web app is a demo at this point. It works, and it is the nicest way to watch answers stream in and see the model's reasoning unfold. But it exists to demonstrate the engine, not as a polished product. Do not feel you need it.

Samosa Chat runs Qwen's 35-billion-parameter model on a Mac that has only 16 GB of RAM.

The model is a Mixture of Experts. It has 35B parameters in total, but it only uses about 3B of them for each token. Samosa never loads all 35B into memory. The shared ("dense") weights stay in RAM. The expert weights are read from the SSD as the model chooses which experts each token needs.

Samosa runs entirely on the CPU. It does not use Metal or the Apple GPU. You do not need a dedicated GPU.

The model is text only. Qwen3.6 can also read images, but Samosa's converted model leaves the image part out.

Where it runs. macOS on Apple Silicon (arm64

) only. It has been tested on one 16 GB M3 MacBook Air. "Runs on the CPU" does not mean it runs on any 16 GB laptop. The installer refuses other systems. Linux and Windows are not supported.

This is the main way to use Samosa. Ask a question, get an answer:

samosa "explain how a hash table handles collisions"

Keep the conversation going with --continue

. Your chat resumes from a saved snapshot, so a follow-up does not re-read the whole history:

samosa "explain how a hash table handles collisions"
samosa --continue "and which strategy does Python use?"
samosa --continue "show me the CPython source for that"

The rest of the options:

samosa --think "solve this logic puzzle"                # general reasoning
samosa --think-code "build a responsive settings page"  # precise coding profile
samosa --fast "summarize this design"                   # all cores, runs warmer
samosa --seed 11 "give me a deterministic sample"       # reproducible sampling
samosa --max-tokens 2048 "write a long explanation"     # change the ceiling
samosa --thinking-budget 512 "..."                      # cap internal reasoning
samosa doctor                                           # check the installation

An answer can run up to 8,192 new tokens. That is an outer ceiling, not a target — the model usually stops earlier on its own when it emits its end-of-turn token. Two threads is the default so the Mac stays cool; --fast

uses all performance cores and runs warmer.

By default the model answers directly. --think

and --think-code

turn on reasoning, which is slower and reads a lot more from your SSD — see SSD wear before using them heavily.

A conversation is capped at 24,576 tokens total (saved history + your new message + the answer ceiling). Samosa checks this before it runs and stops rather than growing memory without limit.

samosa app

starts a local server and opens a chat page in your browser. Everything runs on your machine. The page makes no outside requests.

samosa serve          # start the server in the foreground on 127.0.0.1:8642
samosa app            # start the server in the background and open the chat page
samosa serve --stop   # stop the server

What the app does:

• Streams the answer as the model writes it.
• Shows the model's thinking separately from its final answer.
• Lets you stop a generation at any time.
• Saves your conversations so you can continue them later.
• Shows live speed (tokens per second) and current memory use.
• Has settings for thinking mode, maximum answer length, and a fixed seed.

The server answers these HTTP endpoints:

GET /healthz

— status, memory use, the context limit, queue state, last speedGET /v1/models

POST /v1/chat/completions

— reply as JSON, or stream token by token (SSE)POST /v1/cancel

— stop the current generationPOST /v1/shutdown

— stop the server cleanly

Only one request runs at a time. Extra requests wait in a short queue.

Stopping an answer is safe. When you stop an answer partway through, Samosa saves the conversation only up to the last complete sentence. This matters: before this fix, if you stopped an answer in the middle of a sentence, the next answer in that chat would copy the cut-off style and reply with only a word or two before stopping. That is now fixed. If a stopped answer has no complete sentence yet, Samosa keeps the previous saved state instead of overwriting it.

Context limit. The same 24,576-token cap applies here. The server checks a turn before queueing it and rejects an oversized one before allocating any memory. Only the conversation you are using is loaded into RAM; opening other saved chats does not add to memory.

You do not need this to use Samosa — the installer above compiles the engine for you. This is for working on it.

make            # portable CPU build
make omp        # multithreaded build (needs libomp on macOS)
make test       # run the bounded tests

make

builds the engine only, and has no Python dependency. To run it you also need the model files and tokenizer: either the published group-32 download above, or your own conversion from the original Qwen checkpoint with tools/convert_qwen36.py

. With the engine and model in place, samosa serve

and samosa app

start the server. Full server details and the exact request format are in docs/SERVE_API.md.

Python is only used for conversion, analysis, and testing. It is not needed to run the model.

Every decision follows these three goals, in this order:

It must be stable on a machine like this one— a 16 GB Apple Silicon Mac. Memory stays bounded. It does not grow without limit. It stops at clear limits instead of crashing.It must be actually useful. Real answers, real code, real multi-turn conversations. Not a demo that only loads.It must not wear out the machine. Keep memory bounded so the system does not swap heavily. Use two threads by default so the Mac stays cool. Be careful with the SSD reads that cause the real wear (explained below).

A feature is only called "released" once it meets all three. Until then it stays in this repository as source.

The Qwen model and the colibrì runtime are the starting point. This repository adds:

• A Qwen3.6 text engine written in C. It covers the 30 Gated DeltaNet layers, the 10 gated attention layers, the shared and routed expert path, the tokenizer, and the chat template.
• A converter that turns the original Qwen checkpoint into Samosa's format, shard by shard, with a manifest-based container for the expert weights.
• Three weight formats: the older whole-row int4, the newer group-32 int4, and an experimental mixed format (group int4 for gate/up, row int8 for the down-projection).
• CPU math for those formats: Apple NEON dot-product on Apple Silicon, and a portable AVX2 path for other CPUs.
• An expert cache that keeps a fixed byte budget in RAM, drops the least-recently-used experts first, keeps a floor per layer, reuses freed memory, watches system memory pressure, and reports its I/O.
• Saved conversations ( QWSESS01

files) that are checked against the model geometry, sealed with a SHA-256 hash, written atomically, and can be resumed exactly. - Qwen's published sampling settings for direct, general-thinking, and precise-code modes.

• A thinking-budget limit with a clean hand-off to the answer, separate counts for natural versus forced endings, and a guard that stops a repeating token loop.
• A local HTTP server in C with no dependencies: JSON or streaming replies, a bounded request queue, cancellation, health reporting, and clean shutdown.
• A 32 KB browser chat page with no external scripts, no analytics, and no outside requests, shipped with the Samosa logo.
• An installer that verifies and tests a new version in place before switching to it, and rolls back if the new version is bad.
• Test tooling for output structure, task correctness, upstream comparisons, the quantized math, route traces, installer rollback, and memory-pressure limits.

Samosa uses Qwen's published sampling settings:

mode	temperature	top-p	top-k	presence penalty	thinking budget
direct	0.7	0.80	20	1.5	off
general thinking	1.0	0.95	20	1.5	1,024 tokens
precise code	0.6	0.95	20	0.0	2,048 tokens

The maximum answer length is an outer limit, up to 8,192 new tokens. It is not a fixed length. The model decides when to stop within that limit. If the thinking reaches its budget, Samosa adds Qwen's trained wind-down text before closing the </think>

block, rather than cutting it off with a bare token. Closing the thinking block keeps the output well-formed; it does not prove the answer is correct.

One test compared this against an upstream FP8 reference on a small set of arithmetic problems. The reference used 353–616 thinking tokens. A matching local group-32 run answered correctly and stopped on its own after 933 tokens with a 1,024-token thinking budget. This confirms the path works for that one kind of problem. It is not proof of broad benchmark quality. See the upstream-control report and the regression ledger.

Group-32 is the model format Samosa Chat uses. This section explains what it means, because it is the heart of the product.

Qwen describes the model as 35B parameters in total, about 3B used per token, 40 layers, 256 routed experts, and 8 routed plus 1 shared expert active in each Mixture-of-Experts layer. To fit that on a 16 GB Mac, Samosa stores most weights in int4 — 4 bits each instead of 16. Four bits cannot hold a real number on their own, so each weight also needs a scale: a full-precision number that the 4-bit value is multiplied by to reconstruct the original weight.

The question is how many weights share one scale.

Group-32 gives every block of32 weights its own scale. A scale only has to cover 32 nearby numbers, so it fits them closely. The reconstructed weights are close to the originals, and the model's output quality is good.- The older approach (see below) gave a whole matrix row— hundreds or thousands of weights — a single scale. One scale cannot fit that many different numbers well, so the reconstructed weights drift from the originals and the output shows visible defects.

Finer scales cost more storage, which is why group-32 is larger on disk:

part	group-32 (the product)	older whole-row
expert weights	20.94 GB	16.6 GB
shared weights	3.02 GB	1.3 GB

Group-32 also keeps the down-projection weights at int8 (8 bits) rather than int4, which is the main reason its shared weights are larger. The result is a model that reconstructs the original Qwen weights with measurably less error than the older format. It is what the app runs, and it is the published release: deepanwa/Samosa-Chat-Qwen3.6-35B-A3B-group32. Its quality is measured on one reference machine and one reasoning control, not across a broad benchmark suite — see Known limitations.

Whole-row int4 (the first format). This was the first quantization attempt: one int4 scale for an entire weight row. It is too coarse. A single scale has to cover a whole row of weights with very different sizes, so many weights are reconstructed inaccurately. In use this shows up as word-level defects such as of ofof

. Re-asking or changing the seed can avoid a given case but does not fix the cause. This is exactly why group-32 was built, and group-32 replaced it as the product. The old format still exists in an earlier Hugging Face repo, kept only so existing installs keep working. Do not start there.

Mixed int4/int8 format. An attempt to use int4 for the gate and up projections and int8 for the down projection across the whole model. The code for it exists and its math is tested, but no full model was ever produced in this format. It is not used by the product.

All numbers are from one fanless MacBook Air M3 with 16 GB of RAM. They describe specific runs on this one machine, not a guarantee for other machines.

The product is group-32, so those numbers come first:

group-32 task	threads	speed
direct answer	2	7.27 tokens/sec
933-token thinking answer	2	4.85 tokens/sec
5,000-token code page	4	6.47 tokens/sec

For reference, the older published model runs slightly faster because it is smaller: about 7–8 tokens/sec on 2 threads, about 9.5 on 4 threads (--fast

), and 14–24 tokens/sec for prefill.

Decode is the speed of writing the answer. Prefill is the speed of reading your input before it starts. Prefill is the slow part for long inputs: reading a 5,000-token document once takes about 3.5–6 minutes. Saved conversations mean a document is read only once.

On the command-line tool, older runs used about 2.5–3 GB and group-32 runs used about 3.2–3.9 GB.

In the app, memory grows in three stages:

Model loaded, no chat yet: about2.5 GiB.** After the first answer:about 3.9 GiB**. The first answer fills the expert cache, which adds about 1.3 GB and then holds steady.** As a conversation gets longer:**memory rises slowly with the length of the conversation you are in. In one test it rose about 143 MB while a single chat grew from 176 to 1,017 tokens. The expert cache stayed flat at 1.29 GB the whole time.

That growth is bounded, not a leak. The per-token memory (the KV cache) is about 40 KiB per token across the 10 attention layers. The measured rise is a little higher because the memory allocator holds on to its high point. For a conversation of fixed length, memory levels off — an eight-turn test on the same length held at 3.91–3.92 GiB. The 24,576-token limit caps the worst case at roughly 5–5.5 GiB after a maximum-length chat. Only one conversation is in memory at a time.

The memory number the app shows is the real macOS "physical footprint," which matches Activity Monitor.

A note on swap: on macOS, swap can stay in use from an earlier busy period even after memory frees up. macOS does not shrink the swap file or pull that data back on its own. So swap being in use does not by itself mean Samosa is swapping now. The signal to trust is green memory pressure.

Each saved turn writes a 63–70 MB sealed file to disk. The model files themselves are read-only.

This is the most important part for the health of your machine, so it is stated plainly.

Samosa keeps memory small by not holding all 35B parameters in RAM. Instead it reads each token's expert weights from the SSD as the model needs them. The longer an answer is, the more expert data it reads. The same popular experts get read again and again.

The amount is large. One 933-token thinking answer read 376 GB of expert data from the SSD. For comparison, swap — which people often worry about — is tiny here: over the same session the whole system (Samosa, editor, browser, all of it) wrote under about 9 GB to swap since the machine booted.

So the reads from expert streaming, not swap, are what actually wear the SSD, and the amount scales with how long the model thinks.

What this means for you:

Longer thinking costs disk reads, not just time. A short factual answer reads little. A long chain of thought reads a lot. Use direct mode (thinking: off

in the app,--direct

on the command line) when you do not need step-by-step reasoning.- Two threads is the default so the Mac stays cool. --fast

(4 threads) is something you choose on purpose. - Real-model test runs are kept short on purpose, because one long thinking run can read hundreds of gigabytes.

• SSD speed and lifespan genuinely matter here. This is the basic trade-off of running a 35B model in 16 GB of RAM. It is not a bug. It is the one resource worth spending on purpose.

Everything here was generated by the group-32 model — the product — through the app's chat endpoint in direct mode (no thinking, seed 11) on the 16 GB reference Mac. The screenshot is not edited.

Asked to build its own landing page — a single HTML file with embedded CSS, no JavaScript, dark theme, given the facts about Samosa Chat — the model produced this:

This run: 2,528 tokens, 5.15 tokens/sec, 4.33 GB memory. The exact HTML it wrote is saved at assets/example-landing.html. (The brew install

line in the page is the model's own copy; the real install command is the one above.)

Asked for a Python function, the group-32 model wrote this, and it passes its own tests when run:

from typing import List

def merge_intervals(intervals: List[List[int]]) -> List[List[int]]:
    """Merge overlapping intervals.

    Args:
        intervals: A list of intervals, each represented as [start, end].

    Returns:
        A list of non-overlapping intervals that cover all input intervals.
    """
    if not intervals:
        return []

    sorted_intervals = sorted(intervals, key=lambda x: x[0])
    merged = [sorted_intervals[0]]

    for current in sorted_intervals[1:]:
        last = merged[-1]
        if current[0] <= last[1]:
            last[1] = max(last[1], current[1])
        else:
            merged.append(current)

    return merged

assert merge_intervals([[1, 3], [2, 6], [8, 10], [15, 18]]) == [[1, 6], [8, 10], [15, 18]]
assert merge_intervals([[1, 4], [4, 5]]) == [[1, 5]]
assert merge_intervals([]) == []

This run: 279 tokens, 7.19 tokens/sec, 4.09 GB memory.

make test

covers the expert cache, the long-context KV math, the repetition guard, the thinking wind-down, the quantized math, the server, the command-line wrapper, installer rollback, output structure, route analysis, and the converter layout. The multithreaded build and the shell and Python syntax checks also run.

An earlier test harness had a serious false positive: it reported 14 of 15 passes using substring checks, even though 0 of 15 answers actually closed their </think>

block. Structural closing, natural versus forced endings, repetition, model stop, and task correctness are now scored separately. There is still not enough evidence to publish a general benchmark score. The planned evaluation steps are in docs/BENCHMARK_PLAN.md.

• The model runs on your machine. The engine has no telemetry. The server listens on local loopback only.
• The installer contacts Hugging Face only to download the public release files. Running the model does not need a cloud account.
• The macOS build is CPU-only. It uses NEON and optional OpenMP, not Metal.
• Two threads is the cool default. --fast

is a deliberate choice. - The expert cache watches memory pressure and can drop cached experts before the system is forced to swap.

• In the server, a generation can be cancelled between tokens.
• Real-model test runs are kept short because one long run can read hundreds of gigabytes from the SSD.

Nothing here is promised or dated. This is what the project wants to become, roughly in order of how much it would change things.

Today this is macOS on Apple Silicon only, and that is the single biggest limit on who can use Samosa. The goal is that anyone with 16 GB of RAM can run it — Windows, Linux, and Intel machines included.

The math is not the blocker: the engine is plain C and already carries portable AVX2 kernels next to the Apple NEON ones. The real work is everything around them — the installer refuses non-Apple-Silicon systems today, memory planning and the memory-pressure watcher call macOS-specific APIs (sysctl

, Mach), and no other platform has been tested as a product. None of that is fundamental; it just has not been done.

Qwen3.6 is natively multimodal, but Samosa's converter deliberately drops the vision tower and converts only the text half of the model. That was a scope and memory decision, not a limitation of the idea: the engine has no vision runtime, and a 16 GB budget had no room to keep an image encoder resident.

The goal is to add image input back. It means building a vision encoder alongside the language engine — roughly "colibrì, but for vision": an image encoder, patch embedding, and a projector into the language model's space. The tokenizer still carries Qwen's image and video tokens, so the language side is already ready for it.

The engine is CPU-only today. Metal should help, but it is not free speed: much of a long answer is spent streaming expert weights from the SSD, and a GPU does not make those reads faster. The plan is to move the expert matrix multiplies to Metal while CPU threads keep feeding them from disk, then measure end-to-end tokens/sec, SSD reads, memory, thermals, and battery — not just an isolated fast matmul. The CPU path stays as the correctness fallback.

The web app is a demo. To become a real interface it needs conversations kept in RAM instead of re-read from disk each turn, transcript management on the server, and deleting a chat should remove its saved snapshot, not just the browser copy. Chatting over a document and web access are wanted after that.

Group-32 is measured on one machine against one reasoning control. It needs a proper benchmark suite, comparison against upstream across task types, and a bounded test for very long answers (a crash above 4,096 tokens was fixed, but the regression test for it is not written yet).

macOS on Apple Silicon only. Linux code paths exist but are unverified. Windows is not supported. See the roadmap above.Text only. No images, video, audio, or tool calling.No GPU acceleration. Part of every answer is limited by SSD read speed.Quality evidence is thin. Group-32 is proven on one reference machine and one reasoning control, not across many machines or task types.SSD wear is real. Expert weights are streamed and reread many times during long answers. SeeSSD wear.- Deleting a chat in the app removes it from the browser but does not yet delete its saved file on disk.

App task programServer API and acceptance testsThinking-mode diagnosisGroup-32 model notesStorage migration logUpstream comparisonDetailed work log

Apache-2.0. See LICENSE and NOTICE for the full attribution and derivative-work notice.

source & further reading

github.com — original article