Building a fully offline, portable AI assistant that runs from a USB pendrive on an Apple M4 Mac mini (16 GB RAM) A developer has detailed a methodology for building a fully offline, portable AI assistant that runs from a USB drive on an Apple M4 Mac mini with 16 GB RAM. The approach leverages llamafile and GGUF quantization to enable GPU-accelerated local inference without installation or network access. The work also corrects seven defects in a prior community guide, including a Windows-only binary selection and a missing compiler dependency. Large language models are increasingly delivered as network-dependent services, yet their underlying runtimes llama.cpp, vLLM, and derivatives are trivially portable. This work addresses the gap between portable and deployable : constructing a fully offline, zero-install, GPU-accelerated LLM environment that boots from a removable USB drive on an Apple M4 Mac mini with 16 GB of unified memory. We analyze the llamafile distribution model Cosmopolitan Libc fat binaries, mmap'd weight loading, runtime GPU compilation , the GGUF quantization ecosystem, and the memory/performance physics of Apple Silicon unified memory, and derive a set of falsifiable hypotheses H1–H5 that an empirical benchmark phase will confirm or refute. We further identify and correct seven defects in a prior community guide — most critically a Windows-only binary selection, an overly conservative context window, and a missing first-run compiler dependency that silently disables GPU acceleration. The resulting plan is a complete, reproducible methodology: environment verification, controlled benchmarking via the llama.cpp metrics endpoint, removable-storage characterization, packaging, and security review. Keywords: llamafile, llama.cpp, GGUF, offline inference, Apple Silicon, Metal, portable computing, quantization State-of-the-art language models are conventionally consumed through hosted APIs. Local execution, when pursued, typically presupposes a developer environment: a package manager, a virtual environment, GPU driver stacks, and multi-gigabyte model registries. Each of these assumptions fails in the context of air-gapped, portable, or one-off computing — a clinician on an isolated ward network, a security reviewer in a vault, a field engineer on an aircraft, or a user who simply does not wish to install anything on a machine they do not own. This paper develops the theory and an execution plan for the minimal viable offline LLM appliance: a single USB pendrive that, when inserted and double-clicked, serves a locally hosted, GPU-accelerated chat interface with no network connectivity and no system modifications. Two technologies make this tractable: 1. llamafile Mozilla AI — a single-file distribution of llama.cpp built on Cosmopolitan Libc, executable natively across macOS, Windows, and Linux without installation. 2. GGUF — llama.cpp's self-describing model container, in which weights are stored at aggressively compressed precision quantization , bringing a multi-billion-parameter model within reach of consumer memory. The specific target platform is an Apple M4 Mac mini with 16 GB unified memory running macOS 14.1 or later. Apple Silicon is uniquely suited to this problem class: the GPU shares a single physical memory pool with the CPU, enabling Metal offload without the VRAM/RAM split that complicates discrete-GPU systems. - A corrected, macOS-validated revision of a community offline-LLM guide, cataloguing seven concrete defects Corrections table, §Research Findings . - A mechanistic theory of the llamafile runtime §3 : fat-binary structure, mmap weight loading, and one-time runtime compilation of GPU backends. - A memory and performance model for 16 GB unified memory §3.4–3.5 , including the KV-cache arithmetic that motivates context-size selection. - A falsifiable experimental methodology §4 : five hypotheses, an instrumentation strategy via the /metrics endpoint, and a decision policy for model selection. This work targets macOS ARM64 Metal only; Windows CUDA behavior is referenced solely to bound scope. It does not train or fine-tune models. It evaluates two model tiers Qwen3-4B and Qwen3-8B, Q4 K M and does not claim optimality over the broader GGUF catalogue. Security analysis is limited to threat identification, not formal verification. The llama.cpp substrate. Most open-weight local inference traces to llama.cpp Georgi Gerganov et al. , a C/C++ inference engine that popularized CPU-first execution and the GGUF format. Its server mode exposes an OpenAI-compatible HTTP API plus a bundled web chat UI and a JSON metrics endpoint — the instrumentation used throughout this plan. Portable prior art. Community projects such as Portable-Offline-LLM geevarghesekthomas84-sys demonstrate the concept — a USB drive containing llama.cpp binaries, a models/ directory, and launch scripts. This prior art is predominantly Windows-oriented .bat launchers, .exe binaries and — critically — omits GPU verification methodology; on a misconfigured host it runs silently on CPU. The present work addresses both omissions. The llamafile distribution model. llamafile Mozilla AI packages llama.cpp as a Cosmopolitan "actually portable executable" APE . A single release artifact runs on six OS families; optionally, model weights may be appended to the same file, producing a self-contained "llamafile" that is executed directly. Our plan uses the non-fused form runtime + separate GGUF , keeping the model swappable on a small drive. Model family selection. Qwen3 Alibaba provides strong reasoning-per-parameter efficiency, hybrid "thinking" variants, and 262K-token native context — properties that fit a memory-constrained portable target. Its quantized GGUFs are produced by unsloth, the most widely used quantization house on Hugging Face; the 8B Q4 K M is among the most-downloaded GGUF artifacts platform-wide, which we treat as weak but non-trivial evidence of practical utility. Related commercial alternatives Ollama offline bundles, LM Studio portable mode require installation and are excluded by the zero-install requirement. Cosmopolitan Libc compiles a single ELF/MZ hybrid — a polyglot file that is simultaneously a valid shell script, a Windows PE, and a Unix ELF. The llamafile release binary exploits this in three nested layers: 1. Script wrapper. An MZ-prefixed shell script; Windows parses the PE header, Unix executes the script body. One asset name serves all platforms. 2. Appended, page-aligned ZIP. The true llama-server executable, the GPU module sources , and in fused llamafiles model weights live in a ZIP appended to the script. The ZIP is aligned to the page size by a purpose-built ~500-line zipalign tool: Metal requires page-aligned memory for buffer handoff to the GPU. 3. APE loader 8 KB . On Unix-likes, a small stub maps the binary portions into memory before transferring control. At startup the runtime mmap s its own file, mapping weights directly into the process address space — no extraction, no copy, no installation. Corollary for removable storage: the operating system serves the GGUF from the drive's page cache on first access; subsequent reads hit RAM. Drive throughput therefore bounds cold-start load time but not generation throughput §3.6 . GPU backends cannot be statically linked into a Cosmopolitan binary: vendor runtimes are platform-specific and enormous, and the polyglot build targets six OSes. llamafile instead ships GPU code as source and compiles it on first run: - On macOS ARM64, the Metal backend metal.c extracts ggml-metal.m and ggml-metal.metal from its embedded ZIP and invokes the system C compiler supplied by Xcode Command Line Tools to produce a self-contained ggml-metal.dylib , targeting the native GPU microarchitecture of the host. The dylib is loaded via cosmo dlopen and registered with GGML as a first-class backend. - The compiled module is cached at $TMPDIR/.llamafile or $HOME/.llamafile — compiled once, reused thereafter. - Failure mode: if the compiler is absent, the build fails and llamafile silently falls back to CPU — an order-of-magnitude performance loss indistinguishable from success in logs. This is the dominant failure mode on a fresh Mac, and the central design driver of Phase 1 mandatory xcode-select --install and of the launcher's --gpu apple flag, which converts the silent fallback into a fatal startup error fatal error: support for --gpu apple ... wasn't available . - ABI correctness across this host/dylib boundary is not incidental: on ARM64 the module is compiled with -ffixed-x28 reserving the TLS register so host and dylib share ABI state , and exported functions use ms abi so a single module contract spans Windows and Unix calling conventions. Release-artifact corollary. Only the full release binary llamafile-0.10.5 , 334.5 MB embeds the compiler toolchain; the -thin variant 40.4 MB does not and cannot bring up Metal without a prebuilt dylib. Asset selection is therefore a hard functional requirement, not a size trade-off. GGUF is a single-file container holding weights and the metadata required to serve them — architecture, tokenizer, and the Jinja chat template — which is why no template configuration is required at serve time. Quantization stores weights at reduced precision. GGUF quants are block-based : weights are grouped typically 128 per block with a shared per-block scale, and the "K" family K-quants additionally partitions each block by importance using a k-means-style split, retaining a small subset of sensitive weights at higher precision. Effective size relative to FP16: | Quant | Size vs FP16 | Regime | |---|---|---| | Q8 0 | ~50 % | near-lossless | | Q6 K | ~62 % | high quality | | Q5 K M | ~69 % | strong quality | | Q4 K M | ~75 % | quality/memory optimum | | Q3 K M | ~81 % | degraded | | Q2 K | ~87 % | avoid | For the 8B tier this yields 5.03 GB at Q4 K M versus ~10 GB at FP16 — the difference between "fits a 16 GB machine" and "does not". Q4 K M is the established Pareto optimum of perplexity per byte; the plan adopts it for both model tiers 8B: 5.03 GB; 4B-Thinking: 2.6 GB, the smallest Qwen3 variant judged to retain acceptable reasoning . Steady-state resident memory is approximately: R ≈ W + K + O where W is the quantized weights, K the KV cache, and O the runtime overhead ~1–2 GB . The KV cache is the only term that scales with the context window: K tokens = 2 · n layers · n kv heads · head dim · 2 bytes f16 · tokens Qwen3 uses grouped-query attention with only 8 KV heads ; the cache therefore scales with KV heads, not query heads — a factor-8 reduction versus MHA. For Qwen3-8B 36 layers, head dim 128 : K ≈ 2 · 36 · 8 · 128 · 2 B · tokens ≈ 144 KB / token → 1.18 GB @ 8,192 ctx → 2.36 GB @ 16,384 ctx Projected steady-state budgets on the 16 GB target: | Model | W RAM | K @ 8,192 | Peak R | |---|---|---|---| | 4B-Thinking Q4 K M | ~2.8 GB | ~0.6 GB | ~5 GB | | 8B Q4 K M | ~5.3 GB | ~1.2 GB | ~8 GB | | 8B Q4 K M @ 16,384 ctx | ~5.3 GB | ~2.4 GB | ~9.5 GB | All tiers retain ≥ 6 GB headroom — no swap, no OOM. The original guide's --ctx-size 2048 conservatism is unnecessary on this hardware and wastes reasoning headroom; the plan adopts 8,192 default, 16,384 maximum , beyond which the quality/memory trade-off flattens for 4B–8B models. Metal offload on Apple Silicon exploits a structural advantage: weights live in the same physical memory the GPU reads, at full memory bandwidth, with no PCIe transfer and no VRAM copy. The page-aligned mmap buffers of §3.1 are handed to Metal as-is. This is why a 5 GB model is GPU-accelerated at all on a 16 GB machine, and why Apple Silicon parts typically exceed equivalently priced discrete-GPU laptops in tokens/second for these model sizes. Empirical targets hypotheses to be measured, not claims : ~40–60 t/s for the 4B tier, ~25–40 t/s for the 8B tier, Q4 K M, on M4. Loud-failure principle. CPU fallback §3.2 is indistinguishable in the happy-path log; single-digit tokens/second is the symptom. Any benchmark figure in that range invalidates the GPU hypothesis and reruns Phase 1 verification. - exFAT is the correct filesystem: no 4 GB file ceiling FAT32's limit , negligible per-file overhead, no journaling reduced flash wear , and native read/write on macOS, Windows, and Linux. APFS on external media is macOS-only and journaled to no portable benefit. - Cold start is page-cache fill. The GGUF is read from the drive once and served from RAM thereafter. Throughput expectations: ~5 GB at USB 3.2 Gen 2 ~300–1000 MB/s ≈ 5–17 s; at USB 2.0 ~40 MB/s ≈ 2 min. The plan therefore treats drive speed as a cold-start parameter, not a throughput parameter. - Gatekeeper. Any file delivered via HTTP carries the com.apple.quarantine extended attribute; a quarantined executable refuses to launch. Attribute removal xattr -d or right-click → Open is mandatory setup, applied in Phase 1. The appliance is by construction a self-extracting, runtime-compiling executable — startup executes embedded code by design. The corresponding threat model: - Untrusted-host risk. Running the stick on a machine one does not trust transfers trust to whatever the stick carries. Mitigation: treat the stick as a trusted appliance on trusted machines only. - Unauthenticated server. The bundled server binds to 127.0.0.1 by default. Binding 0.0.0.0 would expose an unauthenticated LLM API to the LAN — prohibited in this plan. - Untrusted model input. The GGUF parser is part of the attack surface. Mitigation: source models from well-known quantizers unsloth and the runtime from the maintainer's GitHub releases; pin checksums at packaging time. - At-rest exposure. Models are unencrypted on the drive; appropriate for public-weight models, incompatible with private weights. The plan is structured as an experiment: five falsifiable hypotheses, each mapped to a test and a falsification criterion. | | Hypothesis | Test | Falsified by | |---|---|---|---| | H1 | Metal builds and loads on the target Mac | Phase 1, --gpu apple | CPU-only logs; fatal error | | H2 | 8B Q4 K M fits 16 GB with ≥ 6 GB headroom | Phase 2 metrics at 8,192 / 16,384 ctx | Peak RAM ≥ 10 GB | | H3 | GPU tokens/second in the §3.5 range | Phase 2 /metrics | Single-digit t/s silent CPU fallback | | H4 | Cold-start from the stick is acceptable | Phase 3 timing | Load 60 s on USB 3.2 | | H5 | Double-click flow needs zero terminal interaction | Phase 4 acceptance | Gatekeeper/CLT failure on cold boot | Instrumentation. llamafile's server exposes /metrics llama.cpp JSON — machine-readable tokens/second, prompt processing, and memory figures. Controlled variables per run: model, quantization, context size, prompt, sampling temperature. Three context levels 2,048 / 8,192 / 16,384 × two model tiers, recorded into a benchmark matrix; the decision policy Table 5, §6 selects the shipped configuration from measured data, keeping model selection empirical rather than a priori. Build a fully offline, portable AI assistant that runs from a USB pendrive on an Apple M4 Mac mini 16 GB RAM — no internet, no accounts, no installation. The stick carries a single self-contained binary llamafile , a GGUF model, and a double-clickable launcher. This document is the research + execution plan . Nothing here has been implemented yet; Phase 4 packaging is the implementation phase and is explicitly deferred. | Property | Value | |---|---| | Machine | Apple M4 Mac mini | | RAM | 16 GB unified memory | | OS | macOS 14.1+ Sonoma required; Apple Silicon ARM64 | | GPU | Built-in Apple Silicon GPU via Metal offload on by default | | Storage | USB drive, exFAT formatted | | Runtime | llamafile single-file binary, no install | | Model | Qwen3 GGUF 4B or 8B tier, see decision matrix | OfflineAI/ ← root of the USB stick │ ├── llamafile-0.10.5 ← full macOS binary 334.5 MB, NOT -thin ├── models/ │ └── Qwen3-8B-Q4 K M.gguf ← 5.03 GB or 4B tier, 2.6 GB │ └── Qwen3-4B-Thinking-2507-Q4 K M.gguf ← optional portable tier │ └── Start.command ← double-click to launch chmod +x When Start.command runs: 1. cd s to its own directory works from any mounted drive path 2. Launches llamafile --server with the model + context size 3. Opens http://localhost:8080 — the built-in llama.cpp web UI 4. First run compiles a Metal GPU module one-time, via Xcode CLT Everything below was verified via web research before writing this plan. - v0.10.5 is the current release published 2026-08-03 , tracking llama.cpp b10103. - Release assets for macOS from the 0.10.5 release page : | Asset | Size | Use it? | |---|---|---| | llamafile-0.10.5 | 334.5 MB | ✅ Yes — full binary, embeds the GPU compiler toolchain needed for Metal | | llamafile-0.10.5-thin | 40.4 MB | ❌ No — lacks the embedded compiler; GPU setup will fail/skip | | llamafile-0.10.5.zip | 255.7 MB | Optional — zip wrapper of the same binary | | llamafile-0.10.5.exe | — | ❌ Windows only | - No -x86 64 asset exists in 0.10.5 — the single llamafile-0.10.5 binary is the macOS one. Verify universal-arch behavior in Phase 1. - Metal is supported only on macOS ARM64 Apple Silicon . Offload is enabled by default — no -ngl flag needed that flag is for NVIDIA/AMD . - First run compiles ggml-metal.dylib from embedded sources using the Xcode Command Line Tools ; cached in $TMPDIR/.llamafile or $HOME/.llamafile . - llamafile silently falls back to CPU if the Metal module fails to build/load. Mitigation: run with --gpu apple , which turns failure into a loud startup error. - The known Windows CUDA breakage issue 961: no pre-built GPU library found on 0.10.x is Windows-only — not applicable on macOS. | Model | Quant | File size | RAM @ load ~2K ctx | Notes | |---|---|---|---|---| | Qwen3-4B-Thinking-2507 | Q4 K M | 2.6 GB | ~3 GB | Portable tier — runs on any machine, any stick | | Qwen3-8B | Q4 K M | 5.03 GB | ~5.5–6 GB | Recommended for M4/16 GB — much better quality, ~10 GB headroom | | Qwen3-8B-Thinking-2507 | Q4 K M | ~5.2 GB | ~6 GB | Reasoning variant of 8B; verify exact size in Phase 2 | | 7B class Llama 3.1 etc. | Q4 K M | ~4.4 GB | ~5–5.5 GB | Alternative if Qwen3 unavailable | Quantization scale vs FP16 : Q8 0 ~50 %, Q6 K ~62 %, Q5 K M ~69 %, Q4 K M ~75 % the sweet spot , Q3 K M ~81 %, Q2 K ~87 % avoid . Source repos: unsloth/Qwen3-4B-Thinking-2507-GGUF , unsloth/Qwen3-8B-GGUF on Hugging Face the 8B non-thinking model is the most-downloaded GGUF on the platform; the doc's original 4B-Thinking pick is confirmed correct for the portable tier . The user's original doc was Windows-oriented and had several issues for this project: | | Original doc / earlier plan | Problem | Correction | |---|---|---|---| | 1 | llamafile-0.10.5.exe | Windows binary | Use the plain llamafile-0.10.5 macOS binary | | 2 | Start.bat with .\ prefix | Windows syntax | Start.command shell script with $ dirname "$0" | | 3 | --ctx-size 2048 default | Too conservative for 16 GB | Default 8192 , max 16384 Qwen3 GQA keeps KV cache small | | 4 | unstated first-run GPU compile | Fresh Macs lack Xcode CLT → silent CPU fallback | Install CLT xcode-select --install and verify Metal with --gpu apple | | 5 | unstated Gatekeeper | Downloaded binary is quarantined → won't launch | xattr -d com.apple.quarantine once | | 6 | Metal dylib path ~/Library/... | Wrong path in an earlier plan draft | Actual: $TMPDIR/.llamafile or $HOME/.llamafile | | 7 | Single 4B model | Undersized for the M4/16 GB target | Add 8B Q4 K M tier 5.03 GB as primary recommendation | Status: PLANNED — nothing implemented. Phase 4 is the implementation phase and will not be started until this plan is approved. Goal: prove the machine can run llamafile with Metal before downloading 5 GB of models. 1. Confirm OS + hardware: sw vers needs Darwin 23.1.0+ macOS 14.1+ uname -m needs arm64 system profiler SPHardwareDataType | grep -E "Chip|Memory" 2. Install Xcode Command Line Tools if missing required for the one-time Metal module compile : xcode-select -p prints path if already installed xcode-select --install only if missing 3. Download the binary full asset, not -thin : curl -L -o llamafile-0.10.5 \ https://github.com/mozilla-ai/llamafile/releases/download/0.10.5/llamafile-0.10.5 chmod +x llamafile-0.10.5 4. Strip Gatekeeper quarantine: xattr -d com.apple.quarantine llamafile-0.10.5 5. Verify Metal builds and loads loud failure mode : Expect a "building/loading Metal module" log line. If it silently reports CPU-only, stop and debug before Phase 2. ./llamafile-0.10.5 --gpu apple --version Exit criteria: --gpu apple succeeds; startup logs show the Metal module build/load. Goal: choose the model tier with measured numbers, not vibes. 1. Download candidates: curl -L -o models/Qwen3-8B-Q4 K M.gguf \ https://huggingface.co/unsloth/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q4 K M.gguf curl -L -o models/Qwen3-4B-Thinking-2507-Q4 K M.gguf \ https://huggingface.co/unsloth/Qwen3-4B-Thinking-2507-GGUF/resolve/main/Qwen3-4B-Thinking-2507-Q4 K M.gguf 2. Benchmark each model × each context size 2048 / 8192 / 16384 : ./llamafile-0.10.5 --server --model models/