Open-sourced a local WAN 2.1 mobile pipeline for on-device video generation

Quartz, a from-scratch Rust inference engine developed by Saient, has been open-sourced, enabling on-device video generation with Wan2.1 T2V (1.3B parameters) on a Samsung Galaxy S24's Vulkan GPU, alongside LLM chat and SDXL image generation, with no cloud round-trip. The engine, previously named tinyq4, supports CPU, CUDA, and Vulkan backends and reads GGUF files directly.

Quartz is the on-device inference engine behind Saient https://saient.co.uk — a from-scratch Rust runtime no llama.cpp, no ggml, no Candle, no PyTorch/libtorch that runs LLM chat and image generation entirely on a phone or desktop, no cloud round-trip. It's the engine that made this possible: Saient mobile generates full text-to-video clips — Wan2.1 T2V https://github.com/Wan-Video/Wan2.1 , 1.3B parameters — running end-to-end, locally, on a stock Samsung Galaxy S24's Vulkan GPU. Same phone also runs LLM chat and SDXL image generation locally through this engine. No server, no API key, nothing leaves the device. The Wan video model itself runs through a separate, MIT-licensed companion binary — see Relationship to the Wan video engine relationship-to-the-wan-video-engine below for how the two fit together. This repo, Quartz proper, is the LLM chat + SDXL image half of the stack. This repo covers Quartz: a minimal, dependency-light Rust inference runtime for GGUF-format transformer language models, with a bundled Stable Diffusion SD1.5/SDXL image pipeline. CI-grade targets are a 4-core desktop CPU, a CUDA GPU, and an Android phone's Vulkan GPU, via a single ~15k-line, mostly- unsafe -free codebase. This project was previously developed under the name tinyq4 . All crate, binary, and environment-variable names now use quartz / QUARTZ ; the git history and any external references to tinyq4 predate the rename. Quartz ships as a single binary quartz with two capabilities: LLM chat inference — GGUF model loading, tokenization native GGUF vocab or a minijinja -templated Jinja2 chat template, matching what the source model ships , forward pass, and an OpenAI-compatible HTTP server for streaming completions. Image generation SD1.5 / SDXL — a from-scratch CLIP text encoder, UNet, VAE, and scheduler implementation, driven by the same binary via --generate-sd15 / --generate-sdxl . Video generation Wan2.1 T2V is a separate binary and a separate codebase — see Relationship to the Wan video engine relationship-to-the-wan-video-engine below. Don't conflate the two; they share the Quartz brand but not a source tree. CPU — portable Rust, with a hand-rolled AVX2/FMA dot-product path on x86 64 auto-dispatched at runtime; falls back to scalar otherwise and a hand-written NEON dot-product path for Q4 K on aarch64 Android . CUDA — optional --features cuda , custom GEMV kernels per quantization format src/cuda kernels.cu , src/cuda.rs , staged/paged weight loading, and a VRAM-aware KV cache that sizes itself to free VRAM rather than a hard token cap. Vulkan — optional --features vulkan , used for both LLM and SDXL compute; this is the backend the Android build ships with, since phones have no CUDA. Reads GGUF files directly src/gguf.rs and dequantizes on the fly: Q4 K, Q6 K, Q8 0, Q5 0/Q5 1, Q4 0/Q4 1, and the IQ2/IQ3 k-quant formats grid/sign tables in src/iq2 tables.rs / src/iq3 tables.rs , generated from upstream ggml-quants.c / ggml-common.h constant tables — the tables are data, not linked code . quartz --server exposes an OpenAI-compatible surface: GET /health GET /v1/health GET /v1/models POST /v1/chat/completions SSE streaming This is what both the desktop and mobile Saient clients speak — same wire format either side. cargo build --release CPU only cargo build --release --features cuda + CUDA GEMV kernels cargo build --release --features vulkan + Vulkan SD image gen, or Android Android arm64-v8a , used by the Saient mobile app: scripts/build-android-arm64.sh → target/aarch64-linux-android/release/quartz Requires ANDROID NDK HOME or a discoverable NDK under $ANDROID SDK ROOT/ndk . QUARTZ ANDROID API default 24 and QUARTZ ANDROID FEATURES default vulkan are overridable. | Variable | Effect | |---|---| QUARTZ BIND | Server bind address. Defaults to 127.0.0.1 loopback only ; set to 0.0.0.0 to opt into LAN/network access. | QUARTZ CTX | Upper-bounds the KV cache context length the host app requests. | QUARTZ CUDA ARCH | SASS target arch es for the CUDA build build.rs ; CI/shipping builds set all-major . | QUARTZ DENSE FFN CPU | Forces dense-FFN compute onto CPU CUDA path , for debugging/comparison. | QUARTZ DEBUG PROMPT | Dumps the fully-rendered prompt before tokenization. | QUARTZ NO JINJA | Skips the Jinja2 chat-template path even when the model has no native template. | QUARTZ NO WARMUP | Set by the Android host app before every engine start see note below . | Note on QUARTZ NO WARMUP: this flag is set by the mobile app plugins/quartz/QuartzEngine.kt in the Saient mobile repo to try to avoid a full-resident mmap warmup that can get the app OOM-killed on memory-constrained phones. As of this writing the Quartz binary does not actually read this variable anywhere — the mmap warmup always runs unconditionally see the "quartz: mmap warmup complete" log line in src/main.rs . This was discovered while renaming TINYQ4 NO WARMUP → QUARTZ NO WARMUP ; it is a pre-existing no-op on the engine side, not something this rename introduced or fixed. Worth following up on if low-memory devices are still seeing OOM kills at startup.The Saient mobile app also ships libquartz-wan.so , a second , unrelated binary used only for Wan2.1 text-to-video generation. It is not built from this repository — it's a pinned fork of leejet/stable-diffusion.cpp https://github.com/leejet/stable-diffusion.cpp MIT-licensed, C++/ggml , patched with a small progress-reporting patch and cross-compiled for Android/Vulkan. Its build lock lives in the mobile repo at engine/wan/BACKEND LOCK.json and engine/wan/saient-progress.patch . Both binaries carry the "Quartz" name in the mobile app's UI/branding QuartzService , QuartzEngine.kt supervise both processes , but they are two separate codebases with two separate license situations — this repo Quartz proper is what's being open-sourced here; the Wan fork's own license terms MIT and upstream attribution travel with it separately. See the mobile repo's docs/VIDEO PIPELINE.md for how the two engines are orchestrated together on-device. This repo has uncommitted local changes beyond the rename performed for open-sourcing a chat-template fix, KV-cache sizing fix, NEON Q4 K dot product, and the SD1.5/SDXL modules sd15.rs / sdxl .rs / tensor.rs / safetensors.rs / vulkan.rs are untracked as of this writing . Review and commit those separately before or as part of publishing — this README describes the working tree as it stands, not necessarily the last pushed commit. MIT — see LICENSE /SaientAI/saient-quartz/blob/master/LICENSE . Matches the license of the bundled Wan video engine see below , so the whole on-device stack is MIT.