GemmaEmbed Swift package optimized for Apple Silicon

Developer Maralski released GemmaEmbed 0.4.0, a dependency-free Swift package that runs Google's EmbeddingGemma 2 text, image and audio encoders in-process on Apple Silicon using its own Metal kernels, without llama.cpp, MLX or Core ML. The package produces 768-dimensional embeddings that match llama.cpp's to a cosine similarity of 0.99999 for text, 0.99999 for images and 0.99998 for audio in one shared space, loading the official embeddinggemma-2-Q8_0.gguf weights (309,855,456 bytes, 24 layers, 271M parameters) and an optional 554,821,024-byte mmproj projector for image and audio input. It requires macOS 14 or later on Apple Silicon and Swift 6, accepts up to 8192 tokens per text (default 2048), and supports Matryoshka truncation to 512, 256 or 128 dimensions.

GemmaEmbed runs Google's EmbeddingGemma 2 text, image and audio encoders in-process on Apple Silicon with its own Metal kernels. It is a Swift package with no dependencies beyond the Apple SDK: no llama.cpp, no MLX, no Core ML. You hand it the official GGUF weight files and get 768-dimensional embeddings that match llama.cpp's to a cosine similarity of 0.99999 for text, 0.99999 for images and 0.99998 for audio, in one shared space. | Model | embeddinggemma-2 GGUF architecture gemma-embedding2 , 24 layers, 271M parameters | | Weight file | embeddinggemma-2-Q8 0.gguf from https://huggingface.co/ggml-org/embeddinggemma-2-GGUF https://huggingface.co/ggml-org/embeddinggemma-2-GGUF | | Size | 309,855,456 bytes | | SHA256 | 2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135 | | Output | 768 dimensions, Matryoshka truncation to 512, 256 or 128 | | Input | up to 8192 tokens per text default limit 2048 , any number of texts per call | | Image and audio encoders optional | mmproj-embeddinggemma-2-Q8 0.gguf from the same repository, 554,821,024 bytes, SHA256 c4a8a52691ecef40618438928bdf9e68379b854e24166f292592353db0aab64f GGUF projectors gemma4v , 16 layers, and gemma4a , 12 Conformer layers; 473M parameters in all | | Image input | JPEG, PNG, HEIC and anything else ImageIO reads, or a CGImage ; any size, scaled to 70 to 1120 vision tokens default cap 256 | | Audio input | WAV parsed by GemmaEmbed , MP3, M4A, AIFF, CAF, FLAC through AVFoundation or PCM samples; any rate and channel count, resampled to 16 kHz mono; 25 rows per second, up to 5 minutes 27 seconds per clip | | Platform | macOS 14 or later on Apple Silicon, Swift 6 | The forward pass is documented in Sources/GemmaEmbed/Model/Architecture.md https://github.com/maralski/gemma-embed-swift/blob/main/Sources/GemmaEmbed/Model/Architecture.md . Add the package to Package.swift : dependencies: .package url: "https://github.com/maralski/gemma-embed-swift.git", from: "0.4.0" , , targets: .target name: "MyApp", dependencies: .product name: "GemmaEmbed", package: "gemma-embed-swift" , In Xcode, choose File Add Package Dependencies and enter the same URL. The weights are not part of the package. Download embeddinggemma-2-Q8 0.gguf and, for images and audio, mmproj-embeddinggemma-2-Q8 0.gguf from the Hugging Face repository above. For a side-by-side comparison with llama.cpp accuracy, speed, features, when to use which , see COMPARISON.md https://github.com/maralski/gemma-embed-swift/blob/main/COMPARISON.md . python import GemmaEmbed let model = try await EmbeddingModel.load from: URL fileURLWithPath: "embeddinggemma-2-Q8 0.gguf" let documents = try await model.embed "Rates were left unchanged.", "A new open-weight model shipped." let query = try await model.embed "central bank decision", options: EmbedOptions task: .query let best = documents.indices.max { model.similarity query, documents $0 < model.similarity query, documents $1 } EmbedOptions selects the model card's task prefix .document , .query , .clustering , .classification , .similarity , .none , the token limit, the output width and whether the vector is L2-normalised. embed batches texts by token count, reports progress after every GPU batch and stops between batches when the surrounding Task is cancelled. Every text is embedded identically whether it travels alone or inside a batch. Load the model with the image encoder and embed files, bytes or CGImage s. The vectors live in the same space as the text embeddings, so a caption query can be scored against pictures and a picture against pictures. js let model = try await EmbeddingModel.load from: textURL, options: LoadOptions imageEncoder: mmprojURL let pictures = try await model.embed images: .url photoURL , .data jpegBytes , .cgImage image let caption = try await model.embed "rolling green vineyard hills", options: EmbedOptions task: .query let score = model.similarity caption, pictures 0 print model.info.imageEncoder?.layerCount // 16 when loaded, nil otherwise ImageEmbedOptions carries the same dimensions and normalise as text, batchSize images per GPU batch, by default packed by patch count and maxImageTokens . The image encoder is a LoadOptions field rather than something added to a loaded model, so a model never changes after load ; loading text alone never opens the mmproj file, and embed images: without it throws imageEncoderNotLoaded . The encoder sees each image resized so that its patch grid gives at most maxImageTokens vision tokens 16 by 16 pixel patches, pooled 3 by 3 , and at least 70. The default cap of 256 is what llama-server applies with its default batch size; llama.cpp's native limit is 1120. The same photo embeds differently at different caps cosine 0.973 between 256 and 1120 on the 1600 by 1600 green hills scene , so a corpus and the queries against it must share one cap. JPEGs decode through a Swift port of stb image's decoder, byte for byte what llama.cpp sees; PNGs match it too. The preprocessing and the vision tower are documented in Sources/GemmaEmbed/Model/Architecture.md https://github.com/maralski/gemma-embed-swift/blob/main/Sources/GemmaEmbed/Model/Architecture.md . One input can mix text, images and audio. The model reads each picture and each clip where it sits among the text, in one forward pass, so a post with its photo and voice note embeds as one document. js let mmproj = URL fileURLWithPath: "mmproj-embeddinggemma-2-Q8 0.gguf" let model = try await EmbeddingModel.load from: textURL, options: LoadOptions imageEncoder: mmproj, audioEncoder: mmproj let post = MultimodalInput .text "Our walk on Saturday:" , .image .url photoURL , .text "and what the guide said:" , .audio .url voiceNoteURL let posts = try await model.embed post, MultimodalInput text: "Quarterly results", images: .data chartPNG let query = try await model.embed "guided hill walk", options: EmbedOptions task: .query let score = model.similarity query, posts 0 The sequence is the one llama.cpp's mtmd builds for EmbeddingGemma 2: bos text <|image image rows <image| text <|audio audio rows <audio| ... eos , with the parts in the order given. MultimodalEmbedOptions.task prefixes the first text, or stands alone before a picture or clip that opens the input. maxTokens counts every row; text is cut from the end until the input fits and pictures and clips are never cut, so an input whose pictures and clips alone exceed it throws invalidOptions . A text-only input embeds exactly as embed : does, and an image-only or audio-only input with task: .none exactly as embed images: or embed audios: does. Inputs with images need the image encoder and inputs with audio the audio encoder one mmproj file holds both ; text-only inputs need neither. Inputs of all three kinds batch together, and each embeds to the same bits alone or in a batch. The example's shape, a sentence, a fixture photo, a second sentence and a 3 s spoken clip 359 rows , embeds to cosine 0.9999997 against llama-server --embedding --mmproj given the same parts, max abs diff 9.3e-5, on 2026-10-26. Reference vectors come from llama.cpp build 11716 llama-server --embedding --mmproj over the same Q8 0 files, and per-node dumps of its vision graph taken through libmtmd 's eval callback scripts/vision dump.cpp . Every fixture image is drawn from code by scripts/make fixture images.swift : 10 procedural scenes green hills, an ocean sunset, a coral reef, snowy peaks, desert dunes, cherry blossom, a lavender field, a night sky, a sunflower field, a city skyline , 10 synthetic CoreGraphics images solid colours, gradients, text in three fonts, shapes, a checkerboard, noise and 10 near-duplicate variants of the scenes. scripts/oracle images.py writes the reference vectors from a running llama-server. | Check | Result | |---|---| | mmproj GGUF, 963 tensors and 30 metadata keys | identical to the directory dump | | Preprocessing on 6 images decode, size choice, Pillow-style bicubic resize, padding, normalisation, patch layout | byte-identical to llama.cpp's pixel tensor | | JPEG decoder on 36 JPEGs every sampling factor, progressive, grey, YCCK, restart markers | byte-identical to stb image | | CPU reference vision tower vs llama.cpp per-node sums, 3 images, 292 nodes each | within 1e-3, worst 8.5e-5 | | CPU reference vision vectors vs llama.cpp 512-wide rows | row cosine 0.99996 or better | | Metal kernels vs CPU reference patch GEMM, position tables, 2D RoPE, head-64 attention, gelu quick epilogue, 3x3 pooling, mixed-row embedding | max relative diff 5.0e-6 | | Metal vision tower vs CPU reference, 3 images | row cosine 0.9999993 or better, max abs diff 1.2e-4 | | Metal image embeddings vs llama-server, all 30 images, cap 256 | min cosine 0.9999985, max abs diff 2.8e-4 | | Metal image embeddings vs llama-server, 4 images at cap 1120 | min cosine 0.9999992, max abs diff 1.4e-4 | | Same image alone and inside a batch of 8 mixed sizes, first or last | bit-identical | | Text embeddings vs llama.cpp, 40 fixtures see Parity | min cosine 0.9999988 | Run swift test -c release to reproduce see Tests for where the tests find the image encoder, the llama.cpp dumps and the two images too large to ship in the package , or gemma-embed --parity-images Tests/GemmaEmbedTests/Fixtures/images . Five scenes scored against ten caption queries EmbedOptions task: .query , cosine . Five captions describe the scenes and five are off-topic. Every scene ranks its own caption first, by 0.125 or more over the next caption, and llama-server's own vectors give the same ranks. The scores below come from llama-server's vectors in Fixtures/images/captions.jsonl and embeddings.jsonl ; the test requires GemmaEmbed's to agree within 0.002. | caption | 00-green-hills | 01-ocean-sunset | 02-coral-reef | 05-cherry-blossom | 08-sunflower-field | |---|---|---|---|---|---| | rolling green hills striped with crop rows, a red barn and trees under a blue sky with white clouds | 0.795 | 0.526 | 0.560 | 0.536 | 0.658 | | the sun setting over the ocean with an orange sky and a palm tree silhouette | 0.488 | 0.750 | 0.602 | 0.521 | 0.560 | | an underwater coral reef with colourful corals and tropical fish | 0.481 | 0.570 | 0.752 | 0.538 | 0.528 | | pink cherry blossoms on dark branches against a pale blue sky | 0.525 | 0.567 | 0.574 | 0.787 | 0.562 | | a field of bright yellow sunflowers under a blue summer sky | 0.601 | 0.575 | 0.595 | 0.578 | 0.784 | | the central bank left interest rates unchanged for a third consecutive meeting | 0.528 | 0.502 | 0.506 | 0.517 | 0.528 | | a recipe for sourdough bread with a long cold fermentation | 0.487 | 0.468 | 0.440 | 0.463 | 0.480 | | quarterly earnings call transcript for a semiconductor company | 0.478 | 0.427 | 0.443 | 0.467 | 0.474 | | how to configure a Kubernetes ingress controller with TLS | 0.448 | 0.450 | 0.436 | 0.431 | 0.455 | | a black and white photograph of a crowded railway station | 0.452 | 0.446 | 0.452 | 0.467 | 0.490 | The smallest margin is the sunflower field, whose own caption 0.784 leads the green hills caption 0.658 . Off-topic captions score 0.43 to 0.53. Cosine between each scene and its variant, for choosing a deduplication threshold. Every variant has its original as nearest neighbour. The lowest variant pair is 0.896 the coral reef re-encoded at JPEG quality 40 , and the highest cosine between two unrelated images is 0.889 synthetic/01-solid-grey against synthetic/02-gradient-h . A threshold between the two separates them on this set, with a gap of 0.007. Among the scenes and their variants alone, the highest unrelated cosine is 0.822 the green hills and the sunflower field . | variant | pairs | min | median | max | |---|---|---|---|---| | resized to half | 1 | 0.996 | 0.996 | 0.996 | | resized to a third | 1 | 0.987 | 0.987 | 0.987 | | central crop keeping 90 percent | 1 | 0.998 | 0.998 | 0.998 | | top-left crop keeping 90 percent | 1 | 0.994 | 0.994 | 0.994 | | JPEG quality 40 | 2 | 0.896 | 0.943 | 0.989 | | flipped horizontally | 2 | 0.991 | 0.993 | 0.996 | | colour shift red +12, blue -12 | 2 | 0.994 | 0.995 | 0.995 | Measured on 2026-10-25 on the same Apple M3 Max as the text numbers, over 153 social media feed pictures of 2026-10-23 a private corpus, not in the repository; real JPEGs, 480 to 1280 pixels wide, 188 vision tokens on average , one picture per GPU batch as the default batching runs them, with gemma-embed --bench-images --batch-size 1 under a GPU lock, three rounds: | Configuration | Images per second median of 3 | Seconds for 153 images | GPU seconds | |---|---|---|---| | GemmaEmbed 0.3.0 | 10.40 | 14.72 | 14.48 | Image parity with llama-server is in Image parity above min cosine 0.9999985 over the 30 fixtures . A feed picture takes about 95 ms of GPU time, almost all of it the vision tower; the text tower over its rows takes about 3 ms. Against llama.cpp, on the same 153 pictures at the same cap of 256, twelve alternating rounds on a loaded machine CPU load moved every figure by a factor of two or more, so treat the ratio as approximate . Wall time includes image decoding on the CPU for both engines. scripts/compare llamacpp.py reproduces it; COMPARISON.md https://github.com/maralski/gemma-embed-swift/blob/main/COMPARISON.md has the details. | Configuration | Images per second, median of 12 rounds | Range | |---|---|---| | GemmaEmbed 0.3.0, one image per GPU batch | 8.46 | 4.48 to 11.04 | | GemmaEmbed 0.3.0, batched by patch count --batch-size auto | 7.84 | 3.00 to 11.64 | | llama.cpp 0.7.0-dev build 11716, llama-server --embedding --mmproj , 1 image per request | 5.91 | 2.87 to 7.91 | | llama.cpp, same build, 8 images per request | 6.04 | 3.03 to 8.29 | The GPU time itself grows with the batch: packing more images into one forward makes the vision GEMMs slower, not faster, from 1 to 8 images per batch. The likely cause is the activation buffers outgrowing the on-chip cache a batch of 8 full-size images is 18,432 patch rows, and its 3072-wide FFN activations take 226 MB : profiled, a batch of 8 pictures runs 17 percent slower than the sum of its kernels timed alone, while one picture runs faster than its kernel sum. Budgets of 8192 to 32768 patches measured 4 to 6 percent slower than 4096 and cost 2 to 4 times the memory. The default therefore packs images up to 4096 patch rows per batch, which is one image at the default cap and several smaller ones; ImageEmbedOptions.batchSize overrides it. Image encoder load time: 0.24 s on the first load after the file left the page cache mapping and prefaulting the 555 MB file, building the tower , 0.02 s on later loads. Loading the text model alone does not open the file. Reproduce: export GEMMA EMBED WEIGHTS=/path/to/embeddinggemma-2-Q8 0.gguf export GEMMA EMBED IMAGE WEIGHTS=/path/to/mmproj-embeddinggemma-2-Q8 0.gguf GEMMA EMBED TRACE=1 .build/release/gemma-embed --bench-images /path/to/images --runs 5 .build/release/gemma-embed --profile-images /path/to/images --batch-size 1 python3 scripts/ab images.py --a /path/to/old/gemma-embed --b .build/release/gemma-embed --dir /path/to/images --profile-images times each vision kernel class over the first --batch-size images, and --bench-images reports gpu seconds minsum , every batch's fastest GPU pass summed. scripts/ab images.py runs two builds alternately and checks image parity, text parity and that the candidate's vectors match the baseline's. Load the model with the audio encoder, which lives in the same mmproj file as the image encoder, and embed files, bytes or samples. The vectors live in the same space as the text and image embeddings, so a spoken sentence can be scored against its transcript, and a sound against a caption. js let model = try await EmbeddingModel.load from: textURL, options: LoadOptions audioEncoder: mmprojURL let clips = try await model.embed audios: .url wavURL , .data mp3Bytes , .samples pcm, sampleRate: 44100, channels: 2 let transcript = try await model.embed "the train now departs from platform four", options: EmbedOptions task: .query let score = model.similarity transcript, clips 0 print model.info.audioEncoder?.layerCount // 12 when loaded, nil otherwise AudioEmbedOptions carries the same dimensions and normalise as text, batchSize clips per GPU batch, by default packed by mel frames, two minutes of audio per batch and maxTokens . A clip costs one row per 40 ms plus four frame tokens; maxTokens defaults to the model's 8192 rows, which is 5 minutes 27 seconds, and a longer clip is refused with invalidOptions rather than cut, because llama.cpp does not cut audio either. Pass the mmproj file as imageEncoder , audioEncoder or both; the file is mapped once and only the towers asked for are built. Embedding audio without the encoder throws audioEncoderNotLoaded . The encoder sees llama.cpp's front end exactly: miniaudio's decode and resample to 16 kHz mono WAV bit for bit, including resampled and multichannel files; other codecs decode through AVFoundation, so an MP3 sits within 3e-5 per sample of llama.cpp's , 128 log-mel bins every 10 ms over a 20 ms window, and a split into 30 second chunks that the tower encodes one by one. A MultimodalInput takes .audio parts beside .text and .image parts, framed with <|audio and <audio| where they sit. The front end and the tower are documented in Sources/GemmaEmbed/Model/Architecture.md https://github.com/maralski/gemma-embed-swift/blob/main/Sources/GemmaEmbed/Model/Architecture.md . Reference vectors come from llama.cpp build 11716 llama-server --embedding --mmproj -ub 8192 over the same Q8 0 files, and per-node dumps of its audio graph taken through libmtmd 's eval callback scripts/audio dump.cpp . Every fixture clip is synthesised by scripts/make fixture audio.py : tones, a chord, bursts, chirps, noise, clicks, AM and FM signals, silence, 0.3 to 7 second signals, the same signal at 8, 22.05, 44.1 and 48 kHz, in stereo and four channels, and as 8, 24 and 32-bit integer and float32 WAV, plus a 30.0 s clip and a 35 s clip that crosses the chunk boundary. No recordings and no voices are in the repository. scripts/oracle audio.py writes the reference vectors from a running llama-server. | Check | Result | |---|---| | WAV decode and miniaudio's downmix and resample, 32 variants bit depths, 3 to 6 channels, 8 to 96 kHz , plus FLAC, AIFF and CAF of one of them | bit-identical to miniaudio's samples | | Log-mel front end vs llama.cpp's graph input, 8 chunks of 7 clips | bins above -2 within 3e-5; bins between -6 and -2 within 1.2e-3; bins at the 0.001 floor within 2.4e-3 | | CPU reference audio tower vs llama.cpp per-node sums, 8 chunks, 2448 nodes | within 2.7e-4 of the absolute sum | | CPU reference vs llama.cpp, 589 full tensors | cosine 0.999995 or better | | Metal audio tower vs CPU reference, 7 clips including the two-chunk 35 s clip | row cosine 0.999999 or better, max abs diff 3.6e-3 on rows of norm 28 | | Metal audio embeddings vs llama-server, all 29 fixture clips | min cosine 0.99998933 the 35 s clip , max abs diff 7.2e-4 | | A generated 5-minute clip 11 chunks, 7504 rows, not in the repository vs llama-server | cosine 0.9999937 | | Same clip alone and inside a batch of 5 mixed lengths, first or last | bit-identical | | Text and image parity | unchanged see Image parity and Parity | Run swift test -c release to reproduce see Tests , or gemma-embed --parity-audio Tests/GemmaEmbedTests/Fixtures/audio . Measured on 2026-10-26 on the same Apple M3 Max as the text numbers, over 40 spoken clips made by macOS say scripts/make speech bench.py , not in the repository: 715 s in all, 3.3 to 85 s each, median 8.7 s , under a GPU lock, at a load average of about 4 to 6. Wall time includes WAV decoding and the log-mel front end on the CPU. | Engine | Audio seconds per second, median of 5 rounds | Range | |---|---|---| | GemmaEmbed, default batching by mel frames | 274.9 | 271.0 to 324.7 | | GemmaEmbed, one clip per GPU batch | 235.9 | 225.9 to 271.9 | | llama.cpp build 11716, llama-server --embedding --mmproj , one clip per request | 153.6 | 148.6 to 160.2 | scripts/compare llamacpp audio.py models/audio/bench 5 reproduces the table; llama-server's side includes HTTP and base64, so the gap slightly overstates the difference between the engines. Over the same clips the two engines' vectors agree to min cosine 0.9999994. Against the first correct implementation, the speed work made audio embedding 1.34 times faster in wall time with default batching and 1.29 times with one clip per batch interleaved A/B runs with scripts/ab audio.py , interquartile range 1.30 to 1.35 , with the same vectors to within 3.4e-7. Most of it came from the CPU: the log-mel front end runs an in-place FFT and sums only the non-zero taps of each mel filter about 1.17 times wall on its own, bit-identical features , and the next batch's clips are decoded while the GPU runs the current one. On the GPU, attention runs one simdgroup per query and head 12 times faster per layer on long clips and the second subsampling convolution is an implicit GEMM on simdgroup matrices about 4 percent of GPU time . Wall time is now bound by the GPU, and three quarters of that is GEMMs already running near 10 TFLOPS. Tuning GEMM tiles for the audio shapes, split-K, fusing the elementwise passes, larger or smaller batches, and double-buffered command encoding each measured within noise. Ten sentences spoken by macOS say scripts/make speech samples.sh , not in the repository are scored against their ten transcripts and five off-topic texts, all embedded as queries. Every clip ranks its own transcript first, by 0.17 or more over the next text, and llama-server's own vectors give the same ranks; GemmaEmbed's similarities agree with llama-server's within 1e-4. Own cosines run from 0.80 to 0.90, and the smallest margin is a recipe step whose nearest other text is the other cooking sentence. The test skips when the speech folder is absent. swift build -c release export GEMMA EMBED WEIGHTS=/path/to/embeddinggemma-2-Q8 0.gguf echo "hello world" | .build/release/gemma-embed --task query --dimensions 256 .build/release/gemma-embed --parity Tests/GemmaEmbedTests/Fixtures export GEMMA EMBED IMAGE WEIGHTS=/path/to/mmproj-embeddinggemma-2-Q8 0.gguf .build/release/gemma-embed --images photo.jpg ~/Pictures .build/release/gemma-embed --parity-images Tests/GemmaEmbedTests/Fixtures/images .build/release/gemma-embed --bench-images ~/Pictures --runs 3 .build/release/gemma-embed --bench-images ~/Pictures --batch-size auto .build/release/gemma-embed --profile-images ~/Pictures --batch-size 1 .build/release/gemma-embed --bench posts.json --runs 3 .build/release/gemma-embed --profile posts.json .build/release/gemma-embed --audio talk.wav ~/Recordings .build/release/gemma-embed --parity-audio Tests/GemmaEmbedTests/Fixtures/audio .build/release/gemma-embed --bench-audio ~/Recordings --runs 3 .build/release/gemma-embed --profile-audio ~/Recordings --batch-size 1 gemma-embed prints one JSON object per input line. --parity checks the tokenizer and the embeddings against the llama.cpp fixtures and exits non-zero below cosine 0.9995. --bench times a throughput run over a feed JSON file {"posts": {"gist", "text"} } or a plain text file with one document per line. Image modes take the mmproj file from --image-encoder or $GEMMA EMBED IMAGE WEIGHTS : --images FILE... | DIR prints one JSON vector per image, --parity-images DIR checks a fixtures folder manifest.json and embeddings.jsonl and --bench-images DIR measures images per second, by default with patch-count batching, 8 and 1 images per batch --batch-size auto or N picks one . Audio modes take the same mmproj file from --audio-encoder , $GEMMA EMBED AUDIO WEIGHTS or the image encoder path: --audio FILE... | DIR prints one JSON vector per clip with its row count and seconds, --parity-audio DIR checks a fixtures folder, --bench-audio DIR measures audio seconds per wall second and per GPU second, and --profile-audio DIR times each audio kernel class. --help lists the rest. The shaders compile from .metal source when a model loads. GemmaEmbed looks for the SwiftPM resource bundle GemmaEmbed GemmaEmbed.bundle in this order: 1. The main bundle's resources: Contents/Resources inside an app. 2. The resources of the bundle that holds the GemmaEmbed code, then the directory next to it. 3. The main bundle's own directory, which is where swift run and a bare executable find it. If no copy is found, GemmaEmbed uses the shader source compiled into the library, so a model still loads with no bundle at all. It never reads the checkout's .build directory, which SwiftPM's Bundle.module falls back to. An app built outside Xcode must copy the bundle into Contents/Resources . Codesign refuses loose files at the root of an .app . A build script can do it with this line, after it sets SHADERS="$BIN DIR/GemmaEmbed GemmaEmbed.bundle" from swift build -c release --show-bin-path : cp -R "$SHADERS" "$APP/Contents/Resources/" After you edit a file in Sources/GemmaEmbed/Shaders , run scripts/embed shaders.py to regenerate Sources/GemmaEmbed/Metal/EmbeddedShaders.swift . ShaderSourceTests fails while the two disagree. swift test -c release runs the suite. The debug configuration also passes, in about six minutes, because the CPU reference runs unoptimised. The tests read the weights and the llama.cpp dumps from models/ at the package root, which is gitignored. An environment variable overrides each path, and a test whose input is missing skips. | Path | Contents | Override | |---|---|---| | models/embeddinggemma-2-Q8 0.gguf | the text weights | GEMMA EMBED WEIGHTS | | models/mmproj-embeddinggemma-2-Q8 0.gguf | the image encoder | GEMMA EMBED IMAGE WEIGHTS | | models/dumps/ | llama.cpp's vision graph for six images scripts/vision dump.cpp | GEMMA EMBED VISION DUMPS | | models/dumps-rgb/ | stb image's decode of every JPEG fixture scripts/vision dump.cpp | GEMMA EMBED DECODED RGB DUMPS | | models/fixtures/images/ | the full image fixture set from scripts/make fixture images.swift , including the lavender field scene and its variant, which are over 250 KB and not in the package | none | | models/audio/dumps/ | llama.cpp's audio graph for seven clips scripts/audio dump.cpp and miniaudio's decode of the decode-check WAVs scripts/make decode checks.py | GEMMA EMBED AUDIO DUMPS | | models/audio/speech/ | ten spoken sentences from scripts/make speech samples.sh with llama-server's vectors from scripts/oracle audio.py --speech | none | scripts/vision dump.cpp and scripts/audio dump.cpp list the commands that write the dump directories. Reference vectors come from llama.cpp build 11716 llama-server --embedding over the same Q8 0 file, which is itself an approximation of the fp32 model, so agreement is bounded by that oracle. | Check | Result | |---|---| | Tokenizer, 40 fixture texts five scripts, URLs, numbers, emoji | token ids identical | | CPU fp32 reference vs llama.cpp per-tensor activation sums, 3 texts | every named tensor within 2 percent | | CPU fp32 reference vs llama.cpp embeddings, 3 texts | min cosine 0.9999992, max abs diff 8.5e-5 | | Metal kernels vs CPU reference GEMM all shapes and tiles, RMSNorm, RoPE, attention at 1 to 2048 tokens, GeGLU, pooling | max relative diff 4.5e-6 | | Metal forward vs llama.cpp, all 40 fixtures | min cosine 0.9999988, max abs diff 1.4e-4 | | Metal forward vs llama.cpp at 1054, 1629 and 3222 tokens sliding window active | cosine 0.9999995 or better | | Metal forward vs CPU reference at 1672 tokens | cosine 0.9999999, max abs diff 1.8e-7 | | Same text alone and inside a batch of 33 | bit-identical | Run swift test -c release to reproduce see Tests . scripts/make text fixtures.py regenerates the text fixtures from llama.cpp. Measured on 2026-10-25 on an Apple M3 Max 40 GPU cores, 128 GB running macOS 26, Xcode 26.3, under a GPU lock with no other GPU work and a load average of 3.3, five runs. The corpus is 285 social media posts of 2026-10-17, a private set that is not in the repository gist plus text cut at 2000 characters, document prefix , 41,675 tokens in all, 146 tokens per post on average, longest 604. | Engine | Posts per second | Seconds for 285 posts | Tokens per second | |---|---|---|---| | GemmaEmbed 0.3.0 | 176.4 | 1.62 1.61 to 1.66 | 25,790 | | llama.cpp 0.7.0-dev build 11716, llama-server --embedding -np 8 -ub 8192 , 32 texts per request | 118.3 | 2.41 | 17,300 | | llama.cpp, same build, -np 1 one sequence at a time | 59.6 | 4.78 | 8,720 | GemmaEmbed 0.3.0 is 1.49 times faster than llama.cpp's best configuration and 2.96 times faster than its single-slot default, with the same parity min cosine 0.9999988 against llama.cpp over the 40 fixtures . A single 22-token query takes 7.4 ms wall. The GEMMs take most of the GPU time and run at 7.6 to 8.8 TFLOPS fp32 in a sustained microbench; attention takes about an eighth. Measuring on a shared machine: GPU time of short command buffers reads high because the GPU runs at a low clock, so per-kernel lines of --profile mislead for kernels under about 2 ms, and a microbench of short bursts can rank GEMM tiles the wrong way round. scripts/ab bench.py runs two builds alternately and calls a difference only when the median, best-of and per-batch-minimum ratios agree; scripts/gpulock.py keeps two measurements off the GPU at once. Load time: 0.27 s when the Metal shader cache is warm tokenizer 0.17 s , about 1 s on the first load after a build shader compilation 0.25 s, pipelines 0.3 s . Autotune, when enabled, adds about 4 s on the first load and nothing afterwards. Reproduce: swift build -c release export GEMMA EMBED WEIGHTS=/path/to/embeddinggemma-2-Q8 0.gguf .build/release/gemma-embed --bench /path/to/posts.json --runs 3 llama-server -m /path/to/embeddinggemma-2-Q8 0.gguf --embedding -c 8192 -b 8192 -ub 8192 -np 8 --port 8092 & python3 scripts/bench llamacpp.py /path/to/posts.json --port 8092 --batch 32 --runs 3 python3 scripts/ab bench.py --a /path/to/old/gemma-embed --b .build/release/gemma-embed --bench /path/to/posts.json The kernels follow these design rules: - The weight file is mapped MAP SHARED and read-only, pre-touched from several threads, and wrapped in one MTLBuffer with no copy. Tensors are offsets into that buffer. - Shaders ship as .metal sources and compile when the model loads 0.2 s , because SwiftPM does not produce a metallib from the command line. Every pipeline is built at load. See Packaging for where the sources are read from. - All arithmetic is fp32. Q8 0 and BF16 are storage formats widened inside the kernels. - GEMM uses simdgroup float8x8 with X and dequantized W tiles staged in threadgroup memory, and stages the output tile so rows leave as coalesced float4 stores. The tile is picked by output size smaller tiles for small batches, a wider one for inner dimensions above 2048 , and every tile accumulates over K in the same order, so a row's result is bit-identical whatever batch it travelled in. Every loop over fragment arrays is fully unrolled: a rolled loop keeps the arrays in memory and runs several times slower. - Attention is the new work: flash-style, bidirectional, over sequences packed back to back without padding. A threadgroup owns 8 to 64 query rows of one sequence for one head and streams 32-key blocks staged once in threadgroup memory; the output accumulators stay in registers and the online softmax runs in fp32 on the score fragments themselves. Small batches split each head into narrower slices to fill the GPU, and the vision tower has its own kernel for its 64-wide heads. Sliding-window layers mask |i - j| 512 , the symmetric half window llama.cpp uses. - The forward runs on a concurrent compute encoder with a buffer barrier after every dispatch, except within small groups of independent dispatches q, k and v, for instance , which may overlap. A stress test runs two batches alternately on it and checks every result against a serial encoder bit for bit. - Padding is avoided entirely: texts are concatenated, every per-token kernel runs on the concatenation, and attention takes a work list of sequence, query block pairs. - Autotune, off by default, times whole forwards on real text for the GEMM tile and the attention query group, requires a 5 percent win, confirms the winner against the default and caches the result per device, kernel digest and model hash. - iOS and visionOS targets. With .iOS .v18 added to the platforms, the library compiles for iOS unchanged iOS 17 lacks MTLCompileOptions.mathMode ; it has not yet run on a device or simulator, and visionOS has not been built. - F16 weight files, which are refused at load today, and a check of BF16 files against llama.cpp the GEMM kernels read BF16 weights, but no BF16 file has been tested . - Fusing the text tower's q, k and v projections into one grouped GEMM, as the vision and audio towers already do, and the per-layer-input gate into fewer dispatches. Peter Marelas @maralski https://github.com/maralski . Copyright 2026 Peter Marelas. Apache License 2.0, the license Google releases EmbeddingGemma 2 and the Gemma 4 models under. See LICENSE https://github.com/maralski/gemma-embed-swift/blob/main/LICENSE and NOTICE https://github.com/maralski/gemma-embed-swift/blob/main/NOTICE . The model weights are not included and are licensed separately by Google; Google asks that deployments follow the Gemma Prohibited Use Policy https://ai.google.dev/gemma/prohibited use policy . Code ported from llama.cpp and stb image keeps its MIT notices in THIRD PARTY NOTICES https://github.com/maralski/gemma-embed-swift/blob/main/THIRD PARTY NOTICES . Gemma is a trademark of Google LLC. This project is not affiliated with or endorsed by Google.