{"slug": "gemmaembed-swift-package-optimized-for-apple-silicon", "title": "GemmaEmbed Swift package optimized for Apple Silicon", "summary": "Developer Maralski released GemmaEmbed 0.4.0, a dependency-free Swift package that runs Google's EmbeddingGemma 2 text, image and audio encoders in-process on Apple Silicon using its own Metal kernels, without llama.cpp, MLX or Core ML. The package produces 768-dimensional embeddings that match llama.cpp's to a cosine similarity of 0.99999 for text, 0.99999 for images and 0.99998 for audio in one shared space, loading the official embeddinggemma-2-Q8_0.gguf weights (309,855,456 bytes, 24 layers, 271M parameters) and an optional 554,821,024-byte mmproj projector for image and audio input. It requires macOS 14 or later on Apple Silicon and Swift 6, accepts up to 8192 tokens per text (default 2048), and supports Matryoshka truncation to 512, 256 or 128 dimensions.", "body_md": "GemmaEmbed runs Google's EmbeddingGemma 2 text, image and audio encoders in-process on Apple Silicon with its own Metal kernels. It is a Swift package with no dependencies beyond the Apple SDK: no llama.cpp, no MLX, no Core ML. You hand it the official GGUF weight files and get 768-dimensional embeddings that match llama.cpp's to a cosine similarity of 0.99999 for text, 0.99999 for images and 0.99998 for audio, in one shared space.\n\n| Model | `embeddinggemma-2` (GGUF architecture`gemma-embedding2` ), 24 layers, 271M parameters | \n| Weight file | `embeddinggemma-2-Q8_0.gguf` from[https://huggingface.co/ggml-org/embeddinggemma-2-GGUF](https://huggingface.co/ggml-org/embeddinggemma-2-GGUF) | \n| Size | 309,855,456 bytes | \n| SHA256 | `2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135` | \n| Output | 768 dimensions, Matryoshka truncation to 512, 256 or 128 | \n| Input | up to 8192 tokens per text (default limit 2048), any number of texts per call | \n| Image and audio encoders (optional) | `mmproj-embeddinggemma-2-Q8_0.gguf` from the same repository, 554,821,024 bytes, SHA256`c4a8a52691ecef40618438928bdf9e68379b854e24166f292592353db0aab64f` (GGUF projectors`gemma4v` , 16 layers, and`gemma4a` , 12 Conformer layers; 473M parameters in all) | \n| Image input | JPEG, PNG, HEIC and anything else ImageIO reads, or a `CGImage` ; any size, scaled to 70 to 1120 vision tokens (default cap 256) | \n| Audio input | WAV (parsed by GemmaEmbed), MP3, M4A, AIFF, CAF, FLAC (through AVFoundation) or PCM samples; any rate and channel count, resampled to 16 kHz mono; 25 rows per second, up to 5 minutes 27 seconds per clip | \n| Platform | macOS 14 or later on Apple Silicon, Swift 6 | \n\nThe forward pass is documented in [Sources/GemmaEmbed/Model/Architecture.md](https://github.com/maralski/gemma-embed-swift/blob/main/Sources/GemmaEmbed/Model/Architecture.md).\n\nAdd the package to `Package.swift`:\n\n```\ndependencies: [\n    .package(url: \"https://github.com/maralski/gemma-embed-swift.git\", from: \"0.4.0\"),\n],\ntargets: [\n    .target(name: \"MyApp\", dependencies: [.product(name: \"GemmaEmbed\", package: \"gemma-embed-swift\")]),\n]\n```\n\nIn Xcode, choose File > Add Package Dependencies and enter the same URL. The weights are not\npart of the package. Download `embeddinggemma-2-Q8_0.gguf` (and, for images and audio,\n`mmproj-embeddinggemma-2-Q8_0.gguf`) from the Hugging Face repository above.\n\nFor a side-by-side comparison with llama.cpp (accuracy, speed, features, when to use which),\nsee [COMPARISON.md](https://github.com/maralski/gemma-embed-swift/blob/main/COMPARISON.md).\n\n``` python\nimport GemmaEmbed\n\nlet model = try await EmbeddingModel.load(from: URL(fileURLWithPath: \"embeddinggemma-2-Q8_0.gguf\"))\nlet documents = try await model.embed([\"Rates were left unchanged.\", \"A new open-weight model shipped.\"])\nlet query = try await model.embed(\"central bank decision\", options: EmbedOptions(task: .query))\nlet best = documents.indices.max { model.similarity(query, documents[$0]) < model.similarity(query, documents[$1]) }\n```\n\n`EmbedOptions` selects the model card's task prefix (`.document`, `.query`, `.clustering`,\n`.classification`, `.similarity`, `.none`), the token limit, the output width and whether the\nvector is L2-normalised. `embed` batches texts by token count, reports progress after every GPU\nbatch and stops between batches when the surrounding `Task` is cancelled. Every text is embedded\nidentically whether it travels alone or inside a batch.\n\nLoad the model with the image encoder and embed files, bytes or `CGImage` s. The vectors live in\nthe same space as the text embeddings, so a caption query can be scored against pictures and a\npicture against pictures.\n\n``` js\nlet model = try await EmbeddingModel.load(from: textURL, options: LoadOptions(imageEncoder: mmprojURL))\nlet pictures = try await model.embed(images: [.url(photoURL), .data(jpegBytes), .cgImage(image)])\nlet caption = try await model.embed(\"rolling green vineyard hills\", options: EmbedOptions(task: .query))\nlet score = model.similarity(caption, pictures[0])\nprint(model.info.imageEncoder?.layerCount)   // 16 when loaded, nil otherwise\n```\n\n`ImageEmbedOptions` carries the same `dimensions` and `normalise` as text, `batchSize` (images\nper GPU batch, by default packed by patch count) and `maxImageTokens`. The image encoder is a\n`LoadOptions` field rather than something added to a loaded model, so a model never changes\nafter `load`; loading text alone never opens the mmproj file, and `embed(images:)` without it\nthrows `imageEncoderNotLoaded`.\n\nThe encoder sees each image resized so that its patch grid gives at most `maxImageTokens`\nvision tokens (16 by 16 pixel patches, pooled 3 by 3), and at least 70. The default cap of 256 is\nwhat `llama-server` applies with its default batch size; llama.cpp's native limit is 1120. The\nsame photo embeds differently at different caps (cosine 0.973 between 256 and 1120 on the 1600\nby 1600 green hills scene), so a corpus and the queries against it must share one cap. JPEGs\ndecode through a Swift port of stb_image's decoder, byte for byte what llama.cpp sees; PNGs\nmatch it too. The preprocessing and the vision tower are documented in\n[Sources/GemmaEmbed/Model/Architecture.md](https://github.com/maralski/gemma-embed-swift/blob/main/Sources/GemmaEmbed/Model/Architecture.md).\n\nOne input can mix text, images and audio. The model reads each picture and each clip where it sits among the text, in one forward pass, so a post with its photo and voice note embeds as one document.\n\n``` js\nlet mmproj = URL(fileURLWithPath: \"mmproj-embeddinggemma-2-Q8_0.gguf\")\nlet model = try await EmbeddingModel.load(from: textURL, options: LoadOptions(imageEncoder: mmproj, audioEncoder: mmproj))\nlet post = MultimodalInput([.text(\"Our walk on Saturday:\"), .image(.url(photoURL)), .text(\"and what the guide said:\"), .audio(.url(voiceNoteURL))])\nlet posts = try await model.embed([post, MultimodalInput(text: \"Quarterly results\", images: [.data(chartPNG)])])\nlet query = try await model.embed(\"guided hill walk\", options: EmbedOptions(task: .query))\nlet score = model.similarity(query, posts[0])\n```\n\nThe sequence is the one llama.cpp's `mtmd` builds for EmbeddingGemma 2:\n`[bos] text [<|image>] image rows [<image|>] text [<|audio>] audio rows [<audio|>] ... [eos]`,\nwith the parts in the order given. `MultimodalEmbedOptions.task` prefixes the first text, or\nstands alone before a picture or clip that opens the input. `maxTokens` counts every row; text is\ncut from the end until the input fits and pictures and clips are never cut, so an input whose\npictures and clips alone exceed it throws `invalidOptions`. A text-only input embeds exactly as\n`embed(_:)` does, and an image-only or audio-only input with `task: .none` exactly as\n`embed(images:)` or `embed(audios:)` does. Inputs with images need the image encoder and inputs\nwith audio the audio encoder (one mmproj file holds both); text-only inputs need neither. Inputs\nof all three kinds batch together, and each embeds to the same bits alone or in a batch.\n\nThe example's shape, a sentence, a fixture photo, a second sentence and a 3 s spoken clip (359\nrows), embeds to cosine 0.9999997 against `llama-server --embedding --mmproj` given the same\nparts, max abs diff 9.3e-5, on 2026-10-26.\n\nReference vectors come from llama.cpp build 11716 (`llama-server --embedding --mmproj`) over the\nsame Q8_0 files, and per-node dumps of its vision graph taken through `libmtmd`'s eval callback\n(`scripts/vision_dump.cpp`). Every fixture image is drawn from code by\n`scripts/make_fixture_images.swift`: 10 procedural scenes (green hills, an ocean sunset, a coral\nreef, snowy peaks, desert dunes, cherry blossom, a lavender field, a night sky, a sunflower field,\na city skyline), 10 synthetic CoreGraphics images (solid colours, gradients, text in three fonts,\nshapes, a checkerboard, noise) and 10 near-duplicate variants of the scenes.\n`scripts/oracle_images.py` writes the reference vectors from a running llama-server.\n\n| Check | Result | \n|---|---|\n| mmproj GGUF, 963 tensors and 30 metadata keys | identical to the directory dump | \n| Preprocessing on 6 images (decode, size choice, Pillow-style bicubic resize, padding, normalisation, patch layout) | byte-identical to llama.cpp's pixel tensor | \n| JPEG decoder on 36 JPEGs (every sampling factor, progressive, grey, YCCK, restart markers) | byte-identical to stb_image | \n| CPU reference vision tower vs llama.cpp per-node sums, 3 images, 292 nodes each | within 1e-3, worst 8.5e-5 | \n| CPU reference vision vectors vs llama.cpp (512-wide rows) | row cosine 0.99996 or better | \n| Metal kernels vs CPU reference (patch GEMM, position tables, 2D RoPE, head-64 attention, gelu_quick epilogue, 3x3 pooling, mixed-row embedding) | max relative diff 5.0e-6 | \n| Metal vision tower vs CPU reference, 3 images | row cosine 0.9999993 or better, max abs diff 1.2e-4 | \n| Metal image embeddings vs llama-server, all 30 images, cap 256 | min cosine 0.9999985, max abs diff 2.8e-4 | \n| Metal image embeddings vs llama-server, 4 images at cap 1120 | min cosine 0.9999992, max abs diff 1.4e-4 | \n| Same image alone and inside a batch of 8 mixed sizes, first or last | bit-identical | \n| Text embeddings vs llama.cpp, 40 fixtures (see Parity) | min cosine 0.9999988 | \n\nRun `swift test -c release` to reproduce (see Tests for where the tests find the image encoder,\nthe llama.cpp dumps and the two images too large to ship in the package), or\n`gemma-embed --parity-images Tests/GemmaEmbedTests/Fixtures/images`.\n\nFive scenes scored against ten caption queries (`EmbedOptions(task: .query)`, cosine). Five\ncaptions describe the scenes and five are off-topic. Every scene ranks its own caption first, by\n0.125 or more over the next caption, and llama-server's own vectors give the same ranks. The\nscores below come from llama-server's vectors in `Fixtures/images/captions.jsonl` and\n`embeddings.jsonl`; the test requires GemmaEmbed's to agree within 0.002.\n\n| caption | 00-green-hills | 01-ocean-sunset | 02-coral-reef | 05-cherry-blossom | 08-sunflower-field | \n|---|---|---|---|---|---|\n| rolling green hills striped with crop rows, a red barn and trees under a blue sky with white clouds | **0.795** | 0.526 | 0.560 | 0.536 | 0.658 | \n| the sun setting over the ocean with an orange sky and a palm tree silhouette | 0.488 | **0.750** | 0.602 | 0.521 | 0.560 | \n| an underwater coral reef with colourful corals and tropical fish | 0.481 | 0.570 | **0.752** | 0.538 | 0.528 | \n| pink cherry blossoms on dark branches against a pale blue sky | 0.525 | 0.567 | 0.574 | **0.787** | 0.562 | \n| a field of bright yellow sunflowers under a blue summer sky | 0.601 | 0.575 | 0.595 | 0.578 | **0.784** | \n| the central bank left interest rates unchanged for a third consecutive meeting | 0.528 | 0.502 | 0.506 | 0.517 | 0.528 | \n| a recipe for sourdough bread with a long cold fermentation | 0.487 | 0.468 | 0.440 | 0.463 | 0.480 | \n| quarterly earnings call transcript for a semiconductor company | 0.478 | 0.427 | 0.443 | 0.467 | 0.474 | \n| how to configure a Kubernetes ingress controller with TLS | 0.448 | 0.450 | 0.436 | 0.431 | 0.455 | \n| a black and white photograph of a crowded railway station | 0.452 | 0.446 | 0.452 | 0.467 | 0.490 | \n\nThe smallest margin is the sunflower field, whose own caption (0.784) leads the green hills caption (0.658). Off-topic captions score 0.43 to 0.53.\n\nCosine between each scene and its variant, for choosing a deduplication threshold. Every variant\nhas its original as nearest neighbour. The lowest variant pair is 0.896 (the coral reef\nre-encoded at JPEG quality 40), and the highest cosine between two unrelated images is 0.889\n(`synthetic/01-solid-grey` against `synthetic/02-gradient-h`). A threshold between the two\nseparates them on this set, with a gap of 0.007. Among the scenes and their variants alone, the\nhighest unrelated cosine is 0.822 (the green hills and the sunflower field).\n\n| variant | pairs | min | median | max | \n|---|---|---|---|---|\n| resized to half | 1 | 0.996 | 0.996 | 0.996 | \n| resized to a third | 1 | 0.987 | 0.987 | 0.987 | \n| central crop keeping 90 percent | 1 | 0.998 | 0.998 | 0.998 | \n| top-left crop keeping 90 percent | 1 | 0.994 | 0.994 | 0.994 | \n| JPEG quality 40 | 2 | 0.896 | 0.943 | 0.989 | \n| flipped horizontally | 2 | 0.991 | 0.993 | 0.996 | \n| colour shift (red +12, blue -12) | 2 | 0.994 | 0.995 | 0.995 | \n\nMeasured on 2026-10-25 on the same Apple M3 Max as the text numbers, over 153 social media feed\npictures of 2026-10-23 (a private corpus, not in the repository; real JPEGs, 480 to 1280 pixels\nwide, 188 vision tokens on average), one picture per GPU batch as the default batching runs\nthem, with `gemma-embed --bench-images --batch-size 1` under a GPU lock, three rounds:\n\n| Configuration | Images per second (median of 3) | Seconds for 153 images | GPU seconds | \n|---|---|---|---|\n| GemmaEmbed 0.3.0 | 10.40 | 14.72 | 14.48 | \n\nImage parity with llama-server is in Image parity above (min cosine 0.9999985 over the 30 fixtures). A feed picture takes about 95 ms of GPU time, almost all of it the vision tower; the text tower over its rows takes about 3 ms.\n\nAgainst llama.cpp, on the same 153 pictures at the same cap of 256, twelve alternating rounds\non a loaded machine (CPU load moved every figure by a factor of two or more, so treat the ratio\nas approximate). Wall time includes image decoding on the CPU for both engines.\n`scripts/compare_llamacpp.py` reproduces it; [COMPARISON.md](https://github.com/maralski/gemma-embed-swift/blob/main/COMPARISON.md) has the details.\n\n| Configuration | Images per second, median of 12 rounds | Range | \n|---|---|---|\n| GemmaEmbed 0.3.0, one image per GPU batch | 8.46 | 4.48 to 11.04 | \n| GemmaEmbed 0.3.0, batched by patch count ( `--batch-size auto` ) | 7.84 | 3.00 to 11.64 | \n| llama.cpp 0.7.0-dev build 11716, `llama-server --embedding --mmproj` , 1 image per request | 5.91 | 2.87 to 7.91 | \n| llama.cpp, same build, 8 images per request | 6.04 | 3.03 to 8.29 | \n\nThe GPU time itself grows with the batch: packing more images into one forward makes the vision\nGEMMs slower, not faster, from 1 to 8 images per batch. The likely cause is the\nactivation buffers outgrowing the on-chip cache (a batch of 8 full-size images is 18,432 patch\nrows, and its 3072-wide FFN activations take 226 MB): profiled, a batch of 8 pictures\nruns 17 percent slower than the sum of its kernels timed alone, while one picture runs faster\nthan its kernel sum. Budgets of 8192 to 32768 patches measured 4 to 6 percent slower than 4096\nand cost 2 to 4 times the memory. The\ndefault therefore packs images up to 4096 patch rows per batch, which is one image at the\ndefault cap and several smaller ones; `ImageEmbedOptions.batchSize` overrides it.\n\nImage encoder load time: 0.24 s on the first load after the file left the page cache (mapping and prefaulting the 555 MB file, building the tower), 0.02 s on later loads. Loading the text model alone does not open the file.\n\nReproduce:\n\n```\nexport GEMMA_EMBED_WEIGHTS=/path/to/embeddinggemma-2-Q8_0.gguf\nexport GEMMA_EMBED_IMAGE_WEIGHTS=/path/to/mmproj-embeddinggemma-2-Q8_0.gguf\nGEMMA_EMBED_TRACE=1 .build/release/gemma-embed --bench-images /path/to/images --runs 5\n.build/release/gemma-embed --profile-images /path/to/images --batch-size 1\npython3 scripts/ab_images.py --a /path/to/old/gemma-embed --b .build/release/gemma-embed --dir /path/to/images\n```\n\n`--profile-images` times each vision kernel class over the first `--batch-size` images, and\n`--bench-images` reports `gpu_seconds_minsum`, every batch's fastest GPU pass summed.\n`scripts/ab_images.py` runs two builds alternately and checks image parity, text parity and that\nthe candidate's vectors match the baseline's.\n\nLoad the model with the audio encoder, which lives in the same mmproj file as the image encoder, and embed files, bytes or samples. The vectors live in the same space as the text and image embeddings, so a spoken sentence can be scored against its transcript, and a sound against a caption.\n\n``` js\nlet model = try await EmbeddingModel.load(from: textURL, options: LoadOptions(audioEncoder: mmprojURL))\nlet clips = try await model.embed(audios: [.url(wavURL), .data(mp3Bytes), .samples(pcm, sampleRate: 44100, channels: 2)])\nlet transcript = try await model.embed(\"the train now departs from platform four\", options: EmbedOptions(task: .query))\nlet score = model.similarity(transcript, clips[0])\nprint(model.info.audioEncoder?.layerCount)   // 12 when loaded, nil otherwise\n```\n\n`AudioEmbedOptions` carries the same `dimensions` and `normalise` as text, `batchSize` (clips per\nGPU batch, by default packed by mel frames, two minutes of audio per batch) and `maxTokens`. A\nclip costs one row per 40 ms plus four frame tokens; `maxTokens` defaults to the model's 8192\nrows, which is 5 minutes 27 seconds, and a longer clip is refused with `invalidOptions` rather\nthan cut, because llama.cpp does not cut audio either. Pass the mmproj file as `imageEncoder`,\n`audioEncoder` or both; the file is mapped once and only the towers asked for are built. Embedding\naudio without the encoder throws `audioEncoderNotLoaded`.\n\nThe encoder sees llama.cpp's front end exactly: miniaudio's decode and resample to 16 kHz mono\n(WAV bit for bit, including resampled and multichannel files; other codecs decode through\nAVFoundation, so an MP3 sits within 3e-5 per sample of llama.cpp's), 128 log-mel bins every 10 ms\nover a 20 ms window, and a split into 30 second chunks that the tower encodes one by one. A\n`MultimodalInput` takes `.audio` parts beside `.text` and `.image` parts, framed with `<|audio>`\nand `<audio|>` where they sit. The front end and the tower are documented in\n[Sources/GemmaEmbed/Model/Architecture.md](https://github.com/maralski/gemma-embed-swift/blob/main/Sources/GemmaEmbed/Model/Architecture.md).\n\nReference vectors come from llama.cpp build 11716 (`llama-server --embedding --mmproj -ub 8192`)\nover the same Q8_0 files, and per-node dumps of its audio graph taken through `libmtmd`'s eval\ncallback (`scripts/audio_dump.cpp`). Every fixture clip is synthesised by\n`scripts/make_fixture_audio.py`: tones, a chord, bursts, chirps, noise, clicks, AM and FM signals,\nsilence, 0.3 to 7 second signals, the same signal at 8, 22.05, 44.1 and 48 kHz, in stereo and four\nchannels, and as 8, 24 and 32-bit integer and float32 WAV, plus a 30.0 s clip and a 35 s clip\nthat crosses the chunk boundary. No recordings and no voices are in the repository.\n`scripts/oracle_audio.py` writes the reference vectors from a running llama-server.\n\n| Check | Result | \n|---|---|\n| WAV decode and miniaudio's downmix and resample, 32 variants (bit depths, 3 to 6 channels, 8 to 96 kHz), plus FLAC, AIFF and CAF of one of them | bit-identical to miniaudio's samples | \n| Log-mel front end vs llama.cpp's graph input, 8 chunks of 7 clips | bins above -2 within 3e-5; bins between -6 and -2 within 1.2e-3; bins at the 0.001 floor within 2.4e-3 | \n| CPU reference audio tower vs llama.cpp per-node sums, 8 chunks, 2448 nodes | within 2.7e-4 of the absolute sum | \n| CPU reference vs llama.cpp, 589 full tensors | cosine 0.999995 or better | \n| Metal audio tower vs CPU reference, 7 clips including the two-chunk 35 s clip | row cosine 0.999999 or better, max abs diff 3.6e-3 on rows of norm 28 | \n| Metal audio embeddings vs llama-server, all 29 fixture clips | min cosine 0.99998933 (the 35 s clip), max abs diff 7.2e-4 | \n| A generated 5-minute clip (11 chunks, 7504 rows, not in the repository) vs llama-server | cosine 0.9999937 | \n| Same clip alone and inside a batch of 5 mixed lengths, first or last | bit-identical | \n| Text and image parity | unchanged (see Image parity and Parity) | \n\nRun `swift test -c release` to reproduce (see Tests), or\n`gemma-embed --parity-audio Tests/GemmaEmbedTests/Fixtures/audio`.\n\nMeasured on 2026-10-26 on the same Apple M3 Max as the text numbers, over 40 spoken clips made by\nmacOS `say` (`scripts/make_speech_bench.py`, not in the repository: 715 s in all, 3.3 to 85 s\neach, median 8.7 s), under a GPU lock, at a load average of about 4 to 6. Wall time includes WAV\ndecoding and the log-mel front end on the CPU.\n\n| Engine | Audio seconds per second, median of 5 rounds | Range | \n|---|---|---|\n| GemmaEmbed, default batching by mel frames | 274.9 | 271.0 to 324.7 | \n| GemmaEmbed, one clip per GPU batch | 235.9 | 225.9 to 271.9 | \n| llama.cpp build 11716, `llama-server --embedding --mmproj` , one clip per request | 153.6 | 148.6 to 160.2 | \n\n`scripts/compare_llamacpp_audio.py models/audio/bench 5` reproduces the table; llama-server's side\nincludes HTTP and base64, so the gap slightly overstates the difference between the engines. Over\nthe same clips the two engines' vectors agree to min cosine 0.9999994.\n\nAgainst the first correct implementation, the speed work made audio embedding 1.34 times faster\nin wall time with default batching and 1.29 times with one clip per batch (interleaved A/B runs\nwith `scripts/ab_audio.py`, interquartile range 1.30 to 1.35), with the same vectors to within\n3.4e-7. Most of it came from the CPU: the log-mel front end runs an in-place FFT and sums only the\nnon-zero taps of each mel filter (about 1.17 times wall on its own, bit-identical features), and\nthe next batch's clips are decoded while the GPU runs the current one. On the GPU, attention runs\none simdgroup per query and head (12 times faster per layer on long clips) and the second\nsubsampling convolution is an implicit GEMM on simdgroup matrices (about 4 percent of GPU time).\nWall time is now bound by the GPU, and three quarters of that is GEMMs already running near 10\nTFLOPS. Tuning GEMM tiles for the audio shapes, split-K, fusing the elementwise passes, larger or\nsmaller batches, and double-buffered command encoding each measured within noise.\n\nTen sentences spoken by macOS `say` (`scripts/make_speech_samples.sh`, not in the repository) are\nscored against their ten transcripts and five off-topic texts, all embedded as queries. Every\nclip ranks its own transcript first, by 0.17 or more over the next text, and llama-server's own\nvectors give the same ranks; GemmaEmbed's similarities agree with llama-server's within 1e-4. Own\ncosines run from 0.80 to 0.90, and the smallest margin is a recipe step whose nearest other text\nis the other cooking sentence. The test skips when the speech folder is absent.\n\n```\nswift build -c release\nexport GEMMA_EMBED_WEIGHTS=/path/to/embeddinggemma-2-Q8_0.gguf\necho \"hello world\" | .build/release/gemma-embed --task query --dimensions 256\n.build/release/gemma-embed --parity Tests/GemmaEmbedTests/Fixtures\nexport GEMMA_EMBED_IMAGE_WEIGHTS=/path/to/mmproj-embeddinggemma-2-Q8_0.gguf\n.build/release/gemma-embed --images photo.jpg ~/Pictures\n.build/release/gemma-embed --parity-images Tests/GemmaEmbedTests/Fixtures/images\n.build/release/gemma-embed --bench-images ~/Pictures --runs 3\n.build/release/gemma-embed --bench-images ~/Pictures --batch-size auto\n.build/release/gemma-embed --profile-images ~/Pictures --batch-size 1\n.build/release/gemma-embed --bench posts.json --runs 3\n.build/release/gemma-embed --profile posts.json\n.build/release/gemma-embed --audio talk.wav ~/Recordings\n.build/release/gemma-embed --parity-audio Tests/GemmaEmbedTests/Fixtures/audio\n.build/release/gemma-embed --bench-audio ~/Recordings --runs 3\n.build/release/gemma-embed --profile-audio ~/Recordings --batch-size 1\n```\n\n`gemma-embed` prints one JSON object per input line. `--parity` checks the tokenizer and the\nembeddings against the llama.cpp fixtures and exits non-zero below cosine 0.9995. `--bench` times\na throughput run over a feed JSON file (`{\"posts\": [{\"gist\", \"text\"}]}`) or a plain text file with\none document per line. Image modes take the mmproj file from `--image-encoder` or\n`$GEMMA_EMBED_IMAGE_WEIGHTS`: `--images FILE... | DIR` prints one JSON vector per image,\n`--parity-images DIR` checks a fixtures folder (`manifest.json` and `embeddings.jsonl`) and\n`--bench-images DIR` measures images per second, by default with patch-count batching, 8 and 1\nimages per batch (`--batch-size auto` or `N` picks one). Audio modes take the same mmproj file\nfrom `--audio-encoder`, `$GEMMA_EMBED_AUDIO_WEIGHTS` or the image encoder path: `--audio FILE... | DIR` prints one JSON vector per clip with its row count and seconds, `--parity-audio DIR` checks a\nfixtures folder, `--bench-audio DIR` measures audio seconds per wall second and per GPU second,\nand `--profile-audio DIR` times each audio kernel class. `--help` lists the rest.\n\nThe shaders compile from `.metal` source when a model loads. GemmaEmbed looks for the SwiftPM\nresource bundle `GemmaEmbed_GemmaEmbed.bundle` in this order:\n\n1. The main bundle's resources: `Contents/Resources` inside an app.\n2. The resources of the bundle that holds the GemmaEmbed code, then the directory next to it.\n3. The main bundle's own directory, which is where `swift run` and a bare executable find it.\n\nIf no copy is found, GemmaEmbed uses the shader source compiled into the library, so a model\nstill loads with no bundle at all. It never reads the checkout's `.build` directory, which\nSwiftPM's `Bundle.module` falls back to.\n\nAn app built outside Xcode must copy the bundle into `Contents/Resources`. Codesign refuses\nloose files at the root of an `.app`. A build script can do it with this line, after it\nsets `SHADERS=\"$BIN_DIR/GemmaEmbed_GemmaEmbed.bundle\"` from `swift build -c release --show-bin-path`:\n\n```\ncp -R \"$SHADERS\" \"$APP/Contents/Resources/\"\n```\n\nAfter you edit a file in `Sources/GemmaEmbed/Shaders`, run `scripts/embed_shaders.py` to\nregenerate `Sources/GemmaEmbed/Metal/EmbeddedShaders.swift`. `ShaderSourceTests` fails while\nthe two disagree.\n\n`swift test -c release` runs the suite. The debug configuration also passes, in about six\nminutes, because the CPU reference runs unoptimised. The tests read the weights and the\nllama.cpp dumps from `models/` at the package root, which is gitignored. An environment variable\noverrides each path, and a test whose input is missing skips.\n\n| Path | Contents | Override | \n|---|---|---|\n| `models/embeddinggemma-2-Q8_0.gguf` | the text weights | `GEMMA_EMBED_WEIGHTS` | \n| `models/mmproj-embeddinggemma-2-Q8_0.gguf` | the image encoder | `GEMMA_EMBED_IMAGE_WEIGHTS` | \n| `models/dumps/` | llama.cpp's vision graph for six images ( `scripts/vision_dump.cpp` ) | `GEMMA_EMBED_VISION_DUMPS` | \n| `models/dumps-rgb/` | stb_image's decode of every JPEG fixture ( `scripts/vision_dump.cpp` ) | `GEMMA_EMBED_DECODED_RGB_DUMPS` | \n| `models/fixtures/images/` | the full image fixture set from `scripts/make_fixture_images.swift` , including the lavender field scene and its variant, which are over 250 KB and not in the package | none | \n| `models/audio/dumps/` | llama.cpp's audio graph for seven clips ( `scripts/audio_dump.cpp` ) and miniaudio's decode of the decode-check WAVs (`scripts/make_decode_checks.py` ) | `GEMMA_EMBED_AUDIO_DUMPS` | \n| `models/audio/speech/` | ten spoken sentences from `scripts/make_speech_samples.sh` with llama-server's vectors from`scripts/oracle_audio.py --speech` | none | \n\n`scripts/vision_dump.cpp` and `scripts/audio_dump.cpp` list the commands that write the dump directories.\n\nReference vectors come from llama.cpp build 11716 (`llama-server --embedding`) over the same Q8_0\nfile, which is itself an approximation of the fp32 model, so agreement is bounded by that oracle.\n\n| Check | Result | \n|---|---|\n| Tokenizer, 40 fixture texts (five scripts, URLs, numbers, emoji) | token ids identical | \n| CPU fp32 reference vs llama.cpp per-tensor activation sums, 3 texts | every named tensor within 2 percent | \n| CPU fp32 reference vs llama.cpp embeddings, 3 texts | min cosine 0.9999992, max abs diff 8.5e-5 | \n| Metal kernels vs CPU reference (GEMM all shapes and tiles, RMSNorm, RoPE, attention at 1 to 2048 tokens, GeGLU, pooling) | max relative diff 4.5e-6 | \n| Metal forward vs llama.cpp, all 40 fixtures | min cosine 0.9999988, max abs diff 1.4e-4 | \n| Metal forward vs llama.cpp at 1054, 1629 and 3222 tokens (sliding window active) | cosine 0.9999995 or better | \n| Metal forward vs CPU reference at 1672 tokens | cosine 0.9999999, max abs diff 1.8e-7 | \n| Same text alone and inside a batch of 33 | bit-identical | \n\nRun `swift test -c release` to reproduce (see Tests). `scripts/make_text_fixtures.py`\nregenerates the text fixtures from llama.cpp.\n\nMeasured on 2026-10-25 on an Apple M3 Max (40 GPU cores, 128 GB) running macOS 26, Xcode 26.3, under a GPU lock with no other GPU work and a load average of 3.3, five runs. The corpus is 285 social media posts of 2026-10-17, a private set that is not in the repository (gist plus text cut at 2000 characters, document prefix), 41,675 tokens in all, 146 tokens per post on average, longest 604.\n\n| Engine | Posts per second | Seconds for 285 posts | Tokens per second | \n|---|---|---|---|\n| GemmaEmbed 0.3.0 | 176.4 | 1.62 (1.61 to 1.66) | 25,790 | \n| llama.cpp 0.7.0-dev build 11716, `llama-server --embedding -np 8 -ub 8192` , 32 texts per request | 118.3 | 2.41 | 17,300 | \n| llama.cpp, same build, `-np 1` (one sequence at a time) | 59.6 | 4.78 | 8,720 | \n\nGemmaEmbed 0.3.0 is 1.49 times faster than llama.cpp's best configuration and 2.96 times faster than its single-slot default, with the same parity (min cosine 0.9999988 against llama.cpp over the 40 fixtures). A single 22-token query takes 7.4 ms wall. The GEMMs take most of the GPU time and run at 7.6 to 8.8 TFLOPS fp32 in a sustained microbench; attention takes about an eighth.\n\nMeasuring on a shared machine: GPU time of short command buffers reads high because the GPU runs\nat a low clock, so per-kernel lines of `--profile` mislead for kernels under about 2 ms, and a\nmicrobench of short bursts can rank GEMM tiles the wrong way round. `scripts/ab_bench.py` runs two\nbuilds alternately and calls a difference only when the median, best-of and per-batch-minimum\nratios agree; `scripts/gpulock.py` keeps two measurements off the GPU at once.\n\nLoad time: 0.27 s when the Metal shader cache is warm (tokenizer 0.17 s), about 1 s on the first load after a build (shader compilation 0.25 s, pipelines 0.3 s). Autotune, when enabled, adds about 4 s on the first load and nothing afterwards.\n\nReproduce:\n\n```\nswift build -c release\nexport GEMMA_EMBED_WEIGHTS=/path/to/embeddinggemma-2-Q8_0.gguf\n.build/release/gemma-embed --bench /path/to/posts.json --runs 3\n\nllama-server -m /path/to/embeddinggemma-2-Q8_0.gguf --embedding -c 8192 -b 8192 -ub 8192 -np 8 --port 8092 &\npython3 scripts/bench_llamacpp.py /path/to/posts.json --port 8092 --batch 32 --runs 3\npython3 scripts/ab_bench.py --a /path/to/old/gemma-embed --b .build/release/gemma-embed --bench /path/to/posts.json\n```\n\nThe kernels follow these design rules:\n\n- The weight file is mapped `MAP_SHARED` and read-only, pre-touched from several threads, and\nwrapped in one`MTLBuffer` with no copy. Tensors are offsets into that buffer.\n- Shaders ship as `.metal` sources and compile when the model loads (0.2 s), because SwiftPM\ndoes not produce a metallib from the command line. Every pipeline is built at load. See\nPackaging for where the sources are read from.\n- All arithmetic is fp32. Q8_0 and BF16 are storage formats widened inside the kernels.\n- GEMM uses `simdgroup_float8x8` with X and dequantized W tiles staged in threadgroup memory,\nand stages the output tile so rows leave as coalesced`float4` stores. The tile is picked by\noutput size (smaller tiles for small batches, a wider one for inner dimensions above 2048), and\nevery tile accumulates over K in the same order, so a row's result is bit-identical whatever\nbatch it travelled in. Every loop over fragment arrays is fully unrolled: a rolled loop keeps\nthe arrays in memory and runs several times slower.\n- Attention is the new work: flash-style, bidirectional, over sequences packed back to back\nwithout padding. A threadgroup owns 8 to 64 query rows of one sequence for one head and\nstreams 32-key blocks staged once in threadgroup memory; the output accumulators stay in\nregisters and the online softmax runs in fp32 on the score fragments themselves. Small batches\nsplit each head into narrower slices to fill the GPU, and the vision tower has its own kernel\nfor its 64-wide heads. Sliding-window layers mask `|i - j| > 512` , the symmetric half window\nllama.cpp uses.\n- The forward runs on a concurrent compute encoder with a buffer barrier after every dispatch, except within small groups of independent dispatches (q, k and v, for instance), which may overlap. A stress test runs two batches alternately on it and checks every result against a serial encoder bit for bit.\n- Padding is avoided entirely: texts are concatenated, every per-token kernel runs on the concatenation, and attention takes a work list of (sequence, query block) pairs.\n- Autotune, off by default, times whole forwards on real text for the GEMM tile and the attention query group, requires a 5 percent win, confirms the winner against the default and caches the result per device, kernel digest and model hash.\n\n- iOS and visionOS targets. With `.iOS(.v18)` added to the platforms, the library compiles for iOS\nunchanged (iOS 17 lacks`MTLCompileOptions.mathMode` ); it has not yet run on a device or\nsimulator, and visionOS has not been built.\n- F16 weight files, which are refused at load today, and a check of BF16 files against llama.cpp (the GEMM kernels read BF16 weights, but no BF16 file has been tested).\n- Fusing the text tower's q, k and v projections into one grouped GEMM, as the vision and audio towers already do, and the per-layer-input gate into fewer dispatches.\n\nPeter Marelas ([@maralski](https://github.com/maralski)).\n\nCopyright 2026 Peter Marelas. Apache License 2.0, the license Google releases EmbeddingGemma 2 and the Gemma 4 models under.\nSee [LICENSE](https://github.com/maralski/gemma-embed-swift/blob/main/LICENSE) and [NOTICE](https://github.com/maralski/gemma-embed-swift/blob/main/NOTICE). The model weights are not included and are\nlicensed separately by Google; Google asks that deployments follow the\n[Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy). Code ported\nfrom llama.cpp and stb_image keeps its MIT notices in [THIRD_PARTY_NOTICES](https://github.com/maralski/gemma-embed-swift/blob/main/THIRD_PARTY_NOTICES).\n\nGemma is a trademark of Google LLC. This project is not affiliated with or endorsed by Google.", "url": "https://wpnews.pro/news/gemmaembed-swift-package-optimized-for-apple-silicon", "canonical_source": "https://github.com/maralski/gemma-embed-swift", "published_at": "2026-10-27 03:07:43+00:00", "updated_at": "2026-10-27 03:41:43.781279+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-infrastructure"], "entities": ["GemmaEmbed", "EmbeddingGemma 2", "Google", "Apple Silicon", "Swift", "Metal", "llama.cpp", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/gemmaembed-swift-package-optimized-for-apple-silicon", "markdown": "https://wpnews.pro/news/gemmaembed-swift-package-optimized-for-apple-silicon.md", "text": "https://wpnews.pro/news/gemmaembed-swift-package-optimized-for-apple-silicon.txt", "jsonld": "https://wpnews.pro/news/gemmaembed-swift-package-optimized-for-apple-silicon.jsonld"}}