GemmaEmbed runs Google's EmbeddingGemma 2 text, image and audio encoders in-process on Apple Silicon with its own Metal kernels. It is a Swift package with no dependencies beyond the Apple SDK: no llama.cpp, no MLX, no Core ML. You hand it the official GGUF weight files and get 768-dimensional embeddings that match llama.cpp's to a cosine similarity of 0.99999 for text, 0.99999 for images and 0.99998 for audio, in one shared space.
| Model | embeddinggemma-2 (GGUF architecturegemma-embedding2 ), 24 layers, 271M parameters |
| Weight file | embeddinggemma-2-Q8_0.gguf fromhttps://huggingface.co/ggml-org/embeddinggemma-2-GGUF |
| Size | 309,855,456 bytes |
| SHA256 | 2188ac1deca4b77dffefd603c2776a9d76d9d74ec01841392982ebb840b09135 |
| Output | 768 dimensions, Matryoshka truncation to 512, 256 or 128 |
| Input | up to 8192 tokens per text (default limit 2048), any number of texts per call |
| Image and audio encoders (optional) | mmproj-embeddinggemma-2-Q8_0.gguf from the same repository, 554,821,024 bytes, SHA256c4a8a52691ecef40618438928bdf9e68379b854e24166f292592353db0aab64f (GGUF projectorsgemma4v , 16 layers, andgemma4a , 12 Conformer layers; 473M parameters in all) |
| Image input | JPEG, PNG, HEIC and anything else ImageIO reads, or a CGImage ; any size, scaled to 70 to 1120 vision tokens (default cap 256) |
| Audio input | WAV (parsed by GemmaEmbed), MP3, M4A, AIFF, CAF, FLAC (through AVFoundation) or PCM samples; any rate and channel count, resampled to 16 kHz mono; 25 rows per second, up to 5 minutes 27 seconds per clip |
| Platform | macOS 14 or later on Apple Silicon, Swift 6 |
The forward pass is documented in Sources/GemmaEmbed/Model/Architecture.md.
Add the package to Package.swift:
dependencies: [
.package(url: "https://github.com/maralski/gemma-embed-swift.git", from: "0.4.0"),
],
targets: [
.target(name: "MyApp", dependencies: [.product(name: "GemmaEmbed", package: "gemma-embed-swift")]),
]
In Xcode, choose File > Add Package Dependencies and enter the same URL. The weights are not
part of the package. Download embeddinggemma-2-Q8_0.gguf (and, for images and audio,
mmproj-embeddinggemma-2-Q8_0.gguf) from the Hugging Face repository above.
For a side-by-side comparison with llama.cpp (accuracy, speed, features, when to use which), see COMPARISON.md.
import GemmaEmbed
let model = try await EmbeddingModel.load(from: URL(fileURLWithPath: "embeddinggemma-2-Q8_0.gguf"))
let documents = try await model.embed(["Rates were left unchanged.", "A new open-weight model shipped."])
let query = try await model.embed("central bank decision", options: EmbedOptions(task: .query))
let best = documents.indices.max { model.similarity(query, documents[$0]) < model.similarity(query, documents[$1]) }
EmbedOptions selects the model card's task prefix (.document, .query, .clustering,
.classification, .similarity, .none), the token limit, the output width and whether the
vector is L2-normalised. embed batches texts by token count, reports progress after every GPU
batch and stops between batches when the surrounding Task is cancelled. Every text is embedded
identically whether it travels alone or inside a batch.
Load the model with the image encoder and embed files, bytes or CGImage s. The vectors live in
the same space as the text embeddings, so a caption query can be scored against pictures and a
picture against pictures.
let model = try await EmbeddingModel.load(from: textURL, options: LoadOptions(imageEncoder: mmprojURL))
let pictures = try await model.embed(images: [.url(photoURL), .data(jpegBytes), .cgImage(image)])
let caption = try await model.embed("rolling green vineyard hills", options: EmbedOptions(task: .query))
let score = model.similarity(caption, pictures[0])
print(model.info.imageEncoder?.layerCount) // 16 when loaded, nil otherwise
ImageEmbedOptions carries the same dimensions and normalise as text, batchSize (images
per GPU batch, by default packed by patch count) and maxImageTokens. The image encoder is a
LoadOptions field rather than something added to a loaded model, so a model never changes
after load; text alone never opens the mmproj file, and embed(images:) without it
throws imageEncoderNotLoaded.
The encoder sees each image resized so that its patch grid gives at most maxImageTokens
vision tokens (16 by 16 pixel patches, pooled 3 by 3), and at least 70. The default cap of 256 is
what llama-server applies with its default batch size; llama.cpp's native limit is 1120. The
same photo embeds differently at different caps (cosine 0.973 between 256 and 1120 on the 1600
by 1600 green hills scene), so a corpus and the queries against it must share one cap. JPEGs
decode through a Swift port of stb_image's decoder, byte for byte what llama.cpp sees; PNGs
match it too. The preprocessing and the vision tower are documented in
Sources/GemmaEmbed/Model/Architecture.md.
One input can mix text, images and audio. The model reads each picture and each clip where it sits among the text, in one forward pass, so a post with its photo and voice note embeds as one document.
let mmproj = URL(fileURLWithPath: "mmproj-embeddinggemma-2-Q8_0.gguf")
let model = try await EmbeddingModel.load(from: textURL, options: LoadOptions(imageEncoder: mmproj, audioEncoder: mmproj))
let post = MultimodalInput([.text("Our walk on Saturday:"), .image(.url(photoURL)), .text("and what the guide said:"), .audio(.url(voiceNoteURL))])
let posts = try await model.embed([post, MultimodalInput(text: "Quarterly results", images: [.data(chartPNG)])])
let query = try await model.embed("guided hill walk", options: EmbedOptions(task: .query))
let score = model.similarity(query, posts[0])
The sequence is the one llama.cpp's mtmd builds for EmbeddingGemma 2:
[bos] text [<|image>] image rows [<image|>] text [<|audio>] audio rows [<audio|>] ... [eos],
with the parts in the order given. MultimodalEmbedOptions.task prefixes the first text, or
stands alone before a picture or clip that opens the input. maxTokens counts every row; text is
cut from the end until the input fits and pictures and clips are never cut, so an input whose
pictures and clips alone exceed it throws invalidOptions. A text-only input embeds exactly as
embed(_:) does, and an image-only or audio-only input with task: .none exactly as
embed(images:) or embed(audios:) does. Inputs with images need the image encoder and inputs
with audio the audio encoder (one mmproj file holds both); text-only inputs need neither. Inputs
of all three kinds batch together, and each embeds to the same bits alone or in a batch.
The example's shape, a sentence, a fixture photo, a second sentence and a 3 s spoken clip (359
rows), embeds to cosine 0.9999997 against llama-server --embedding --mmproj given the same
parts, max abs diff 9.3e-5, on 2026-10-26.
Reference vectors come from llama.cpp build 11716 (llama-server --embedding --mmproj) over the
same Q8_0 files, and per-node dumps of its vision graph taken through libmtmd's eval callback
(scripts/vision_dump.cpp). Every fixture image is drawn from code by
scripts/make_fixture_images.swift: 10 procedural scenes (green hills, an ocean sunset, a coral
reef, snowy peaks, desert dunes, cherry blossom, a lavender field, a night sky, a sunflower field,
a city skyline), 10 synthetic CoreGraphics images (solid colours, gradients, text in three fonts,
shapes, a checkerboard, noise) and 10 near-duplicate variants of the scenes.
scripts/oracle_images.py writes the reference vectors from a running llama-server.
| Check | Result |
|---|---|
| mmproj GGUF, 963 tensors and 30 metadata keys | identical to the directory dump |
| Preprocessing on 6 images (decode, size choice, Pillow-style bicubic resize, padding, normalisation, patch layout) | byte-identical to llama.cpp's pixel tensor |
| JPEG decoder on 36 JPEGs (every sampling factor, progressive, grey, YCCK, restart markers) | byte-identical to stb_image |
| CPU reference vision tower vs llama.cpp per-node sums, 3 images, 292 nodes each | within 1e-3, worst 8.5e-5 |
| CPU reference vision vectors vs llama.cpp (512-wide rows) | row cosine 0.99996 or better |
| Metal kernels vs CPU reference (patch GEMM, position tables, 2D RoPE, head-64 attention, gelu_quick epilogue, 3x3 pooling, mixed-row embedding) | max relative diff 5.0e-6 |
| Metal vision tower vs CPU reference, 3 images | row cosine 0.9999993 or better, max abs diff 1.2e-4 |
| Metal image embeddings vs llama-server, all 30 images, cap 256 | min cosine 0.9999985, max abs diff 2.8e-4 |
| Metal image embeddings vs llama-server, 4 images at cap 1120 | min cosine 0.9999992, max abs diff 1.4e-4 |
| Same image alone and inside a batch of 8 mixed sizes, first or last | bit-identical |
| Text embeddings vs llama.cpp, 40 fixtures (see Parity) | min cosine 0.9999988 |
Run swift test -c release to reproduce (see Tests for where the tests find the image encoder,
the llama.cpp dumps and the two images too large to ship in the package), or
gemma-embed --parity-images Tests/GemmaEmbedTests/Fixtures/images.
Five scenes scored against ten caption queries (EmbedOptions(task: .query), cosine). Five
captions describe the scenes and five are off-topic. Every scene ranks its own caption first, by
0.125 or more over the next caption, and llama-server's own vectors give the same ranks. The
scores below come from llama-server's vectors in Fixtures/images/captions.jsonl and
embeddings.jsonl; the test requires GemmaEmbed's to agree within 0.002.
| caption | 00-green-hills | 01-ocean-sunset | 02-coral-reef | 05-cherry-blossom | 08-sunflower-field |
|---|---|---|---|---|---|
| rolling green hills striped with crop rows, a red barn and trees under a blue sky with white clouds | 0.795 | 0.526 | 0.560 | 0.536 | 0.658 |
| the sun setting over the ocean with an orange sky and a palm tree silhouette | 0.488 | 0.750 | 0.602 | 0.521 | 0.560 |
| an underwater coral reef with colourful corals and tropical fish | 0.481 | 0.570 | 0.752 | 0.538 | 0.528 |
| pink cherry blossoms on dark branches against a pale blue sky | 0.525 | 0.567 | 0.574 | 0.787 | 0.562 |
| a field of bright yellow sunflowers under a blue summer sky | 0.601 | 0.575 | 0.595 | 0.578 | 0.784 |
| the central bank left interest rates unchanged for a third consecutive meeting | 0.528 | 0.502 | 0.506 | 0.517 | 0.528 |
| a recipe for sourdough bread with a long cold fermentation | 0.487 | 0.468 | 0.440 | 0.463 | 0.480 |
| quarterly earnings call transcript for a semiconductor company | 0.478 | 0.427 | 0.443 | 0.467 | 0.474 |
| how to configure a Kubernetes ingress controller with TLS | 0.448 | 0.450 | 0.436 | 0.431 | 0.455 |
| a black and white photograph of a crowded railway station | 0.452 | 0.446 | 0.452 | 0.467 | 0.490 |
The smallest margin is the sunflower field, whose own caption (0.784) leads the green hills caption (0.658). Off-topic captions score 0.43 to 0.53.
Cosine between each scene and its variant, for choosing a deduplication threshold. Every variant
has its original as nearest neighbour. The lowest variant pair is 0.896 (the coral reef
re-encoded at JPEG quality 40), and the highest cosine between two unrelated images is 0.889
(synthetic/01-solid-grey against synthetic/02-gradient-h). A threshold between the two
separates them on this set, with a gap of 0.007. Among the scenes and their variants alone, the
highest unrelated cosine is 0.822 (the green hills and the sunflower field).
| variant | pairs | min | median | max |
|---|---|---|---|---|
| resized to half | 1 | 0.996 | 0.996 | 0.996 |
| resized to a third | 1 | 0.987 | 0.987 | 0.987 |
| central crop keeping 90 percent | 1 | 0.998 | 0.998 | 0.998 |
| top-left crop keeping 90 percent | 1 | 0.994 | 0.994 | 0.994 |
| JPEG quality 40 | 2 | 0.896 | 0.943 | 0.989 |
| flipped horizontally | 2 | 0.991 | 0.993 | 0.996 |
| colour shift (red +12, blue -12) | 2 | 0.994 | 0.995 | 0.995 |
Measured on 2026-10-25 on the same Apple M3 Max as the text numbers, over 153 social media feed
pictures of 2026-10-23 (a private corpus, not in the repository; real JPEGs, 480 to 1280 pixels
wide, 188 vision tokens on average), one picture per GPU batch as the default batching runs
them, with gemma-embed --bench-images --batch-size 1 under a GPU lock, three rounds:
| Configuration | Images per second (median of 3) | Seconds for 153 images | GPU seconds |
|---|---|---|---|
| GemmaEmbed 0.3.0 | 10.40 | 14.72 | 14.48 |
Image parity with llama-server is in Image parity above (min cosine 0.9999985 over the 30 fixtures). A feed picture takes about 95 ms of GPU time, almost all of it the vision tower; the text tower over its rows takes about 3 ms.
Against llama.cpp, on the same 153 pictures at the same cap of 256, twelve alternating rounds
on a loaded machine (CPU load moved every figure by a factor of two or more, so treat the ratio
as approximate). Wall time includes image decoding on the CPU for both engines.
scripts/compare_llamacpp.py reproduces it; COMPARISON.md has the details.
| Configuration | Images per second, median of 12 rounds | Range |
|---|---|---|
| GemmaEmbed 0.3.0, one image per GPU batch | 8.46 | 4.48 to 11.04 |
GemmaEmbed 0.3.0, batched by patch count ( --batch-size auto ) |
7.84 | 3.00 to 11.64 |
llama.cpp 0.7.0-dev build 11716, llama-server --embedding --mmproj , 1 image per request |
5.91 | 2.87 to 7.91 |
| llama.cpp, same build, 8 images per request | 6.04 | 3.03 to 8.29 |
The GPU time itself grows with the batch: packing more images into one forward makes the vision
GEMMs slower, not faster, from 1 to 8 images per batch. The likely cause is the
activation buffers outgrowing the on-chip cache (a batch of 8 full-size images is 18,432 patch
rows, and its 3072-wide FFN activations take 226 MB): profiled, a batch of 8 pictures
runs 17 percent slower than the sum of its kernels timed alone, while one picture runs faster
than its kernel sum. Budgets of 8192 to 32768 patches measured 4 to 6 percent slower than 4096
and cost 2 to 4 times the memory. The
default therefore packs images up to 4096 patch rows per batch, which is one image at the
default cap and several smaller ones; ImageEmbedOptions.batchSize overrides it.
Image encoder load time: 0.24 s on the first load after the file left the page cache (mapping and prefaulting the 555 MB file, building the tower), 0.02 s on later loads. the text model alone does not open the file.
Reproduce:
export GEMMA_EMBED_WEIGHTS=/path/to/embeddinggemma-2-Q8_0.gguf
export GEMMA_EMBED_IMAGE_WEIGHTS=/path/to/mmproj-embeddinggemma-2-Q8_0.gguf
GEMMA_EMBED_TRACE=1 .build/release/gemma-embed --bench-images /path/to/images --runs 5
.build/release/gemma-embed --profile-images /path/to/images --batch-size 1
python3 scripts/ab_images.py --a /path/to/old/gemma-embed --b .build/release/gemma-embed --dir /path/to/images
--profile-images times each vision kernel class over the first --batch-size images, and
--bench-images reports gpu_seconds_minsum, every batch's fastest GPU pass summed.
scripts/ab_images.py runs two builds alternately and checks image parity, text parity and that
the candidate's vectors match the baseline's.
Load the model with the audio encoder, which lives in the same mmproj file as the image encoder, and embed files, bytes or samples. The vectors live in the same space as the text and image embeddings, so a spoken sentence can be scored against its transcript, and a sound against a caption.
let model = try await EmbeddingModel.load(from: textURL, options: LoadOptions(audioEncoder: mmprojURL))
let clips = try await model.embed(audios: [.url(wavURL), .data(mp3Bytes), .samples(pcm, sampleRate: 44100, channels: 2)])
let transcript = try await model.embed("the train now departs from platform four", options: EmbedOptions(task: .query))
let score = model.similarity(transcript, clips[0])
print(model.info.audioEncoder?.layerCount) // 12 when loaded, nil otherwise
AudioEmbedOptions carries the same dimensions and normalise as text, batchSize (clips per
GPU batch, by default packed by mel frames, two minutes of audio per batch) and maxTokens. A
clip costs one row per 40 ms plus four frame tokens; maxTokens defaults to the model's 8192
rows, which is 5 minutes 27 seconds, and a longer clip is refused with invalidOptions rather
than cut, because llama.cpp does not cut audio either. Pass the mmproj file as imageEncoder,
audioEncoder or both; the file is mapped once and only the towers asked for are built. Embedding
audio without the encoder throws audioEncoderNotLoaded.
The encoder sees llama.cpp's front end exactly: miniaudio's decode and resample to 16 kHz mono
(WAV bit for bit, including resampled and multichannel files; other codecs decode through
AVFoundation, so an MP3 sits within 3e-5 per sample of llama.cpp's), 128 log-mel bins every 10 ms
over a 20 ms window, and a split into 30 second chunks that the tower encodes one by one. A
MultimodalInput takes .audio parts beside .text and .image parts, framed with <|audio>
and <audio|> where they sit. The front end and the tower are documented in
Sources/GemmaEmbed/Model/Architecture.md.
Reference vectors come from llama.cpp build 11716 (llama-server --embedding --mmproj -ub 8192)
over the same Q8_0 files, and per-node dumps of its audio graph taken through libmtmd's eval
callback (scripts/audio_dump.cpp). Every fixture clip is synthesised by
scripts/make_fixture_audio.py: tones, a chord, bursts, chirps, noise, clicks, AM and FM signals,
silence, 0.3 to 7 second signals, the same signal at 8, 22.05, 44.1 and 48 kHz, in stereo and four
channels, and as 8, 24 and 32-bit integer and float32 WAV, plus a 30.0 s clip and a 35 s clip
that crosses the chunk boundary. No recordings and no voices are in the repository.
scripts/oracle_audio.py writes the reference vectors from a running llama-server.
| Check | Result |
|---|---|
| WAV decode and miniaudio's downmix and resample, 32 variants (bit depths, 3 to 6 channels, 8 to 96 kHz), plus FLAC, AIFF and CAF of one of them | bit-identical to miniaudio's samples |
| Log-mel front end vs llama.cpp's graph input, 8 chunks of 7 clips | bins above -2 within 3e-5; bins between -6 and -2 within 1.2e-3; bins at the 0.001 floor within 2.4e-3 |
| CPU reference audio tower vs llama.cpp per-node sums, 8 chunks, 2448 nodes | within 2.7e-4 of the absolute sum |
| CPU reference vs llama.cpp, 589 full tensors | cosine 0.999995 or better |
| Metal audio tower vs CPU reference, 7 clips including the two-chunk 35 s clip | row cosine 0.999999 or better, max abs diff 3.6e-3 on rows of norm 28 |
| Metal audio embeddings vs llama-server, all 29 fixture clips | min cosine 0.99998933 (the 35 s clip), max abs diff 7.2e-4 |
| A generated 5-minute clip (11 chunks, 7504 rows, not in the repository) vs llama-server | cosine 0.9999937 |
| Same clip alone and inside a batch of 5 mixed lengths, first or last | bit-identical |
| Text and image parity | unchanged (see Image parity and Parity) |
Run swift test -c release to reproduce (see Tests), or
gemma-embed --parity-audio Tests/GemmaEmbedTests/Fixtures/audio.
Measured on 2026-10-26 on the same Apple M3 Max as the text numbers, over 40 spoken clips made by
macOS say (scripts/make_speech_bench.py, not in the repository: 715 s in all, 3.3 to 85 s
each, median 8.7 s), under a GPU lock, at a load average of about 4 to 6. Wall time includes WAV
decoding and the log-mel front end on the CPU.
| Engine | Audio seconds per second, median of 5 rounds | Range |
|---|---|---|
| GemmaEmbed, default batching by mel frames | 274.9 | 271.0 to 324.7 |
| GemmaEmbed, one clip per GPU batch | 235.9 | 225.9 to 271.9 |
llama.cpp build 11716, llama-server --embedding --mmproj , one clip per request |
153.6 | 148.6 to 160.2 |
scripts/compare_llamacpp_audio.py models/audio/bench 5 reproduces the table; llama-server's side
includes HTTP and base64, so the gap slightly overstates the difference between the engines. Over
the same clips the two engines' vectors agree to min cosine 0.9999994.
Against the first correct implementation, the speed work made audio embedding 1.34 times faster
in wall time with default batching and 1.29 times with one clip per batch (interleaved A/B runs
with scripts/ab_audio.py, interquartile range 1.30 to 1.35), with the same vectors to within
3.4e-7. Most of it came from the CPU: the log-mel front end runs an in-place FFT and sums only the
non-zero taps of each mel filter (about 1.17 times wall on its own, bit-identical features), and
the next batch's clips are decoded while the GPU runs the current one. On the GPU, attention runs
one simdgroup per query and head (12 times faster per layer on long clips) and the second
subsampling convolution is an implicit GEMM on simdgroup matrices (about 4 percent of GPU time).
Wall time is now bound by the GPU, and three quarters of that is GEMMs already running near 10
TFLOPS. Tuning GEMM tiles for the audio shapes, split-K, fusing the elementwise passes, larger or
smaller batches, and double-buffered command encoding each measured within noise.
Ten sentences spoken by macOS say (scripts/make_speech_samples.sh, not in the repository) are
scored against their ten transcripts and five off-topic texts, all embedded as queries. Every
clip ranks its own transcript first, by 0.17 or more over the next text, and llama-server's own
vectors give the same ranks; GemmaEmbed's similarities agree with llama-server's within 1e-4. Own
cosines run from 0.80 to 0.90, and the smallest margin is a recipe step whose nearest other text
is the other cooking sentence. The test skips when the speech folder is absent.
swift build -c release
export GEMMA_EMBED_WEIGHTS=/path/to/embeddinggemma-2-Q8_0.gguf
echo "hello world" | .build/release/gemma-embed --task query --dimensions 256
.build/release/gemma-embed --parity Tests/GemmaEmbedTests/Fixtures
export GEMMA_EMBED_IMAGE_WEIGHTS=/path/to/mmproj-embeddinggemma-2-Q8_0.gguf
.build/release/gemma-embed --images photo.jpg ~/Pictures
.build/release/gemma-embed --parity-images Tests/GemmaEmbedTests/Fixtures/images
.build/release/gemma-embed --bench-images ~/Pictures --runs 3
.build/release/gemma-embed --bench-images ~/Pictures --batch-size auto
.build/release/gemma-embed --profile-images ~/Pictures --batch-size 1
.build/release/gemma-embed --bench posts.json --runs 3
.build/release/gemma-embed --profile posts.json
.build/release/gemma-embed --audio talk.wav ~/Recordings
.build/release/gemma-embed --parity-audio Tests/GemmaEmbedTests/Fixtures/audio
.build/release/gemma-embed --bench-audio ~/Recordings --runs 3
.build/release/gemma-embed --profile-audio ~/Recordings --batch-size 1
gemma-embed prints one JSON object per input line. --parity checks the tokenizer and the
embeddings against the llama.cpp fixtures and exits non-zero below cosine 0.9995. --bench times
a throughput run over a feed JSON file ({"posts": [{"gist", "text"}]}) or a plain text file with
one document per line. Image modes take the mmproj file from --image-encoder or
$GEMMA_EMBED_IMAGE_WEIGHTS: --images FILE... | DIR prints one JSON vector per image,
--parity-images DIR checks a fixtures folder (manifest.json and embeddings.jsonl) and
--bench-images DIR measures images per second, by default with patch-count batching, 8 and 1
images per batch (--batch-size auto or N picks one). Audio modes take the same mmproj file
from --audio-encoder, $GEMMA_EMBED_AUDIO_WEIGHTS or the image encoder path: --audio FILE... | DIR prints one JSON vector per clip with its row count and seconds, --parity-audio DIR checks a
fixtures folder, --bench-audio DIR measures audio seconds per wall second and per GPU second,
and --profile-audio DIR times each audio kernel class. --help lists the rest.
The shaders compile from .metal source when a model loads. GemmaEmbed looks for the SwiftPM
resource bundle GemmaEmbed_GemmaEmbed.bundle in this order:
- The main bundle's resources:
Contents/Resourcesinside an app. - The resources of the bundle that holds the GemmaEmbed code, then the directory next to it.
- The main bundle's own directory, which is where
swift runand a bare executable find it.
If no copy is found, GemmaEmbed uses the shader source compiled into the library, so a model
still loads with no bundle at all. It never reads the checkout's .build directory, which
SwiftPM's Bundle.module falls back to.
An app built outside Xcode must copy the bundle into Contents/Resources. Codesign refuses
loose files at the root of an .app. A build script can do it with this line, after it
sets SHADERS="$BIN_DIR/GemmaEmbed_GemmaEmbed.bundle" from swift build -c release --show-bin-path:
cp -R "$SHADERS" "$APP/Contents/Resources/"
After you edit a file in Sources/GemmaEmbed/Shaders, run scripts/embed_shaders.py to
regenerate Sources/GemmaEmbed/Metal/EmbeddedShaders.swift. ShaderSourceTests fails while
the two disagree.
swift test -c release runs the suite. The debug configuration also passes, in about six
minutes, because the CPU reference runs unoptimised. The tests read the weights and the
llama.cpp dumps from models/ at the package root, which is gitignored. An environment variable
overrides each path, and a test whose input is missing skips.
| Path | Contents | Override |
|---|---|---|
models/embeddinggemma-2-Q8_0.gguf |
the text weights | GEMMA_EMBED_WEIGHTS |
models/mmproj-embeddinggemma-2-Q8_0.gguf |
the image encoder | GEMMA_EMBED_IMAGE_WEIGHTS |
models/dumps/ |
llama.cpp's vision graph for six images ( scripts/vision_dump.cpp ) |
GEMMA_EMBED_VISION_DUMPS |
models/dumps-rgb/ |
stb_image's decode of every JPEG fixture ( scripts/vision_dump.cpp ) |
GEMMA_EMBED_DECODED_RGB_DUMPS |
models/fixtures/images/ |
the full image fixture set from scripts/make_fixture_images.swift , including the lavender field scene and its variant, which are over 250 KB and not in the package |
none |
models/audio/dumps/ |
llama.cpp's audio graph for seven clips ( scripts/audio_dump.cpp ) and miniaudio's decode of the decode-check WAVs (scripts/make_decode_checks.py ) |
GEMMA_EMBED_AUDIO_DUMPS |
models/audio/speech/ |
ten spoken sentences from scripts/make_speech_samples.sh with llama-server's vectors fromscripts/oracle_audio.py --speech |
none |
scripts/vision_dump.cpp and scripts/audio_dump.cpp list the commands that write the dump directories.
Reference vectors come from llama.cpp build 11716 (llama-server --embedding) over the same Q8_0
file, which is itself an approximation of the fp32 model, so agreement is bounded by that oracle.
| Check | Result |
|---|---|
| Tokenizer, 40 fixture texts (five scripts, URLs, numbers, emoji) | token ids identical |
| CPU fp32 reference vs llama.cpp per-tensor activation sums, 3 texts | every named tensor within 2 percent |
| CPU fp32 reference vs llama.cpp embeddings, 3 texts | min cosine 0.9999992, max abs diff 8.5e-5 |
| Metal kernels vs CPU reference (GEMM all shapes and tiles, RMSNorm, RoPE, attention at 1 to 2048 tokens, GeGLU, pooling) | max relative diff 4.5e-6 |
| Metal forward vs llama.cpp, all 40 fixtures | min cosine 0.9999988, max abs diff 1.4e-4 |
| Metal forward vs llama.cpp at 1054, 1629 and 3222 tokens (sliding window active) | cosine 0.9999995 or better |
| Metal forward vs CPU reference at 1672 tokens | cosine 0.9999999, max abs diff 1.8e-7 |
| Same text alone and inside a batch of 33 | bit-identical |
Run swift test -c release to reproduce (see Tests). scripts/make_text_fixtures.py
regenerates the text fixtures from llama.cpp.
Measured on 2026-10-25 on an Apple M3 Max (40 GPU cores, 128 GB) running macOS 26, Xcode 26.3, under a GPU lock with no other GPU work and a load average of 3.3, five runs. The corpus is 285 social media posts of 2026-10-17, a private set that is not in the repository (gist plus text cut at 2000 characters, document prefix), 41,675 tokens in all, 146 tokens per post on average, longest 604.
| Engine | Posts per second | Seconds for 285 posts | Tokens per second |
|---|---|---|---|
| GemmaEmbed 0.3.0 | 176.4 | 1.62 (1.61 to 1.66) | 25,790 |
llama.cpp 0.7.0-dev build 11716, llama-server --embedding -np 8 -ub 8192 , 32 texts per request |
118.3 | 2.41 | 17,300 |
llama.cpp, same build, -np 1 (one sequence at a time) |
59.6 | 4.78 | 8,720 |
GemmaEmbed 0.3.0 is 1.49 times faster than llama.cpp's best configuration and 2.96 times faster than its single-slot default, with the same parity (min cosine 0.9999988 against llama.cpp over the 40 fixtures). A single 22-token query takes 7.4 ms wall. The GEMMs take most of the GPU time and run at 7.6 to 8.8 TFLOPS fp32 in a sustained microbench; attention takes about an eighth.
Measuring on a shared machine: GPU time of short command buffers reads high because the GPU runs
at a low clock, so per-kernel lines of --profile mislead for kernels under about 2 ms, and a
microbench of short bursts can rank GEMM tiles the wrong way round. scripts/ab_bench.py runs two
builds alternately and calls a difference only when the median, best-of and per-batch-minimum
ratios agree; scripts/gpulock.py keeps two measurements off the GPU at once.
Load time: 0.27 s when the Metal shader cache is warm (tokenizer 0.17 s), about 1 s on the first load after a build (shader compilation 0.25 s, pipelines 0.3 s). Autotune, when enabled, adds about 4 s on the first load and nothing afterwards.
Reproduce:
swift build -c release
export GEMMA_EMBED_WEIGHTS=/path/to/embeddinggemma-2-Q8_0.gguf
.build/release/gemma-embed --bench /path/to/posts.json --runs 3
llama-server -m /path/to/embeddinggemma-2-Q8_0.gguf --embedding -c 8192 -b 8192 -ub 8192 -np 8 --port 8092 &
python3 scripts/bench_llamacpp.py /path/to/posts.json --port 8092 --batch 32 --runs 3
python3 scripts/ab_bench.py --a /path/to/old/gemma-embed --b .build/release/gemma-embed --bench /path/to/posts.json
The kernels follow these design rules:
-
The weight file is mapped
MAP_SHAREDand read-only, pre-touched from several threads, and wrapped in oneMTLBufferwith no copy. Tensors are offsets into that buffer. -
Shaders ship as
.metalsources and compile when the model loads (0.2 s), because SwiftPM does not produce a metallib from the command line. Every pipeline is built at load. See Packaging for where the sources are read from. -
All arithmetic is fp32. Q8_0 and BF16 are storage formats widened inside the kernels.
-
GEMM uses
simdgroup_float8x8with X and dequantized W tiles staged in threadgroup memory, and stages the output tile so rows leave as coalescedfloat4stores. The tile is picked by output size (smaller tiles for small batches, a wider one for inner dimensions above 2048), and every tile accumulates over K in the same order, so a row's result is bit-identical whatever batch it travelled in. Every loop over fragment arrays is fully unrolled: a rolled loop keeps the arrays in memory and runs several times slower. -
Attention is the new work: flash-style, bidirectional, over sequences packed back to back without padding. A threadgroup owns 8 to 64 query rows of one sequence for one head and streams 32-key blocks staged once in threadgroup memory; the output accumulators stay in registers and the online softmax runs in fp32 on the score fragments themselves. Small batches split each head into narrower slices to fill the GPU, and the vision tower has its own kernel for its 64-wide heads. Sliding-window layers mask
|i - j| > 512, the symmetric half window llama.cpp uses. -
The forward runs on a concurrent compute encoder with a buffer barrier after every dispatch, except within small groups of independent dispatches (q, k and v, for instance), which may overlap. A stress test runs two batches alternately on it and checks every result against a serial encoder bit for bit.
-
Padding is avoided entirely: texts are concatenated, every per-token kernel runs on the concatenation, and attention takes a work list of (sequence, query block) pairs.
-
Autotune, off by default, times whole forwards on real text for the GEMM tile and the attention query group, requires a 5 percent win, confirms the winner against the default and caches the result per device, kernel digest and model hash.
-
iOS and visionOS targets. With
.iOS(.v18)added to the platforms, the library compiles for iOS unchanged (iOS 17 lacksMTLCompileOptions.mathMode); it has not yet run on a device or simulator, and visionOS has not been built. -
F16 weight files, which are refused at load today, and a check of BF16 files against llama.cpp (the GEMM kernels read BF16 weights, but no BF16 file has been tested).
-
Fusing the text tower's q, k and v projections into one grouped GEMM, as the vision and audio towers already do, and the per-layer-input gate into fewer dispatches.
Peter Marelas (@maralski).
Copyright 2026 Peter Marelas. Apache License 2.0, the license Google releases EmbeddingGemma 2 and the Gemma 4 models under. See LICENSE and NOTICE. The model weights are not included and are licensed separately by Google; Google asks that deployments follow the Gemma Prohibited Use Policy. Code ported from llama.cpp and stb_image keeps its MIT notices in THIRD_PARTY_NOTICES.
Gemma is a trademark of Google LLC. This project is not affiliated with or endorsed by Google.