If you have tried Nemotron-3-Nano-Omni-30B as a GGUF and found that audio or video silently does nothing, here is why, and the fix. Why the popular GGUFs are vision-only. The model has two encoder towers: C-RADIO for vision and a Parakeet/FastConformer encoder for audio. The widely downloaded GGUF repos ship mmproj files containing only the vision tower, so llama.cpp never sees the audio weights at all. And video needs more than weights: llama.cpp had no inference graph for Nemotron’s video path (temporal 2-frame patches plus EVS token pruning), so no projector file alone could enable it.
What a complete projector looks like. I released a unified mmproj that carries all of it in one file: the vision tower (v.), the full audio tower (a., 24-layer FastConformer with mel filterbank and STFT window), and the temporal video embedder, stored as a second patch embedding (v.patch_embd.weight.1, shape 16x16x6x1280: two RGB frames channel-concatenated). One llama-server flag serves image, audio, and video, including video WITH its soundtrack in a single pass.
Where:
Models + unified projector (9 quantizations, each tested on all three modalities before upload, graded matrix in TEST-MATRIX.md):
The llama.cpp fork with the audio and video graphs, plus prebuilt arm64 CUDA binaries (DGX Spark, Jetson Thor):
Verification: video graph matches NVIDIA’s PyTorch reference to 0.0019 percent relative L2 with exact EVS parity; the audio graph transcribes the test narration word for word.
Compatibility note: image and text work with the usual tooling; audio and video currently need the fork’s binaries. The next release adopts the upstream audio projector layout so audio will also run on stock llama.cpp; video stays fork-only until the planned upstream PR lands.
Base model and encoders are NVIDIA’s (NVIDIA Open Model Agreement). I wrote the conversion, the graphs, and the projector unification. Questions welcome.