{"slug": "nemotron-3-omni-in-gguf-why-audio-and-video-need-more-than-the-usual-mmproj-and", "title": "Nemotron-3-Omni in GGUF: why audio and video need more than the usual mmproj (and where to get ones that work)", "summary": "NVIDIA's Nemotron-3-Nano-Omni-30B model, when distributed as a GGUF, often lacks audio and video functionality because popular GGUF repos ship mmproj files containing only the vision tower, omitting the audio encoder and the video inference graph. A developer has released a unified mmproj and a llama.cpp fork that enable all three modalities, with the video graph matching NVIDIA's PyTorch reference to 0.0019 percent relative L2 and the audio graph transcribing test narration word for word.", "body_md": "If you have tried Nemotron-3-Nano-Omni-30B as a GGUF and found that audio or video silently does nothing, here is why, and the fix.\n\n**Why the popular GGUFs are vision-only.** The model has two encoder towers: C-RADIO for vision and a Parakeet/FastConformer encoder for audio. The widely downloaded GGUF repos ship mmproj files containing only the vision tower, so llama.cpp never sees the audio weights at all. And video needs more than weights: llama.cpp had no inference graph for Nemotron’s video path (temporal 2-frame patches plus EVS token pruning), so no projector file alone could enable it.\n\n**What a complete projector looks like.** I released a unified mmproj that carries all of it in one file: the vision tower (v.*), the full audio tower (a.*, 24-layer FastConformer with mel filterbank and STFT window), and the temporal video embedder, stored as a second patch embedding (v.patch_embd.weight.1, shape 16x16x6x1280: two RGB frames channel-concatenated). One llama-server flag serves image, audio, and video, including video WITH its soundtrack in a single pass.\n\n**Where:**\n\nModels + unified projector (9 quantizations, each tested on all three modalities before upload, graded matrix in TEST-MATRIX.md):\n\nThe llama.cpp fork with the audio and video graphs, plus prebuilt arm64 CUDA binaries (DGX Spark, Jetson Thor):\n\n**Verification:** video graph matches NVIDIA’s PyTorch reference to 0.0019 percent relative L2 with exact EVS parity; the audio graph transcribes the test narration word for word.\n\n**Compatibility note:** image and text work with the usual tooling; audio and video currently need the fork’s binaries. The next release adopts the upstream audio projector layout so audio will also run on stock llama.cpp; video stays fork-only until the planned upstream PR lands.\n\nBase model and encoders are NVIDIA’s (NVIDIA Open Model Agreement). I wrote the conversion, the graphs, and the projector unification. Questions welcome.", "url": "https://wpnews.pro/news/nemotron-3-omni-in-gguf-why-audio-and-video-need-more-than-the-usual-mmproj-and", "canonical_source": "https://discuss.huggingface.co/t/nemotron-3-omni-in-gguf-why-audio-and-video-need-more-than-the-usual-mmproj-and-where-to-get-ones-that-work/179221#post_1", "published_at": "2026-08-24 17:59:04+00:00", "updated_at": "2026-08-24 18:14:57.144444+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["NVIDIA", "Nemotron-3-Nano-Omni-30B", "llama.cpp", "C-RADIO", "Parakeet", "FastConformer"], "alternates": {"html": "https://wpnews.pro/news/nemotron-3-omni-in-gguf-why-audio-and-video-need-more-than-the-usual-mmproj-and", "markdown": "https://wpnews.pro/news/nemotron-3-omni-in-gguf-why-audio-and-video-need-more-than-the-usual-mmproj-and.md", "text": "https://wpnews.pro/news/nemotron-3-omni-in-gguf-why-audio-and-video-need-more-than-the-usual-mmproj-and.txt", "jsonld": "https://wpnews.pro/news/nemotron-3-omni-in-gguf-why-audio-and-video-need-more-than-the-usual-mmproj-and.jsonld"}}