Local Gemma 4 Voice Assistant on Jetson Orin Nano

A hands-on tutorial adapts CAVEDU's original Gemma 4 Jetson guide to build a fully local voice assistant on NVIDIA's Jetson Orin Nano Super Developer Kit, running Gemma 4 E2B, NVIDIA Parakeet TDT 0.6B v3 ASR, and Kokoro TTS with no cloud inference API. The demo, built on llama.cpp compiled with CUDA and JetPack 6.2, lets Gemma 4 E2B request a webcam frame through its multimodal projector when it needs visual context, and requires an 8 GB swap file to handle memory demands. The setup credits Tzu-Hsien's CAVEDU tutorial and asierarranz's Cool_Demos Gemma4 example, and notes llama.cpp compilation on the Orin Nano can take roughly 40–60 minutes.

At GTC Taipei 2026, NVIDIA demonstrated a voice assistant running on Jetson Orin Nano that can listen, speak, and call on a camera when it needs visual context. This project recreates that local-first workflow with Gemma 4 E2B, NVIDIA Parakeet ASR, and Kokoro TTS. Speech recognition, language-model inference, text-to-speech, and image understanding run on the Jetson; no cloud inference API is required. This is a hands-on adaptation of CAVEDU’s original tutorial, https://blog.cavedu.com/2026/07/09/gemma4-jetson-local-voice-assistant/ written by Tzu-Hsien. The model and demo code are credited to their upstream authors below. Press Space to record a question. Parakeet transcribes the audio, and a local llama.cpp server passes the text to Gemma 4 E2B. Kokoro speaks the response. When Gemma decides it needs visual information, the Python demo captures a webcam frame and sends it to the model’s vision projector. For example, ask, “Do you see anyone in front of you?” and the demo can inspect the camera image before answering. - NVIDIA Jetson Orin Nano Super Developer Kit CAVEDU tested with JetPack 6.2; an SSD is recommended - USB webcam or another V4L2 camera - Microphone and speaker; a webcam microphone and HDMI monitor speakers can work - Keyboard, mouse, and display for setup - JetPack 6.2 / Ubuntu on Jetson - llama.cpp built with CUDA - Gemma 4 E2B GGUF plus the matching multimodal projector - Cool Demos Gemma4 example: github.com/asierarranz/Cool Demos https://github.com/asierarranz/Cool Demos - Parakeet TDT 0.6B v3 ASR and Kokoro TTS, downloaded by the demo on first run sudo apt updatesudo apt install -y git build-essential cmake curl wget pkg-config python3-pip python3-venv python3-dev alsa-utils pulseaudio-utils v4l-utils psmisc ffmpeg libsndfile1 2. Add swap and free memory The demo needs substantial memory. Check that an 8 GB swap file does not already exist before creating it. Close applications you do not need; stop Docker only if you are not using its containers. sudo fallocate -l 8G /swapfilesudo chmod 600 /swapfilesudo mkswap /swapfilesudo swapon /swapfileecho '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstabfree -h 3. Build llama.cpp with CUDA git clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML CUDA=ON -DCMAKE CUDA ARCHITECTURES="87" -DGGML NATIVE=ON -DCMAKE BUILD TYPE=Releasecmake --build build --config Release -j4 On the Orin Nano, compilation can take roughly 40–60 minutes depending on the setup. 4. Download Gemma 4 E2B and its vision projector mkdir -p ~/models && cd ~/modelswget -O gemma-4-E2B-it-Q4 K M.gguf https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/90f9618340396838ee7ff5b0ba2da27da62953d3/gemma-4-E2B-it-Q4 K M.ggufwget -O mmproj-gemma4-e2b-f16.gguf https://huggingface.co/ggml-org/gemma-4-E2B-it-GGUF/resolve/4b90c7b785141802608550fc3cd3c715201532e2/mmproj-gemma-4-e2b-it-f16.gguf 5. Start the local inference server ~/llama.cpp/build/bin/llama-server -m ~/models/gemma-4-E2B-it-Q4 K M.gguf --mmproj ~/models/mmproj-gemma4-e2b-f16.gguf -c 2048 --image-min-tokens 70 --image-max-tokens 70 --ubatch-size 512 --batch-size 512 --host 127.0.0.1 --port 8080 -ngl 99 --flash-attn on --no-mmproj-offload --jinja -np 1 Keep this terminal open. Binding to localhost keeps the inference endpoint on the device; use a trusted network and appropriate access controls if you intentionally expose it. 6. Set up the voice demo git clone https://github.com/asierarranz/Cool Demos.gitcd ~/Cool Demos/Gemma4python3 -m venv .venvsource .venv/bin/activatepip install --upgrade pippip install opencv-python-headless onnx asr kokoro-onnx soundfile huggingface-hub numpy Find your device names with arecord -l , pactl list short sinks , and v4l2-ctl --list-devices . Set the values to match your machine: export MIC DEVICE="plughw:0,0"export SPK DEVICE="alsa output.platform-3510000.hda.hdmi-stereo"export WEBCAM=0export VOICE="af jessica" Test audio before starting the app: arecord -D "$MIC DEVICE" -f S16 LE -r 16000 -c 1 -d 3 /tmp/test.wavpaplay --device="$SPK DEVICE" /tmp/test.wav 7. Run the assistant cd ~/Cool Demos/Gemma4source .venv/bin/activatepython3 Gemma4 vla.py The first run downloads the ASR and TTS models. Press Space to start recording and press Space again to stop. The transcript appears in the terminal, then the assistant’s response is spoken aloud. Ask it about something in view to try the camera-assisted response. 📷📷📷 Notes - The published demo’s Parakeet model is documented for English and several European languages; do not assume Chinese ASR support without changing and testing the ASR model. - Model performance and available memory depend on Jetson power mode, cooling, other running services, and model settings. - This is a maker demonstration, not a safety-rated or always-listening system. Camera input is captured by the local demo when requested by the model workflow. English project adaptation by CAVEDU Education, based on the original CAVEDU article. https://blog.cavedu.com/2026/07/09/gemma4-jetson-local-voice-assistant/ Demo code: asierarranz/Cool Demos. https://github.com/asierarranz/Cool Demos Gemma 4 and Parakeet model references are linked in the original guide. Read more <javascript:void 0