Local Gemma 4 Voice Assistant on Jetson Orin Nano

At GTC Taipei 2026, NVIDIA demonstrated a voice assistant running on Jetson Orin Nano that can listen, speak, and call on a camera when it needs visual context. This project recreates that local-first workflow with Gemma 4 E2B, NVIDIA Parakeet ASR, and Kokoro TTS. Speech recognition, language-model inference, text-to-speech, and image understanding run on the Jetson; no cloud inference API is required.

This is a hands-on adaptation of CAVEDU’s original tutorial, written by Tzu-Hsien. The model and demo code are credited to their upstream authors below.

Press Space to record a question. Parakeet transcribes the audio, and a local llama.cpp server passes the text to Gemma 4 E2B. Kokoro speaks the response. When Gemma decides it needs visual information, the Python demo captures a webcam frame and sends it to the model’s vision projector. For example, ask, “Do you see anyone in front of you?” and the demo can inspect the camera image before answering.

• NVIDIA Jetson Orin Nano Super Developer Kit (CAVEDU tested with JetPack 6.2; an SSD is recommended)

• USB webcam (or another V4L2 camera)

• Microphone and speaker; a webcam microphone and HDMI monitor speakers can work

• Keyboard, mouse, and display for setup

• JetPack 6.2 / Ubuntu on Jetson

• llama.cpp built with CUDA

• Gemma 4 E2B GGUF plus the matching multimodal projector

• Cool_Demos Gemma4 example: github.com/asierarranz/Cool_Demos

• Parakeet TDT 0.6B v3 ASR and Kokoro TTS, downloaded by the demo on first run

sudo apt updatesudo apt install -y git build-essential cmake curl wget pkg-config   python3-pip python3-venv python3-dev alsa-utils pulseaudio-utils   v4l-utils psmisc ffmpeg libsndfile1

2. Add swap and free memory The demo needs substantial memory. Check that an 8 GB swap file does not already exist before creating it. Close applications you do not need; stop Docker only if you are not using its containers.

sudo fallocate -l 8G /swapfilesudo chmod 600 /swapfilesudo mkswap /swapfilesudo swapon /swapfileecho '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstabfree -h

3. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="87"   -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Releasecmake --build build --config Release -j4

On the Orin Nano, compilation can take roughly 40–60 minutes depending on the setup.

4. Download Gemma 4 E2B and its vision projector

mkdir -p ~/models && cd ~/modelswget -O gemma-4-E2B-it-Q4_K_M.gguf   https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/90f9618340396838ee7ff5b0ba2da27da62953d3/gemma-4-E2B-it-Q4_K_M.ggufwget -O mmproj-gemma4-e2b-f16.gguf   https://huggingface.co/ggml-org/gemma-4-E2B-it-GGUF/resolve/4b90c7b785141802608550fc3cd3c715201532e2/mmproj-gemma-4-e2b-it-f16.gguf

5. Start the local inference server

~/llama.cpp/build/bin/llama-server   -m ~/models/gemma-4-E2B-it-Q4_K_M.gguf   --mmproj ~/models/mmproj-gemma4-e2b-f16.gguf   -c 2048 --image-min-tokens 70 --image-max-tokens 70   --ubatch-size 512 --batch-size 512   --host 127.0.0.1 --port 8080   -ngl 99 --flash-attn on --no-mmproj-offload --jinja -np 1

Keep this terminal open. Binding to localhost keeps the inference endpoint on the device; use a trusted network and appropriate access controls if you intentionally expose it.

6. Set up the voice demo

git clone https://github.com/asierarranz/Cool_Demos.gitcd ~/Cool_Demos/Gemma4python3 -m venv .venvsource .venv/bin/activatepip install --upgrade pippip install opencv-python-headless onnx_asr kokoro-onnx   soundfile huggingface-hub numpy

Find your device names with arecord -l, pactl list short sinks, and v4l2-ctl --list-devices. Set the values to match your machine:

export MIC_DEVICE="plughw:0,0"export SPK_DEVICE="alsa_output.platform-3510000.hda.hdmi-stereo"export WEBCAM=0export VOICE="af_jessica"

Test audio before starting the app:

arecord -D "$MIC_DEVICE" -f S16_LE -r 16000 -c 1 -d 3 /tmp/test.wavpaplay --device="$SPK_DEVICE" /tmp/test.wav

7. Run the assistant

cd ~/Cool_Demos/Gemma4source .venv/bin/activatepython3 Gemma4_vla.py

The first run downloads the ASR and TTS models. Press Space to start recording and press Space again to stop. The transcript appears in the terminal, then the assistant’s response is spoken aloud. Ask it about something in view to try the camera-assisted response.

📷📷📷

Notes

• The published demo’s Parakeet model is documented for English and several European languages; do not assume Chinese ASR support without changing and testing the ASR model.
• Model performance and available memory depend on Jetson power mode, cooling, other running services, and model settings.
• This is a maker demonstration, not a safety-rated or always-listening system. Camera input is captured by the local demo when requested by the model workflow.

English project adaptation by CAVEDU Education, based on the original CAVEDU article. Demo code: asierarranz/Cool_Demos. Gemma 4 and Parakeet model references are linked in the original guide.

[Read more](<javascript:void(0)>)