{"slug": "local-gemma-4-voice-assistant-on-jetson-orin-nano", "title": "Local Gemma 4 Voice Assistant on Jetson Orin Nano", "summary": "A hands-on tutorial adapts CAVEDU's original Gemma 4 Jetson guide to build a fully local voice assistant on NVIDIA's Jetson Orin Nano Super Developer Kit, running Gemma 4 E2B, NVIDIA Parakeet TDT 0.6B v3 ASR, and Kokoro TTS with no cloud inference API. The demo, built on llama.cpp compiled with CUDA and JetPack 6.2, lets Gemma 4 E2B request a webcam frame through its multimodal projector when it needs visual context, and requires an 8 GB swap file to handle memory demands. The setup credits Tzu-Hsien's CAVEDU tutorial and asierarranz's Cool_Demos Gemma4 example, and notes llama.cpp compilation on the Orin Nano can take roughly 40–60 minutes.", "body_md": "At GTC Taipei 2026, NVIDIA demonstrated a voice assistant running on Jetson Orin Nano that can listen, speak, and call on a camera when it needs visual context. This project recreates that local-first workflow with Gemma 4 E2B, NVIDIA Parakeet ASR, and Kokoro TTS. Speech recognition, language-model inference, text-to-speech, and image understanding run on the Jetson; no cloud inference API is required.\n\nThis is a hands-on adaptation of [CAVEDU’s original tutorial,](https://blog.cavedu.com/2026/07/09/gemma4-jetson-local-voice-assistant/) written by Tzu-Hsien. The model and demo code are credited to their upstream authors below.\n\nPress Space to record a question. Parakeet transcribes the audio, and a local llama.cpp server passes the text to Gemma 4 E2B. Kokoro speaks the response. When Gemma decides it needs visual information, the Python demo captures a webcam frame and sends it to the model’s vision projector. For example, ask, “Do you see anyone in front of you?” and the demo can inspect the camera image before answering.\n\n- NVIDIA Jetson Orin Nano Super Developer Kit (CAVEDU tested with JetPack 6.2; an SSD is recommended)\n- USB webcam (or another V4L2 camera)\n- Microphone and speaker; a webcam microphone and HDMI monitor speakers can work\n- Keyboard, mouse, and display for setup\n\n- JetPack 6.2 / Ubuntu on Jetson\n- llama.cpp built with CUDA\n- Gemma 4 E2B GGUF plus the matching multimodal projector\n- Cool_Demos Gemma4 example: [github.com/asierarranz/Cool_Demos](https://github.com/asierarranz/Cool_Demos)\n- Parakeet TDT 0.6B v3 ASR and Kokoro TTS, downloaded by the demo on first run\n\n```\nsudo apt updatesudo apt install -y git build-essential cmake curl wget pkg-config   python3-pip python3-venv python3-dev alsa-utils pulseaudio-utils   v4l-utils psmisc ffmpeg libsndfile1\n```\n\n2. Add swap and free memory\nThe demo needs substantial memory. Check that an 8 GB swap file does not already exist before creating it. Close applications you do not need; stop Docker only if you are not using its containers.\n\n```\nsudo fallocate -l 8G /swapfilesudo chmod 600 /swapfilesudo mkswap /swapfilesudo swapon /swapfileecho '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstabfree -h\n```\n\n3. Build llama.cpp with CUDA\n\n```\ngit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppcmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=\"87\"   -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Releasecmake --build build --config Release -j4\n```\n\nOn the Orin Nano, compilation can take roughly 40–60 minutes depending on the setup.\n\n4. Download Gemma 4 E2B and its vision projector\n\n```\nmkdir -p ~/models && cd ~/modelswget -O gemma-4-E2B-it-Q4_K_M.gguf   https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/90f9618340396838ee7ff5b0ba2da27da62953d3/gemma-4-E2B-it-Q4_K_M.ggufwget -O mmproj-gemma4-e2b-f16.gguf   https://huggingface.co/ggml-org/gemma-4-E2B-it-GGUF/resolve/4b90c7b785141802608550fc3cd3c715201532e2/mmproj-gemma-4-e2b-it-f16.gguf\n```\n\n5. Start the local inference server\n\n```\n~/llama.cpp/build/bin/llama-server   -m ~/models/gemma-4-E2B-it-Q4_K_M.gguf   --mmproj ~/models/mmproj-gemma4-e2b-f16.gguf   -c 2048 --image-min-tokens 70 --image-max-tokens 70   --ubatch-size 512 --batch-size 512   --host 127.0.0.1 --port 8080   -ngl 99 --flash-attn on --no-mmproj-offload --jinja -np 1\n```\n\nKeep this terminal open. Binding to localhost keeps the inference endpoint on the device; use a trusted network and appropriate access controls if you intentionally expose it.\n\n6. Set up the voice demo\n\n```\ngit clone https://github.com/asierarranz/Cool_Demos.gitcd ~/Cool_Demos/Gemma4python3 -m venv .venvsource .venv/bin/activatepip install --upgrade pippip install opencv-python-headless onnx_asr kokoro-onnx   soundfile huggingface-hub numpy\n```\n\nFind your device names with `arecord -l`, `pactl list short sinks`, and `v4l2-ctl --list-devices`. Set the values to match your machine:\n\n```\nexport MIC_DEVICE=\"plughw:0,0\"export SPK_DEVICE=\"alsa_output.platform-3510000.hda.hdmi-stereo\"export WEBCAM=0export VOICE=\"af_jessica\"\n```\n\nTest audio before starting the app:\n\n```\narecord -D \"$MIC_DEVICE\" -f S16_LE -r 16000 -c 1 -d 3 /tmp/test.wavpaplay --device=\"$SPK_DEVICE\" /tmp/test.wav\n```\n\n7. Run the assistant\n\n```\ncd ~/Cool_Demos/Gemma4source .venv/bin/activatepython3 Gemma4_vla.py\n```\n\nThe first run downloads the ASR and TTS models. Press Space to start recording and press Space again to stop. The transcript appears in the terminal, then the assistant’s response is spoken aloud. Ask it about something in view to try the camera-assisted response.\n\n📷📷📷\n\nNotes\n- The published demo’s Parakeet model is documented for English and several European languages; do not assume Chinese ASR support without changing and testing the ASR model.\n- Model performance and available memory depend on Jetson power mode, cooling, other running services, and model settings.\n- This is a maker demonstration, not a safety-rated or always-listening system. Camera input is captured by the local demo when requested by the model workflow.\n\nEnglish project adaptation by CAVEDU Education, based on [the original CAVEDU article.](https://blog.cavedu.com/2026/07/09/gemma4-jetson-local-voice-assistant/) Demo code: [asierarranz/Cool_Demos.](https://github.com/asierarranz/Cool_Demos) Gemma 4 and Parakeet model references are linked in the original guide.\n\n[Read more](<javascript:void(0)>)", "url": "https://wpnews.pro/news/local-gemma-4-voice-assistant-on-jetson-orin-nano", "canonical_source": "https://www.hackster.io/CAVEDU/local-gemma-4-voice-assistant-on-jetson-orin-nano-970215", "published_at": "2026-11-11 05:24:27+00:00", "updated_at": "2026-11-11 06:22:24.787149+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure", "computer-vision"], "entities": ["NVIDIA", "Jetson Orin Nano Super Developer Kit", "Gemma 4 E2B", "NVIDIA Parakeet TDT 0.6B v3", "Kokoro TTS", "llama.cpp", "CAVEDU", "Tzu-Hsien"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/local-gemma-4-voice-assistant-on-jetson-orin-nano", "markdown": "https://wpnews.pro/news/local-gemma-4-voice-assistant-on-jetson-orin-nano.md", "text": "https://wpnews.pro/news/local-gemma-4-voice-assistant-on-jetson-orin-nano.txt", "jsonld": "https://wpnews.pro/news/local-gemma-4-voice-assistant-on-jetson-orin-nano.jsonld"}}