Building a Robot Owl with No Robotics Experience

A developer with no prior robotics experience has built Hoot, a prototype robot owl companion for Chinese-language learning, entered in the Nebius × NVIDIA Global AI Hackathon. Hoot runs on a Jetson Orin Nano with a camera, USB microphone and speaker, and Sense HAT sensors, using NVIDIA Nemotron 3.5 Super VL through Nebius Token Factory with Hermes managing context and tools, plus local Faster-Whisper recognition and Kokoro speech with optional Gemini transcription and TTS. The project captures book pages, saves vocabulary and generates practice cards backed by local SQLite, served through a Vite and Three.js web dashboard; a shoulder mount and fully local reasoning mode remain plans.

I’m building Hoot https://github.com/matthewbcool/Hoot , a small robot companion for learning. A little owl that can look at what I’m working on, talk things through with me, and help me practice. It’s my noncompetitive project for the Nebius × NVIDIA Global AI Hackathon https://nebiusglobalaihackathon.devpost.com/ . Learning together Chinese storytime is my starting point. I’ve been making tools to help me study Chinese https://mattcool.tech/posts/hands-free-chinese-flashcard-maker/ for a while, and Hoot builds on that: capture book pages, explore unfamiliar characters, and save words for later practice. I’d like to see where else a little learning companion could be useful. What’s inside - Conversation and vision: NVIDIA Nemotron 3.5 Super VL through Nebius Token Factory https://docs.tokenfactory.nebius.com/quickstart , with Hermes https://hermes-agent.nousresearch.com/docs/ managing context and tools. - The physical owl: a Jetson Orin Nano https://developer.nvidia.com/embedded/learn/getting-started-jetson , camera, USB microphone and speaker, and Sense HAT https://www.raspberrypi.com/documentation/accessories/sense-hat.html sensors and LED expressions. Visual indicators show when the microphone is recording, when the camera is on, and when Hoot is speaking. - English and Mandarin voice: local Faster-Whisper https://github.com/SYSTRAN/faster-whisper recognition and Kokoro https://github.com/hexgrad/kokoro speech, or optional Gemini transcription https://ai.google.dev/gemini-api/docs/transcribe and TTS https://ai.google.dev/gemini-api/docs/speech-generation . In cloud mode, Nemotron still writes the answers. - Reading and review: captured pages, editable text, pinyin https://github.com/zh-lx/pinyin-pro , saved vocabulary, and practice cards, backed by local SQLite https://nodejs.org/api/sqlite.html . - Web dashboard: Vite https://vite.dev/guide/ and Three.js https://threejs.org/docs/ power the animated owl, camera preview, chat, and device controls. Hoot on screen The dashboard runs in the browser. The Jetson serves it through a local web server, which connects to Hermes running on the device. It’s a working prototype. In cloud mode, Hoot needs an internet connection to Nebius for the assistant’s reasoning. Speech can already run locally, and a local mode could use a model running directly on the Jetson Orin Nano for the reasoning too. A shoulder owl? I already have a battery for Hoot, and I want to work out a way to mount it on my shoulder. I’d also like to be able to position it in different places around a room, wherever it has a useful view of what I’m working on. The shoulder mount is still a plan. But having a little owl hanging out on my shoulder while I learn something sounds pretty fun. On the software side, I want faster responses, better reading alignment, and pronunciation help that notices when I hesitate or get stuck. Still plenty to work on, but I’m glad to have the pieces working together.