Show HN: Smart Mouth Billy Bass – another fishy LLM assistant

Developer ryandrake08 published a Show HN project called Smart Mouth Billy Bass, a local-only LLM assistant built into a Big Mouth Billy Bass toy, with code, schematic, and PCB layout on GitHub. The project was inspired by a prior Show HN post called BillAI Bass and aims for zero internet dependencies, LAN or on-device functionality only, original toy battery power, and no visible physical modifications to the chassis. The build involved a TFLite wake-word detector, ESP32 development, I²S devices, a MEMS microphone, KiCad, ideal diode controllers and MOSFETs, and SMT reflow soldering.

Since it matters to some people: This project was executed with some AI assistance. If you don't like this, feel free to stop reading. This writeup was human-written with little LLM involvement initial outline and formatting . The image above is pure slop. Sorry, I'm not an artist. | Software design and code | Hardware design and PCB layout | README / Documentation | |---|---|---| This project was inspired by a Show HN post called BillAI Bass https://news.ycombinator.com/item?id=48896599 where someone took a Big Mouth Billy Bass toy and shoved a Raspberry Pi in it to turn it into an AI assistant. I thought to myself, wow, this is cool, but can I improve on it? Thus began a months-long project to see if I could. I also learned a few things along the way, including GPU setup in an LXC container, LLM model applications, small-model training for a TFLite wake-word detector, ESP32 development, I²S devices, acoustic transmission to a MEMS microphone, KiCad, ideal diode controllers and MOSFETs, SMT reflow soldering, and more If you want to skip to the end result, here's a video demo-video , further down this page. If you want to skip to the code, schematic, and PCB layout: GitHub https://github.com/ryandrake08/billy The first step was to see what else was out there. Is there other prior art that can be built upon? Did someone else do the exact thing I was trying to build? Are people rage quitting these projects half way through? Besides the above Show HN project, there are quite a few other success stories to build upon: Someone used an ESP32 and custom chassis to integrate the toy with Home Assistant https://ciangallagher.me/2025/10/04/Turning-A-Singing-Fish-Into-A-SMart-AI-Assistant/ , closest to what I had in mind on the hardware side, but it leans on OpenAI's API for the actual conversation. I kind of want to not lean on any cloud AI providers for this thing. There is a Raspberry Pi / Arduino project that bundles everything together as a local AI assistant https://hackaday.io/project/201939-big-mouth-billy-bass-offline-ai . And some other Arduino/Assistant hacks: an early write-up on hacking the toy https://automateeverythingsite.wordpress.com/2016/11/20/hacking-big-mouth-billy-bass-part-13/ , a piece on animating the mouth from any audio source https://www.hackster.io/donald/animate-a-billy-bass-mouth-with-any-audio-source-c2b029 , a Bluetooth-audio variant https://maker.pro/arduino/projects/how-to-animate-billy-bass-with-bluetooth-audio-source , and an Alexa-hack forum thread https://forum.arduino.cc/t/help-wanted-for-big-mouth-billy-bass-alexa-hack/1361748 . All cool projects but not really what I had in mind. I started thinking about requirements: I want zero dependencies on the Internet. Local LAN or on-device functionality only. I hate it when you're supposed to talk to a computer, yet it introduces huge, unnatural latency into the conversation by requiring a round-trip to the Internet. No way. Local latency only. I want it to run off the original toy's batteries and/or power supply, and have low/no idle/standby power usage. I don't want to be changing the batteries every day, and I should not need to carry around an awkward bench DC supply to use it. I want minimal preferably no physical mods to the chassis. It should be totally stealth and look exactly like the original toy. No hint that it's different until it starts talking to you. I want it to be a simple drop-in replacement to the existing electronics. Most of the other projects involved stuffing huge Raspberry Pis or dev boards and a rat's nest of wiring into the existing toy. I wanted to see if I could do something simpler and cleaner. Finally, I didn't really want a useful assistant, just a goofy toy I could play around with and amuse guests with. A silly or witty voice was necessary. The biggest decision we'll tackle this later was whether to make it self-contained or whether it can rely on external compute. I don't want the latency and dependencies of Internet services like Google or Amazon, but I wouldn't be able to do everything in the fish chassis itself. I figured a local machine on the LAN wouldn't be cheating, as long as I could connect to it using the original toy form factor no ethernet wires sticking out , and I could keep latency under control. Bonus points if I could retain the toy's exact circuit board size and mounting footprint. Apparently, not all Big Mouth Billy Bass toys are the same. The hardware, electronics, and chassis have all undergone revisions since the original 1999 release. Some models have 3 motors one for each of mouth, tail, head , some have two one for the mouth, one doing double duty for tail/head . Some models have a barrel connector for regulated DC, and others just run off batteries. Some models take AA batteries and others take C batteries. Some sing Christmas songs, others sing normal songs. There appear to also be major chassis changes. I decided that this project would be tailored to the actual toy I had on hand. It would not be made generic enough to power all the different Billy Bass variants released over the years. Detecting and supporting multiple revisions of the toy might be an interesting future project for someone. Toy in hand, I decided to tear it down and see what I had. The chassis of my toy consisted of six small screws and four soft "feet". Once the screws were removed, the front and back halves separated easily. The front chassis contains the actual fish figure, and I learned that I have the three-motor variant, not the two-motor one this has circuit board BOM and software implications addressed later on . The front chassis also houses a pushbutton and what the box calls a "motion sensor." Closer inspection shows it's actually just a photocell LDR : it doesn't detect motion or heat at all, only a change in light level, like a shadow passing over it. The rear chassis contains the battery holder. I have the four-C-battery variant with a separate barrel connector for regulated 6V power. When a plug is inserted, the barrel connector physically disconnects the batteries also impacts the circuit design later . The rear chassis also houses a speaker for audio output, and a 2-position ON-ON switch — physically a DPDT package, but wired as a plain SPST toggle that just enables/disables the photocell trigger. The button is wired independently and works either way. There's no actual OFF position, but that's because the toy's animatronics chip is normally unpowered and only wakes up via a trigger circuit. The rear chassis also comes with a retractable stand, if you want to rest the toy on a desk. All wires from the motors, switches, power supplies and so on were connected to a small 70mm x 50mm circuit board attached to the inside of the front chassis with two screws, which implemented all of the toy's outputs. Although there is plenty of room inside the chassis to stuff electronics, I wanted to see if I could do the entire project with just that 70mm x 50mm circuit board constraint. My physical constraints ruled out having a lot of compute inside the toy chassis. So, I settled on a "thin client" architecture: the fish toy electronics would just run the motors, sensors, buttons, and audio, and the heavy lifting compute would happen on a more powerful backend machine on my LAN. I needed an in-toy board that had GPIOs, could drive motors and output/process audio, and had WiFi to wirelessly connect to my backend. On the backend, I'd run the Speech-To-Text, LLM I/O, and Text-To-Speech, where I had heaps of compute/GPU to spare. For the in-toy CPU, I settled on the tried and true ESP32-S3 chip. This should boot within seconds, give me about 8MB of RAM, 8MB of flash, plenty of GPIOs and, importantly, built-in 2.4 GHz WiFi. I could have used a Raspberry Pi, but the form factor is kind of bulky and it contains much more computing power than I need for the in-toy board, but not quite enough to do the backend's STT+LLM+TTS job. Along with the ESP32, we'll need a few other peripherals: three DC motor drives, an audio amplifier to drive the speaker I²S or similar , a microphone to listen to my voice I²S or similar , a 6V-to-3.3V power step-down to power the ESP32 and peripherals, and some kind of serial and/or JTAG connector for flashing and debugging. We'll go into the BOM and hardware choices later. One more decision: the firmware itself could also have been built on ESPHome, which plugs an ESP32-S3 straight into Home Assistant's ecosystem. Several of the prior-art projects above do exactly this. I have zero interest in running a Home Assistant stack myself, so I wrote custom C firmware instead, but it's a legitimate path if you're already living in HA. For the backend, I planned to simply use my existing dual Xeon E5-2697 v2 @ 2.70GHz "homelab" server already running Proxmox with a number of VMs and containers doing various things. I'd add a dedicated LXC container to run Billy's backend processes: I picked whisper.cpp for Speech To Text, llama.cpp running a local LLM, and Kokoro for Text To Speech; all three fit comfortably on an NVIDIA RTX 4000 Ada Generation GPU with 20GB VRAM. The other latency trick is architectural: the backend doesn't wait for the LLM's complete reply before doing anything. As soon as the LLM streams back a complete sentence, that sentence gets sent to Kokoro for synthesis and queued for playback, while the LLM keeps generating the rest of the response in the background. So the fish starts talking after the first sentence is ready. This gets first-audio latency down to a few seconds. My main concern about requiring a separate backend was latency. Whisper.cpp and llama.cpp were pretty clear choices for serving Speech To Text and the LLM from the backend, but I had a few choices for Text To Speech. I tried a couple out, including Chatterbox and Qwen3-TTS, but quickly settled on Kokoro TTS with its 82 million parameter model, due to its extremely high speed and low latency using the GPU , accuracy and lack of hallucination, and "OK" voices. The speed gap comes down to architecture: Qwen3-TTS is autoregressive, generating audio one speech token at a time, each conditioned on everything generated before it, which is inherently sequential, so it doesn't get meaningfully faster just by throwing more GPU at it. Chatterbox's token-generation stage is effectively the same story, even though its downstream vocoder uses flow matching. Kokoro is non-autoregressive: it produces a whole sentence's waveform in a single parallel forward pass, which is what gets it down to ~0.2 seconds instead of the multi-second waits I saw from the other two. The built-in voices are Kokoro's weakest attribute, at least for my application. They were very plain and corporate-sounding. It would have been nice to have more "silly" options. I picked one of the canned voices am onyx that sounded at least moderately silly. I suppose I could play with the voice using a DSP or something, but instead, I chose to build the fish's personality by way of a system prompt for the LLM. This provided infinite flexibility to improve/tweak personality, and it's actually kind of funny to hear a dry corporate-sounding voice try to tell fish jokes if it has the right attitude. Power management, sleep, and activation was the next major concern/decision. On one hand, I wanted the device to last a long time on battery power. The ESP32 can be put in a deep sleep mode, but that powers down the I²S peripheral entirely. I also wanted to be able to "activate" the toy via a typical assistant-like wake word like "Hey Billy". There's no low-power listening state: either the CPU is fully powered or it is in deep sleep. The only thing that stays alive during deep sleep is the chip's ULP ultra-low-power coprocessor. It can only read RTC-GPIO and RTC-ADC pins. It cannot read the digital I²S bus my mic talks over, and even if it could hear the audio, it doesn't have anywhere near enough compute to run a keyword-spotting neural net. So "half-listening" just isn't a state that exists on this chip. As a compromise, I planned to use the switch on the rear chassis to switch the unit between "button activation" which can interrupt deep sleep, and can idle for months on the toy's C-cell batteries and "wakeword activation," which needs the CPU and mic pipeline running continuously and drains the same batteries in about two days. By now, I had the overall system designed to my liking. My plan was to work on the backend first, which would primarily be container setup and off-the-shelf software installation, and work backwards from there toward the user. After the backend was done, I'd implement a basic CLI test client that recorded audio from a PC, sent it to the backend, and received response audio to play back on the PC, ensuring all backend components were working and latency was fine. After that, I'd breadboard the electronics and write the ESP32-S3 firmware, matching the "reference implementation" proven by the CLI. The breadboard would validate all peripherals on the device, including mic, amp, all motors, the button, the switch, and the photocell sensor. Finally, once all the on-fish and off-fish functionality was working via the breadboard, I'd design and produce a custom PCB and test it on the toy. First, I set up an LXC container on my host to allow GPU passthrough the GPU was already installed in the homelab, but didn't have much to do yet and got all the NVIDIA/CUDA drivers installed and working. I kept all three AI services in this one container rather than splitting them up. More containers would each need their own GPU passthrough. Then I installed whisper.cpp, llama.cpp, and Kokoro onto that container, and downloaded Qwen3-8B-Q4 K M. I independently verified each service was listening and responding properly, and the AI models fit in VRAM. My homelab has heaps of system RAM, so I was pretty much unconstrained there, but the single GPU only has 20GB VRAM, which could be tight. Ultimately, llama.cpp+Qwen3-8B took up around 5.4GB, whisper.cpp 3.7GB, and Kokoro 1.2GB, in total using only half of my VRAM. Very nice, lots of space With the off-the-shelf software installed and working, I needed to start thinking a little bit about the boundary between the in-fish software and the backend. Would the fish call each backend service directly, or would it operate through a higher-level application layer also running on the host? Pros and cons for each scenario. It would have been nice to have a single application endpoint for the fish to interact with, letting it handle the entire STT+LLM+TTS round trip. On the other hand, I'd be concerned about latency and copying audio all over the place, and the ability to play back audio sentence-by-sentence as the LLM generates it. This sentence-by-sentence pipelining means TTS synthesis and LLM decode briefly share GPU compute. The other design, fish firmware calling each backend independently, would be simple, but I'd have to bake more application logic than I'd like into the embedded firmware, like the system prompt, conversation history, and so on. I ended up with a design where the fish firmware calls whisper.cpp and Kokoro directly, but calls into the LLM via a Python "shim" application running on the host. The shim is built on FastAPI/HTTPX and sits in front of llama.cpp , holding on to the system prompt and per-session data, handling all model-specific quirks such as Qwen3's /no think and stripping stray <think tags, scrubbing markdown and emoji from the output, and doing sentence-splitting on it. Later, I also decided to host firmware configuration flags in the shim, so I could experiment with changing various firmware settings without reflashing the device. I also revisited the LLM choice. Qwen3-8B did the job, but I found Qwen3.5-9B's answers to be actually pretty informative, too. Non-joke questions were noticeably more accurate and smarter, so I swapped it in. Also, at Q8 0 rather than the Q4 K M I'd been running. Q8 0 runs at roughly 1.7x the bytes-per-weight of Q4 K M, which reduced token generation speed from 63/sec to 36/sec. I decided it's worth it: the jump in how smart and accurate Billy's non-joke answers feel outweighs the extra ~100ms to 200ms of latency, at least for now. VRAM capacity is still fine. A live snapshot with all three engines resident: llama.cpp now Qwen3.5-9B at 8.5GB, whisper.cpp at 3.7GB, Kokoro at 2.3GB: about 14.2GB combined. Once the backend applications and "shim application" were built and running, I wanted to validate the entire STT+LLM+TTS path end-to-end, to judge whether it was fast enough to be considered conversational. For this, I built a small Python CLI that prompts the user to record audio, sends it off to whisper.cpp to receive the prompt text, sends that prompt to the shim app for processing, receives the LLM's answer sentence-by-sentence, sends the answer text to Kokoro for vocalization, and finally plays the response back through the PC. I tried a few utterances through the command-line client and, with sentence-level pipelining doing its job, measured roughly 2 seconds to first audio: comfortably within what feels conversational. Only at this point was I convinced this was truly a feasible project Kokoro, whisper.cpp, and Qwen all turned out to be multilingual already, so there was no reason to keep Billy English-only. I picked a handful of languages Kokoro had male voices for Spanish, Brazilian Portuguese, Mandarin, Japanese, Hindi, Italian , wired whisper's detected language through to pick a matching TTS voice, and added a small classifier so a phrase like "say it in Chinese" mid-conversation switches just that phrase's voice. The one surprise: building a proper test with clean reference audio turned up a bug where whisper.cpp had been silently translating non-English speech to English text instead of transcribing it. Fixed with one missing request parameter. Billy now holds a conversation in seven languages. The ESP32 occupies a nice middle ground between the simple, low-powered Arduino-class devices and something like a Raspberry Pi, which is overkill for the application. There are other SoCs in this category, but none as well documented and supported. ESP32s have built-in WiFi, two I²S, built-in ADC, µA deep-sleep, and are offered with many RAM and Flash configurations suitable for this application. The dev board I started with was the N8R8 8MB flash, 8MB octal PSRAM , fine for development, but I found later that my final firmware fit comfortably in a smaller configuration. The app binary itself ended up being ~1.3MB, and PSRAM usage peaked at around 1MB. That's small enough that the final board will use the smaller ESP32-S3-N4R2 4MB flash / 2MB quad PSRAM instead. Same footprint and pinout as the N8R8, and about 14% cheaper. As for peripherals, honestly my choices were largely driven by availability of ready-to-test breakout boards from vendors like Adafruit and SparkFun. For the microphone, I was looking for a simple I²S device. I²S sidesteps two problems an analog electret mic would have on this chip: the ESP32-S3's second ADC ADC2 is unusable whenever WiFi is active, and an analog signal chain needs its own amplification/filtering stage, adding noise and CPU load. I²S also pairs naturally with the ESP32-S3's two independent I²S controllers, giving true full-duplex mic-plus-speaker with no shared-clock juggling. It needed to run on 3.3V and be positionable somewhere on the fish's chassis, away from the main board. The TDK ICS-43434 was available on a very small breakout board that could be placed anywhere. It's also TDK's current, in-production drop-in for the INMP441, an equally capable but now end-of-life part with identical wiring, so either would have worked equally well. Instead of building a separate board, my plan was to use the breakout board as-is and attach it somewhere inside the toy, one of the few toy hardware mods I planned to accept. I did skip the SPH0645, a similarly priced I²S mic, for its narrower 100 Hz–10 kHz frequency response and a known DC-offset erratum. For the speaker amplifier, again I wanted an I²S device that runs on 3.3V. I settled on the Analog Devices MAX98357A due to availability of a nice tidy breakout board and its low quiescent and standby current. Its datasheet rates it for 3.2W into a 4Ω load, but that's at a 5V supply. On this design's single 3.3V rail, driving the stock 8Ω speaker, realistic output is closer to ~0.7W. I also later looked at the newer MAX98360A, which drops the external gain-set resistor and MCLK requirement for a slightly simpler board, but stuck with the MAX98357A since I'd already scoped the rest of the design around it. For the motor controllers, this is where the 3-motor vs. 2-motor toys made a difference: most of the motor driver ICs I considered handle only 1 or 2 channels per chip. So a 3-motor toy needs at least two separate chips, whereas a 2-motor toy gets away with a single dual-channel one. One exception I did look at, TI's DRV8908/8912 family, packs many channels behind an SPI interface. Overkill firmware complexity for three motors, so I passed. I also considered cheaper options: the MX1508 my original assumption before I'd opened the toy and the older L293D/SN754410, both rejected for the same reasons: no sleep or fault pins, higher dropout voltage, and a generally poor fit for battery operation. Since all three of this toy's motors turned out to be spring-return and unidirectional, I also considered driving them with simple discrete low-side MOSFETs plus a flyback diode instead of a dedicated driver chip, which would be cheaper and simpler, but it forfeits fault detection and overcurrent/thermal protection. You'll see later that this became irrelevant due to all of the problems I encountered with fault detection. A breakout board for the TI DRV8833-series chip was readily available, which is what mostly swayed my decision. Since the device's battery pack and DC barrel socket provide ~6V power, a voltage-reducing buck is required to deliver stable 3.3V to the ESP32, mic, amp, and GPIO pull-ups. I originally considered the MPS MPM3610 with built-in MOSFET and inductor. Again, it came packaged in a nice, easy-to-integrate Adafruit breakout board, and its datasheet quotes a low 200μA quiescent current, though that turned out to be an optimistic best case. It's a multi-mode part with three different working modes, and its worst-case mode continuous conduction draws closer to 0.7mA. Between that, it being fiddly to wire up and control, and its relatively difficult-to-work-with QFN-20 package, I moved on. During PCB layout, I chose to switch over to a DIODES Inc. AP63201 low-IQ power converter, which also includes integrated power MOSFETs, requires a separate inductor, but cuts quiescent current to 22µA: roughly 30x better than the MPM3610's worst case. In order to support the battery, the toy's native 6V barrel jack, and added later a Micro-USB port for bench convenience, I needed a way to combine multiple power sources safely ahead of the buck, and to protect the battery pack against reverse polarity. I ended up using three TI LM74700-Q1 ideal-diode controllers, each driving an external n-channel MOSFET as the actual pass element, instead of simple Schottky diodes: a good Schottky still costs ~0.3–0.5V at the ~3A worst-case motor draw, while the ideal-diode's MOSFET drops only tens of millivolts at the same current. One cost of that choice, found later: each LM74700-Q1 draws 80µA and can't be gated off during deep sleep, since it's literally part of the power path feeding the very chip that would need to switch it off. Finally, I added two 5A slow-blow fuses, one each on the battery and 6VDC power inputs, for circuit protection. Slow-blow matters specifically here: measured worst-case current with all three motors stalled simultaneously came in around 1.5A and I've seen it run closer to 1.8A if WiFi happens to be transmitting at the same moment , so a fast-blow fuse anywhere near that range would nuisance-trip during ordinary use. 5A slow-blow rides through that comfortably while still popping on an actual short. Dev boards in hand, I went to work wiring up each of them to a small breadboard, powered by the toy's battery box, matching the final design as closely as possible. I added them one at a time, validating each one before moving on to the next. This required some small amount of test "Hello world" firmware development. First up was the bare ESP32-S3 dev board, and I verified it would turn on, accept a flash write, and read a GPIO pin. At first, I powered it only via the Micro-USB connection. Next up was the microphone, where I added a simple tester that logged the received sound level over and over. Next was the amp, where I played a few tones on startup to verify proper function. Before getting to the motors, I switched the whole breadboard over from USB to ~6V power, which meant wiring in the voltage step-down buck to keep the ESP32 and mic/amp running at their regulated 3.3V now that USB wasn't there to provide it. The motors themselves, by contrast, run off the 6V rail with no buck involved at all. Finally, I connected the motor controllers and ensured their power was pulled directly from the battery / DC adapter. I hooked each motor controller output to the fish motors and strobed them all to verify function. I wanted to understand what duty cycle was needed to actuate the actual fish parts, so I wrote a short test that swept duty in steps and logged how the toy responded at each one. I found the mechanism barely moves at all below about 65% duty, needing close to 100% for a full stroke; the spring preload and gearing eat almost the entire useful range. I also wrote a stress test to run all three motors at 100% simultaneously to quantify worst-case current draw. This is where I got the ~1.5A figure I used earlier to size the fuses. Once all controls and peripherals were confirmed working on the breadboard, it was time to move on to real firmware. The overall structure of the firmware is an app main function that brings up every device in turn. Peripherals status LED, photocell, Vmotor-sense first, then motors, audio, activation, the wake-word detector, and networking, before spawning runloop task to run the fish's main loop forever. At a very simplified high level, the main loop is essentially: IDLE ACTIVATE LISTEN THINK SPEAK IDLE. It also handles invoking and returning from deep sleep, fetching backend-controlled configuration once per loop cycle, driving the status LED and motors, and rebooting if needed. The main loop calls into peripheral code contained in audio.c , motors.c , activation.c , and peripherals.c : audio.c handles I²S in both directions: energy-based voice-activity detection on the mic side wait for speech to start, keep capturing until about 0.8 seconds of silence , and playback plus the mouth lip-sync envelope on the amp side. motors.c drives the two DRV8833s across three channels mouth, head, tail , each spring-return and unidirectional. After a board rev, the two chips were organized into two independently controlled groups. Mouth+head share one driver, tail has the other. Each with a nSLEEP and nFAULT line, so waking, sleeping, or latching a fault for one group never touches the other. A falling edge on either group's nFAULT fires an interrupt that latches a fault for that group and wakes a task. I currently just log, but there is also experimental code to stop the motors in this case. It's not enabled becuase I saw lots of spurrious faults from the motors while they were active, and couldn't explain them. activation.c owns the button, the mode switch, and BUTTON-mode deep sleep. The interesting part is a subtle bug it has to design around: on a button-triggered wake from deep sleep, the chip fully reboots and runs app main again from scratch, so if that fresh boot just re-entered the normal idle state, it would immediately call back into deep sleep and re-sleep on the very button press that woke it, without ever acting on it. The fix is activation deep sleep wake pin , which reads which of the two armed deep-sleep wake pins button or mode switch actually caused this boot; the runloop uses that to skip idle entirely on a button-caused wake, treating the press that woke the chip as this turn's activation event, while a cold boot or a mode-switch-caused wake goes through idle normally. Every wake candidate, button or wake word, is also gated against the photocell peripherals.c , so a wake in a pitch-dark room gets rejected. peripherals.c combines the photocell, the Vmotor-sense divider, and the status LED: three small pieces of hardware that all share one gate, BOARD PERIPHERALS EN . The photocell read compares a raw ADC1 reading against a compile-time threshold currently 20 to decide whether a wake candidate is plausible or just a dark-room false positive. The Vmotor read averages eight ADC1 samples. Bench data showed 1–1.6% scatter on a single read, from rail ripple under motor load and/or ADC jitter, and applies a slope/offset to turn the count into volts. The status LED is a single WS2812 pixel that shows one color per state white=boot, green=idle, blue=listening, cyan=thinking, violet=speaking, red=fatal error . Underneath all of those sits hal.c , a low-level hardware abstraction layer. It only knows about GPIO, the shared ADC1 unit, one WS2812 LED strip over RMT, PWM via the LEDC peripheral one shared timer driving several independent duty channels , one I²S TX and one I²S RX channel, and deep sleep. Every function takes a pin or channel number as a parameter; nothing in this file hardcodes what any of those pins are for . That mapping lives entirely in board.h and gets interpreted by the device-layer files. net.c is the fish's entire relationship with the outside world: three calls per turn: audio sent to whisper for STT, transcript to the shim for the brain hop which streams back sentences over SSE , and each sentence to Kokoro for TTS. It's also abstract, in that if I ever moved the fish off custom HTTP and onto ESPHome or Home Assistant, this is the only layer that would need to change. WiFi join and reconnect run asynchronously on their own task for the entire life of the app. The fish doesn't block boot waiting for an IP, and a turn that needs the network before it's ready just fails gracefully. fish config.c manages server-controlled config values: instead of hardcoding things like VAD sensitivity, mouth lip-sync tuning, tail choreography timing, backend HTTP timeouts as define s, they're fetched at runtime from the shim's /v1/config endpoint. That means retuning any of these can be accomplished by editing the shim application. Every field also has a compiled-in fallback. I left a few compile-time constants, mostly things that we need as soon as the device boots. Last but not least is the wake-word detector. wakeword.cpp is just a thin C-callable wrapper around a TensorFlow Lite Micro model. It takes 10ms slices of mic audio and returns true when it hears "Hey Billy." activation.c 's wake-word listening loop just calls it in a tight loop with live mic samples. This is the one piece of the project I leaned on LLM knowledge and coding the most, and understood the least well going in: on-device wake-word detection. Rolling my own neural network felt like overkill, and not something I wanted to do. Home Assistant's voice hardware the ESP32-S3-based Voice PE does exactly this job on exactly this chip, via ESPHome's micro wake word component. Instead of pulling in big chunks of Home Assistant just for this one piece, I ported the two parts that actually matter the audio-to-features frontend and the streaming TensorFlow Lite Micro model into a synchronous C API wakeword init / wakeword reset / wakeword feed that the firmware's mic-read loop drives directly. That gets you the detector, but not a model that knows what "hey billy" sounds like. For that I had to train my own. The training framework is OHF-Voice/micro-wake-word . I ran the whole thing on my homelab LXC container. Getting positive samples for the wake phrathee is the first problem any of these frameworks has to solve, since nobody's uploaded a dataset of people saying "hey billy". piper-sample-generator synthesized 1000 clips of the phrase using a single TTS voice. Negative examples came from the training framework's pre-built HuggingFace sets: general speech, "dinner party" background chatter, and pure non-speech noise, weighted differently against each other during training speech and dinner-party chatter weighted twice as heavily as plain non-speech, since they're the harder negatives to correctly reject . A fourth negative set is held out entirely from training and used only to validate against afterward. 1000 TTS clips of one voice isn't much to train a real-world detector on by itself. Each clip gets run through a pipeline that randomly applies EQ, pitch shifting, distortion, and band-stop filtering about a 10% chance each , then background noise 75% chance, mixed in anywhere from -5 dB to +10 dB SNR and simulated room reverb 50% chance . The model would therefore be trained on hundreds of noisy, reverberant, pitch-shifted variations of the wakeword. Training itself turned out to be the easy part. 10,000 steps, batch size 128, negative examples weighted 20x more heavily than positive ones in the loss, biasing against false accepts. A fish that wakes up when nobody said anything is more annoying than one that occasionally needs a second try. All training took place on the same RTX 4000 Ada GPU the backend runs on. Wall-clock time for the whole pipeline was about 45 minutes. The resulting model is tiny: 26,049 parameters, about 100KB, a small stack of depthwise-separable convolutions. Picking the detection cutoff is an interesting judgment call: the model outputs a wake-word probability, and where you draw the line trades false rejections against false accepts. At a cutoff of 0.84, the model missed 5% of correct utterances but only false-triggered about once every five hours 0.187/hour on the validation audio; loosening the cutoff to 0.59 cut the miss rate to 2%, but pushed false accepts up to 2/hour. 0.84 was chosen to reduce false-triggers, which, again, are more problematic than missed triggers. The best part: it just worked. The LLM one-shotted this and out the other end popped a .tflite file. I tested the trained model on real hardware expecting to iterate, and it detected "hey billy" reliably on the first pass, with no retraining or retuning needed. Prior to this project, I had a small amount of experience using Eagle to put together basic thru-hole PCBs for small toy projects, like 555 timers, RS232-to-TTL circuits, simple GPIO toggles and flashing LEDs, but nothing involving a microcontroller, digital IO, USB signaling, multiple voltages with step-downs, power brownouts, or ESD/circuit protection. Now, Eagle has been discontinued by Autodesk thanks, assholes , which has forced me over the KiCad learning cliff. The first hurdle was the mental model. Eagle, at least at the toy-project level I'd used it, let value and footprint and symbol blur together into one part you'd just pick off a list. KiCad insists on splitting a part into a library definition a symbol and a footprint, living in a library file and a project that only ever places instances pointing at that definition. Copy a placed part from one project straight into another and it just breaks, because the destination project's library table has no idea what you're talking about. It took embarrassingly long for this to click. Once it did, though, one thing KiCad does noticeably better than what I remembered of Eagle: there's exactly one resistor symbol and one capacitor symbol. Value is just a text field, and footprint gets assigned separately, even in bulk, across every resistor on the board in one pass, instead of hunting down a different symbol per value/package combination. Since most of my peripherals mic, amp, motor driver, the ESP32 module itself already had Adafruit or SparkFun reference designs, I didn't want to redraw every IC's symbol and footprint from a datasheet by hand. I wanted to harvest the existing ones. That turned out to be its own small adventure, and a completely different process for symbols versus footprints. Symbols come out through KiCad's Eagle importer into a flat library file, then a genuinely fiddly Symbol Editor dance: add the imported library, add your project's library, single-click not double to select the part without opening it for editing, copy, click the destination library itself, paste, save. Footprints are worse: they're not extracted anywhere on import at all, they're embedded straight into the imported PCB file, so getting one out means opening that PCB, right-clicking the part, hitting "Edit Footprint," and then clicking through not one but two separate "this is read-only, click here to actually save an editable copy" banners before it lands somewhere reusable. None of this is documented anywhere obvious; I mostly reconstructed the sequence by clicking around until each step's action revealed itself. A few KiCad rough edges: KiCad runs single-instance on macOS. Opening a second project doesn't open a second window, it reuses the same process, so I spent a while confused about why the Symbol Editor seemed to think I was still in the wrong project, before realizing I needed to launch a second, fully independent process from the terminal just to view a reference schematic and my board side by side. Separately, and this one's an actual open KiCad bug: a newly-added project-local library can silently fail to even show up in the Symbol/Footprint Editor's library panel. The fix was registering my parts library globally instead of project-locally. And once a library's global, its path has to be a fixed absolute path. The project-relative syntax that's normal for a project-local entry silently re-resolves to whatever project happens to be open at the time for a global one, which showed up as KiCad cheerfully offering to create my library inside a completely unrelated project. Honestly, fighting so much with KiCad libraries and the interface to these libraries, encouraged me to, as much as possible, use stock KiCad symbols and footprints rather than fool around with creating my own. The struggle around libraries is real, and IMO is the worst part of KiCad. Not everything on the schematic was actually breadboarded first, but most of it was: breakout boards for the microphone, amplifier, motor controllers, and 3.3V buck were all tested directly on my breadboard before being realized into a circuit board. The Activation Button, Mode Switch, Photocell, and Voltage Sense circuits were all validated on-breadboard, too. The TTL, boot selector, reset switch, Status LED, and USB hardware/signaling came by default with my ESP32-S3 dev board, so they weren't independently validated until the board showed up. Most of the peripheral design was lifted off existing imported Eagle schematics from Adafruit/SparkFun, or painstakingly reproduced from each peripheral device's datasheet. Most of the power distribution system, including the ideal diode controllers, was conjured up from the datasheet itself, and won't be tested for real until the rev.0 board is manufactured. I agonized over resistor and capacitor values before realizing that a lot of the specifications were approximate, and it was only important that the final product "met or exceeded." Once the schematic was acceptable, the board layout task was much more familiar and easy to ramp up on. The GUI and concepts are largely the same and carried over from Eagle. I don't care what anyone says, board layout is artwork. I thoroughly enjoyed the challenge of packing all of these components into a small 70mm x 50mm circuit board. And, yes, the board ended up densely packed. The physical size constraint wasn't actually a dealbreaker, as there was actually plenty of space inside the cavity of the toy chassis for a larger board. But, the aesthetic and challenge, and the principle of matching the original toy's form factor exactly, was a nice challenge and worth pursuing for its own sake. Before the first PCB order went out, I hit two separate flavors of the same symptom on the breadboard: the ESP32 would spontaneously reset. Sometimes a soft BROWNOUT RST mid-boot, sometimes a harder POWERON reset with no panic at all. This would happen under WiFi's TX bursts, and the fish's head+mouth motors moving together. My first suspect was the buck converter itself: maybe it just couldn't keep up with a fast enough load step. After some diagnostic work, I narrowed the cause down to the actual long breadboard power rails. When I hooked things up physically closer to the buck converter, the brownouts stopped happening, but when attached further down the breadboard, the brownouts returned. That made the PCB the deciding test. And that's exactly what the first rev.0 board confirmed once it was built and running: no brownouts, on battery or adapter, under any load I threw at it, including a worst-case three-motor stress test. The lesson I took from the whole saga: a breadboard build's iffy connections can look exactly like a failing component. One of the design goals of this project was to be totally stealth, indistinguishable from the "dumb" toy. So far, I have managed to honor this, but there was one problem that could not be solved in a satisfying way without compromise: How to get the user's voice to the microphone port with the circuit board mounted inside the plastic chassis? Some ideas: Finally, what I chose: I took the Adafruit breakout board itself, and positioned the on-board microphone port right up against the inside of the toy's front chassis. I drilled a tiny 52 ~1.6mm hole in the chassis, exactly aligned with the microphone port. I countersunk the inside of the chassis around the hole a bit, to improve acoustic transmission. Then I 3D printed a small "caddy" to fit around the existing button hardware, to hold the microphone board in place, flat against the inside chassis of the fish. Finally I sealed up the edges around the circuit board with a little form-a-gasket to prevent vibration. The resulting hole is not easily visible, and mostly hidden behind the fish, but the sound quality received is excellent. This small hole is the only hint that the Big Mouth Billy Bass has been modified The two DRV8833 motor controllers already protect themselves against overcurrent, overheating, and undervoltage, and their sense resistors limit motor current in hardware. Both chips also exposed an open-drain nFAULT signal, wire-OR'd onto one ESP32 input. Initially the firmware merely sampled that pin after a movement and printed a warning. That was not especially useful: an overcurrent indication can be brief, and by the time a movement or spoken response finished, the signal could already be gone. It was time to actually handle the nFAULT signal from the chips and use it to stop the motors safely. I added code to the firmware to make the falling edge interrupt the ESP32, immediately pulling the controllers' shared nSLEEP line low, to stop the motors. I also latch the fault until later, when I can perform a controlled recovery. Unfortunately, when I tested rev.0 of the PCB, I found the chips' actual behavior forced me to disable the fault handling code. More on that later. I briefly considered paying for both board fabrication and part assembly, but the cost was pretty high for a hobby project, and JLCPCB charges extra for any parts not in their "basic" category. I had access to a hot plate so I figured now was a good time to learn how to reflow a board full of SMD parts. So I ordered the board from OSHPARK and the components from DigiKey and Mouser. This part was pretty tedious, because every part including the resistors and capacitors had multiple slightly-different variants, which I felt I needed to scrutinize before ordering. Kind of a pain. Also, it's painful to get a whole plastic bag with a single 0603 resistor in it. The poor environment You'd think parts distributors could solve this in a more eco-friendly way. Maybe I should just buy a kit with thousands of little parts to save on all this packaging. The components came from DigiKey and Mouser almost instantly, but the boards took two weeks to arrive, which felt pretty slow. When the boards finally came, it was show time. I'm used to thru-hole soldering, so this was all new to me. I also ordered a polyimide stencil, so applying solder paste was just a matter of globbing it on and spreading it with a credit card over the stencil. Easy. The next step was to populate the components. These things are SMALL and you really need a steady hand to grip them with your tweezers and position them correctly. I found myself holding my breath because it felt like I could easily blow these little components off my desk. Noted for future revisions: I need to change the PCB to use a minimum of 0603 sized parts. 0402 is just too small to work with by hand, and I'm lucky I managed to get them correct. One thing that was surprising: the solder paste was just sticky enough to grab the components. I thought I'd have to hold the board extremely carefully, but once you get everything into the paste, the components don't really move around. Once the board was populated, I fired up the hot plate and set it to around the solder paste's melting point. When it was up to temperature, I set the board down on it, raised the temperature about 25°C hotter, and waited. Within a minute or so, all the solder was nicely melted, so I backed off the temperature and took the finished board once it cooled down. Finally, I inspected each solder pad with a 30X jeweler's loupe to ensure a good bond. There were several solder bridges that I had to fix with a little heat from my soldering iron and a wick. My first bench test revealed the audio and motors were not working, but nothing popped or smoked, so I ran a bead of solder paste across those components' pads and carefully melted it with my iron. After that, the board tested functional. The only major problem I encountered with the PCB was the unexpected behavior from the DRV8833's nFAULT pin. The chip is supposed to drive the pin low to indicate overcurrent protection, thermal shutdown, or undervoltage lockout. But on the board, at least one of the chips we don't know which because the lines are tied together asserted FAULT as soon as the motors were enabled, and held nFAULT low until the motors slept. And the motors still worked fine during this period. So nFAULT ended up being useless to listen for in the firmware, and I had to disable the code that handled it. On the power-delivery side, though, rev.0 was a clean bill of health: the brownout resets I'd chased on the breadboard never showed up at all, on battery or on an AC adapter. The primary goals of the rev.1 board were: reducing standby deep sleep power consumption and adding more debug diagnostics. For diagnostics, I added a Vmotor sense divider connected to an ADC channel on the ESP32-S3, to allow me to measure and log motor voltage. I added numerous test points to allow me to more easily probe critical nets. For power consumption, the rev.0 board, measured at 6V input: idle wakeword mode fluctuated 32–44 mA; deep sleep was steady at 5.7 mA. Orders of magnitude greater than my 40–50µA deep sleep target. I switched the buck regulator yet again to the AP63201, with its 22µA ultra-low quiescent current. I added FETs to turn off the photocell and Vmotor divider during deep sleep, to save some current, just like we already have for the status LED. I got rid of all the LM74700-Q1 ideal diodes and replaced them with more purpose-built circuitry. The battery input LM74700-Q1 was replaced with a passive P-FET reverse polarity circuit, which feeds the Vmotor rail. The AC adapter input goes straight to the Vmotor rail, no need for a diode or RP protection, since the connector itself has polarity and it also acts as a switch. At least on my Billy Bass, inserting the barrel plug turns off the battery supply, so no need to diode-or them. I replaced the USB power input ideal diode with a simple Schottky, since we don't care about USB voltage because we don't power the motors with it. USB only exists to flash/monitor the device. I also isolated the USB output from the motor output, so that an accidental activation of the motors would never be powered from the USB port. Finally, the input to the 3.3V voltage converter still needed an ideal diode to switch between onboard power and USB power, but I exchanged that LM74700-Q1 with an Analog Devices LTC4412 + FET to reduce that circuit to 22µA quiescent current. To ease assembly, I switched all 0402 sized parts to at least 0603 sizes. To help diagnose the DRV8833 nFAULT issue, I split each DRV8833's nSLEEP and nFAULT pairs into independent GPIOs so they could be controlled and read independently. This gave me the side benefit of being able to only energize the motor controller needed. The tail motor controller now gets turned on only when the tail flaps, then it turns off and we turn on the head/mouth motor controller for the "speaking" part of the turn. Finally, I replaced the huge 1000µF bulk motor cap with two 220µF caps, one for each motor. 1000µF was way overkill and the physical size of the cap was unwieldy and interfered with my ability to physically inspect the rest of the board under magnification. Once the boards arrived: deep sleep power dropped to 342–377µA from rev.0's 5.7mA, a ~15x win, finally in the neighborhood of my original target . However, it turned out I'd shipped the exact same undiagnosed nFAULT bug onto the new board, eventually root-caused to a dumb one-line fix: the tail driver's unused second H-bridge left its sense/ground pin floating instead of tied to GND like the datasheet actually calls for. As long as we are ignoring motor faults in the firmware, this isn't a huge problem. The current schematic/board is labeled rev.2 and resolves this problem, but I never built rev.2. The final board is an exact replacement fit for the original. You'd have no idea it can talk back to you. Here's a video of it in action: All source code, schematics, and PCB layout can be found on GitHub https://github.com/ryandrake08/billy Here are some things I didn't know how to do before I started this project: I had a GPU sitting in my homelab and toyed around with ollama like everyone else, but I never put together an "AI stack," selected local applications and LLM models based on my application and VRAM capacity, and properly passed it through so it was usable in an LXC container. Local AI is fun and setting it up was pretty exciting. I've never trained a model before, either, nor worked with TFLite. The wakeword engineering was pretty much voodoo to me and admittedly still is because I just had the LLM do it. This is my first time working with the ESP32 chip and developing for it, and I have to say it is an absolute pleasure. I've done small "embedded" Raspberry Pi projects that banged on GPIOs but they all seemed so overkill. It's good to have some experience working with something a step down from the Pi. I'd like to experiment with other microcontrollers now. I have never worked with applications that captured audio before, especially from a MEMS microphone that needed a decent acoustic environment. Getting good sound quality into that fish toy was very satisfying. I've used Eagle before, but KiCad was a tough learning cliff. I feel like I am proficient, now, but it took a lot of trial and error. I have done PCB layout hobby projects before, but have never been constrained to a specific PCB form factor like I purposely did to myself in this project. Good learning experience. I have never put ideal diode controllers or MOSFETs to use before either, good tools in my toolbelt. Finally, I had always built projects with thru-hole circuitry. This was my first time working with SMD technology and it's great. I'll never make another dinosaur thru-hole board again. Some of the things I would have done differently if starting over: I believe a lot of the circuitry is overkill and over-specified. I would have tried to cut down on the number of distinct components on the BOM and/or buy resistors and capacitors in bulk. Next project, I'll just buy an SMD component kit from eBay and draw from that rather than buying 1 capacitor at a time from DigiKey. I would have taken a closer look at the existing board and used the circuitry there to better size components, especially the bulk motor capacitors, whose sizes I'm still just guessing. I might have been able to forego the fancy motor controllers. I'm really not taking much advantage of them. I would have added more test points and debug flexibility to my rev.0 design. I was hoping this would be a "one and done" board spin, and I was very wrong. I might have experimented with different PCB manufacturers. I went with OSHPARK because they are familiar, and I've used them in the past, but they are slooooooowwww. Finally, if I really could go back in time, I'd have paid more attention in my undergrad Electrical Engineering classes. I kind of stopped paying attention to them when I decided on a software career, but today I regret it and the knowledge gap that came from that lack of attention.