From Soundwaves to Sleep Health: Building a Local AI Snoring & Apnea Detector with Whisper and TensorFlow.js

A developer built a fully local, browser-based sleep monitoring system that uses Whisper.cpp compiled to WebAssembly to filter out speech for privacy and a TensorFlow.js CNN to classify Mel Spectrograms of breathing sounds as snoring, apnea, or normal. All audio is processed client-side at 16kHz via the Web Audio API, with no recordings leaving the device, and anomalous events are logged to local storage.

Have you ever wondered if your "heroic" snoring is actually a sign of something more serious? Sleep apnea affects millions, yet most diagnostic tools involve bulky wires in a lab or sending your bedroom recordings to a mysterious cloud server. 😱 Today, we’re building a 100% local, privacy-first Sleep Monitoring System . By leveraging Edge Computing , we’ll transform raw audio into Mel Spectrograms, use Whisper.cpp to filter out speech for privacy, and deploy a TensorFlow.js CNN to detect abnormal breathing patterns—all inside the browser. We will dive deep into Edge Computing , Digital Signal Processing DSP , and real-time audio classification . If you're looking for advanced production patterns for edge AI, you should definitely check out the deep dives over at WellAlly Tech Blog https://www.wellally.tech/blog , which inspired the architecture for this build. The biggest hurdle with audio monitoring in the bedroom is privacy. We solve this by processing everything on the client side. No audio ever leaves the device. php graph TD A Web Audio API -- |Stream PCM Data| B Audio Buffer B -- C{Privacy Filter: Whisper.cpp} C -- |Speech Detected| D Discard Data/Mute C -- |No Speech| E DSP Engine: Mel Spectrogram E -- F CNN Model: TensorFlow.js F -- |Output| G Real-time Dashboard F -- |Anomalous Event| H Local Storage Log We need a clean stream of 16kHz mono audio the standard for most audio ML models . js // Initializing the audio context for 16kHz const audioContext = new window.AudioContext || window.webkitAudioContext { sampleRate: 16000, } ; const stream = await navigator.mediaDevices.getUserMedia { audio: true } ; const source = audioContext.createMediaStreamSource stream ; const processor = audioContext.createScriptProcessor 4096, 1, 1 ; source.connect processor ; processor.connect audioContext.destination ; processor.onaudioprocess = e = { const inputData = e.inputBuffer.getChannelData 0 ; // Pipe this to our processing pipeline processAudioChunk inputData ; }; Before analyzing breathing, we must ensure we aren't recording private conversations. We use Whisper.cpp compiled to WebAssembly. If the model detects a high probability of speech tokens, we discard the buffer immediately. js import { Whisper } from 'whisper-wasm'; const whisper = await Whisper.load 'base-en-q5 1.bin' ; async function processAudioChunk buffer { const result = await whisper.transcribe buffer ; // Logic: If words are detected, it's speech - Ignore if result.text.trim .length 0 { console.log "Speech detected. Redacting for privacy... 🤐" ; return; } // Proceed to Snoring/Apnea analysis analyzeBreathingPattern buffer ; } CNNs are great at processing images. By converting 1D audio waves into a 2D Mel Spectrogram , we treat the "sound of a snore" as a visual pattern. function computeMelSpectrogram audioBuffer { // 1. Apply Hann Window // 2. Compute Fast Fourier Transform FFT // 3. Map to Mel Scale Logarithmic frequency // 4. Return as a Float32Array Image-like tensor const tensor = tf.browser.fromPixels spectrogramCanvas ; return tensor.div 255.0 .expandDims 0 ; } We use a lightweight CNN trained on the "AudioSet" and custom snoring datasets. It looks for the rhythmic signature of snoring vs. the silence/gasping signature of Obstructive Sleep Apnea OSA . js async function analyzeBreathingPattern buffer { const model = await tf.loadLayersModel '/models/sleep-cnn/model.json' ; const spectrogram = computeMelSpectrogram buffer ; const prediction = model.predict spectrogram ; const snore, apnea, normal = await prediction.data ; if apnea 0.8 { triggerAlert "Potential Apnea Event Detected 🚨" ; } else if snore 0.7 { updateDashboard "Snoring activity detected. 💤" ; } } Building for the edge requires a different mindset than traditional cloud-based AI. You have to balance memory constraints, battery life on mobile , and inference speed. For more production-ready examples and advanced patterns on optimizing WASM-based AI models, I highly recommend checking out the technical guides at WellAlly Tech Blog https://www.wellally.tech/blog . They have some incredible resources on deploying high-performance models in resource-constrained environments that helped me optimize the FFT calculations for this project. By combining the Web Audio API with Whisper.cpp and TensorFlow.js , we’ve built a powerful, diagnostic-grade tool that respects user privacy. What’s next? Are you ready to stop sending your data to the cloud and start processing it on the edge? Let me know in the comments if you’ve tried running Whisper in the browser 👇