From Soundwaves to Sleep Health: Building a Local AI Snoring & Apnea Detector with Whisper and TensorFlow.js

Have you ever wondered if your "heroic" snoring is actually a sign of something more serious? Sleep apnea affects millions, yet most diagnostic tools involve bulky wires in a lab or sending your bedroom recordings to a mysterious cloud server. 😱

Today, we’re building a 100% local, privacy-first Sleep Monitoring System. By leveraging Edge Computing, we’ll transform raw audio into Mel Spectrograms, use Whisper.cpp to filter out speech for privacy, and deploy a TensorFlow.js CNN to detect abnormal breathing patterns—all inside the browser.

We will dive deep into Edge Computing, Digital Signal Processing (DSP), and real-time audio classification. If you're looking for advanced production patterns for edge AI, you should definitely check out the deep dives over at WellAlly Tech Blog, which inspired the architecture for this build.

The biggest hurdle with audio monitoring in the bedroom is privacy. We solve this by processing everything on the client side. No audio ever leaves the device.

graph TD
    A[Web Audio API] -->|Stream PCM Data| B[Audio Buffer]
    B --> C{Privacy Filter: Whisper.cpp}
    C -->|Speech Detected| D[Discard Data/Mute]
    C -->|No Speech| E[DSP Engine: Mel Spectrogram]
    E --> F[CNN Model: TensorFlow.js]
    F -->|Output| G[Real-time Dashboard]
    F -->|Anomalous Event| H[Local Storage Log]

We need a clean stream of 16kHz mono audio (the standard for most audio ML models).

// Initializing the audio context for 16kHz
const audioContext = new (window.AudioContext || window.webkitAudioContext)({
  sampleRate: 16000,
});

const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const source = audioContext.createMediaStreamSource(stream);
const processor = audioContext.createScriptProcessor(4096, 1, 1);

source.connect(processor);
processor.connect(audioContext.destination);

processor.onaudioprocess = (e) => {
  const inputData = e.inputBuffer.getChannelData(0);
  // Pipe this to our processing pipeline
  processAudioChunk(inputData);
};

Before analyzing breathing, we must ensure we aren't recording private conversations. We use Whisper.cpp compiled to WebAssembly. If the model detects a high probability of speech tokens, we discard the buffer immediately.

import { Whisper } from 'whisper-wasm';

const whisper = await Whisper.load('base-en-q5_1.bin');

async function processAudioChunk(buffer) {
    const result = await whisper.transcribe(buffer);

    // Logic: If words are detected, it's speech -> Ignore
    if (result.text.trim().length > 0) {
        console.log("Speech detected. Redacting for privacy... 🤐");
        return;
    }

    // Proceed to Snoring/Apnea analysis
    analyzeBreathingPattern(buffer);
}

CNNs are great at processing images. By converting 1D audio waves into a 2D Mel Spectrogram, we treat the "sound of a snore" as a visual pattern.

function computeMelSpectrogram(audioBuffer) {
  // 1. Apply Hann Window
  // 2. Compute Fast Fourier Transform (FFT)
  // 3. Map to Mel Scale (Logarithmic frequency)
  // 4. Return as a Float32Array (Image-like tensor)
  const tensor = tf.browser.fromPixels(spectrogramCanvas);
  return tensor.div(255.0).expandDims(0);
}

We use a lightweight CNN trained on the "AudioSet" and custom snoring datasets. It looks for the rhythmic signature of snoring vs. the silence/gasping signature of Obstructive Sleep Apnea (OSA).

async function analyzeBreathingPattern(buffer) {
  const model = await tf.loadLayersModel('/models/sleep-cnn/model.json');
  const spectrogram = computeMelSpectrogram(buffer);

  const prediction = model.predict(spectrogram);
  const [snore, apnea, normal] = await prediction.data();

  if (apnea > 0.8) {
    triggerAlert("Potential Apnea Event Detected! 🚨");
  } else if (snore > 0.7) {
    updateDashboard("Snoring activity detected. 💤");
  }
}

Building for the edge requires a different mindset than traditional cloud-based AI. You have to balance memory constraints, battery life (on mobile), and inference speed.

For more production-ready examples and advanced patterns on optimizing WASM-based AI models, I highly recommend checking out the technical guides at WellAlly Tech Blog. They have some incredible resources on deploying high-performance models in resource-constrained environments that helped me optimize the FFT calculations for this project.

By combining the Web Audio API with Whisper.cpp and TensorFlow.js, we’ve built a powerful, diagnostic-grade tool that respects user privacy.

What’s next?

Are you ready to stop sending your data to the cloud and start processing it on the edge? Let me know in the comments if you’ve tried running Whisper in the browser! 👇