Experimental firmware for an AI-Thinker ESP32 Audio Kit V2.2 voice-agent handset with a typical total device hardware cost under US$20, excluding the phone/tablet and bridge computer. The included demo sends live microphone audio and conversation data to a free full-duplex voice API endpoint operated by a company in China, which may raise privacy and data-residency concerns. Users can instead self-host MiniCPM-o 4.5 or adapt the bridge to another realtime voice API, including services offered by Western providers; review any provider's privacy and retention terms before sending sensitive audio.
The board pairs with a phone or tablet as a Bluetooth HFP Hands-Free Unit, allowing a compatible mobile app to exchange bidirectional audio with a voice agent. It sends that audio over WebRTC to a Go bridge on the local network; the bridge owns the voice API WebSocket session and an embedded STUN/TURN server.
The ESP32 becomes the mobile device's Bluetooth handset while WebRTC carries bidirectional audio over the local 2.4 GHz network. The Go bridge handles signaling, realtime voice API conversion, browser monitoring, and relational SQLite telemetry.
- Advertises as
ESP32 Voice Agent
using Bluetooth Classic HFP. - Advertises the Bluetooth audio/telephony hands-free device class so iOS or Android can offer it as a communication input and output.
- Uses Secure Simple Pairing (Just Works); legacy PIN is
0000
. - Starts a local WebRTC session and requests SCO whenever the phone/tablet connects; a cellular call is not required.
- Also answers an incoming cellular call after 1.8 seconds.
- Uses decoded CVSD (8 kHz) PCM from ESP-IDF. Wideband mSBC is disabled because WebRTC is already 8 kHz and the extra coexistence load destabilizes Wi-Fi.
- Carries 8 kHz PCMU audio to and from the laptop using DTLS-SRTP.
- Bundles three 20 ms PCMU frames into each 60 ms downlink RTP packet and suppresses idle RTP to reduce Wi-Fi/Bluetooth coexistence pressure.
- Negotiates RTCP NACK and uses a hardware-tested ESP receive/jitter budget to recover short UDP loss bursts without turning them into long playback stalls.
- The Go bridge converts PCMU to MiniCPM's 16 kHz input and converts MiniCPM's 24 kHz output back to PCMU.
- Sends silence back to the app while the model or network is not ready.
- Closes WebRTC and the laptop's MiniCPM session when Bluetooth disconnects.
- Optionally shows Wi-Fi, Bluetooth, SCO, call, and agent status on a 128x64
SSD1306 I2C OLED at address
0x3C
(SDA GPIO21, SCL GPIO22).
This first version automatically answers every incoming call. Use it only for controlled testing. Caller allow-lists, physical enable/disable controls, and production privacy controls have not been added yet.
The hardware-tested target is the AI-Thinker ESP32 Audio Kit V2.2 with an ESP32-A1S module, Classic Bluetooth, PSRAM, and an attached 2.4 GHz IPEX/U.FL antenna. Boards without Classic Bluetooth or PSRAM are not equivalent targets. The current firmware stores its persistent status buffer in PSRAM and will fail initialization if that allocation is unavailable.
Typical total device hardware cost: under US$20, including the ESP32-A1S board, 2.4 GHz antenna, OLED, and an optional small LiPo battery. This estimate does not include the phone/tablet or the computer running the bridge.
The original ESP32 supports only 2.4 GHz Wi-Fi. A 5 GHz-only network will not appear during scanning. Because Wi-Fi and Bluetooth share the same 2.4 GHz radio, place the access point reasonably close to the board during testing.
Attach a matching 2.4 GHz IPEX/U.FL-style antenna before normal Wi-Fi and Bluetooth use. Press the connector straight down, avoid sideways force, and strain-relieve the coax. Some board revisions have an antenna-selection link or zero-ohm resistor; confirm that it selects the IPEX connector before use.
The board has separate UART
and POWER
Micro-USB ports:
- Use the
UART
port with a known data-capable cable for flashing and serial monitoring. On the tested board it also supplies 5 V power. - The
POWER
port is power-only and does not expose the USB-to-UART adapter. - Avoid powering the board simultaneously through multiple USB ports and the battery connector unless the exact board revision's power-path documentation confirms that configuration.
The display is optional. The default driver expects a generic 128x64 SSD1306
I2C OLED at address 0x3C
:
| OLED pin | ESP32 Audio Kit connection |
|---|---|
GND |
|
GND |
|
VCC |
|
3V3 |
|
SDA |
|
GPIO21 |
|
SCL |
|
GPIO22 |
Power the OLED from 3.3 V even if its listing claims 5 V compatibility; ESP32 GPIO is not 5 V tolerant. Module pin order varies, so follow the labels printed on the actual OLED rather than relying on connector position. Wire it only while the board is powered off.
Use only a protected single-cell LiPo/Li-ion battery: 3.7 V nominal and 4.2 V maximum. Verify the connector polarity against the board's battery and ground markings with a multimeter before insertion. Prewired XH2.54 leads are sold in both polarities, so do not trust wire color or plug orientation. Never connect 5 V or a multi-cell pack to the battery input. Charging and load-sharing behavior varies by board revision; confirm it from the matching schematic and do not leave an initial charging test unattended.
The onboard ES8388 microphone/speaker path is not used by this application. Voice audio travels directly through Bluetooth HFP/SCO.
- PlatformIO Core 6.1.19 or newer
- PlatformIO Espressif 32 platform 6.12.0
- ESP-IDF 5.5.0
espressif/esp_websocket_client
1.5.0- Espressif
esp_peer
1.5.0 fromesp-webrtc-solution
revision7850d1c
- Go 1.24 or newer
The versions are pinned by platformio.ini
, main/idf_component.yml
,
bridge/go.mod
, and the WebRTC revision in scripts/bootstrap_webrtc.sh
.
PlatformIO's downloaded toolchain is kept under .pio-core/
and ignored by
Git. The generated dependencies.lock
is also ignored because the local
esp_peer
override records an absolute checkout path on each machine.
Fetch the ignored Espressif source checkout before building firmware:
scripts/bootstrap_webrtc.sh
Copy the template and edit the new local file:
cp main/credentials.example.h main/credentials.local.h
Set APP_WIFI_SSID
, APP_WIFI_PASSWORD
, and APP_BRIDGE_SIGNAL_URL
. The local credentials file is ignored by Git.
Without Wi-Fi and a reachable bridge, the firmware still boots and pairs but the mobile app receives silence from the Bluetooth microphone route.
Copy bridge/config.example.json
to the ignored
bridge/config.local.json
. Set bind_ip
to the laptop's active LAN address,
choose a private TURN password, and update APP_BRIDGE_SIGNAL_URL
to the same
address. telemetry_db
selects the local SQLite history file. Then run:
cd bridge
go test ./...
go build -o esp32-voice-agent-bridge .
./esp32-voice-agent-bridge -config config.local.json
The bridge is a foreground process. Stopping its terminal—or closing the app that launched it—stops signaling, TURN, and MiniCPM. Run the compiled binary from a persistent terminal or install it as an operating-system service for unattended use.
The default ports are HTTP/WebSocket 8080
, STUN/TURN UDP 3478
, TURN relay
UDP 50000-50100
, and direct WebRTC UDP 50101-50200
. Health and live counters
are available at /healthz
and /status
. The generic bilingual system prompt
is stored in bridge/system-prompt.txt
.
The bridge currently has no application-layer authentication. Its monitor can expose live audio and model text, and its reset/test controls change the active session. Bind it only to a trusted LAN interface, use a private TURN password, and never forward these ports directly to the public internet.
The bridge keeps full ICE for STUN/TURN support, but uses continuous ESP RTP as its authoritative media-liveness signal. This avoids an interoperability issue where the ESP peer misses Pion consent responses despite healthy media. Eight seconds without uplink RTP forces a clean signaling and peer restart.
Open /monitor
in a browser on the same network to see connection state, live RTP counters, the latest model text, and separate level meters for mobile app audio entering the agent and agent audio sent as the Bluetooth microphone. Optional playback for either direction is off by default; use headphones before enabling it to avoid an acoustic feedback loop.
The bridge also records telemetry asynchronously to bridge/telemetry.sqlite3
by default. A dedicated writer goroutine batches WAL-mode SQLite transactions;
the WebRTC path only makes non-blocking channel sends. Each input/output audio
frame stores its timestamp, direction, sample count, peak, and RMS, while status
events retain the same timestamps but store counters and state in typed columns
in telemetry_status
, joined through event_id
. New status rows do not duplicate JSON payloads. Database files and compressed history archives are ignored by Git.
The offline migration mode imports a legacy JSON-history database without
putting JSON blobs back into the active database. It preserves exact bridge-run
timestamps, optional status fields, status-level session presence, ordered ICE
candidate arrays, and the distinction between SQL NULL
, an empty-string
payload sentinel, and a valid migrated status object. Existing event IDs are
never reassigned; imported IDs are recorded in telemetry_imported_events
.
Stop the bridge and serial monitor before migration. A check-only pass accepts
either an uncompressed SQLite file or its .gz
archive:
cd bridge
go build -o esp32-voice-agent-bridge .
./esp32-voice-agent-bridge \
-config config.local.json \
-migrate-telemetry-from legacy-telemetry.sqlite3.gz \
-migration-source-name legacy-telemetry \
-check-only
Apply only after check-only succeeds:
./esp32-voice-agent-bridge \
-config config.local.json \
-migrate-telemetry-from legacy-telemetry.sqlite3.gz \
-migration-source-name legacy-telemetry \
-apply-migration
Apply mode verifies both databases, creates a compressed consistent backup of
the active database, imports and verifies everything in one transaction, then
checkpoints and integrity-checks the merged database. The source archive and
backup are retained. A decompressed temporary source is deleted only after a
successful verification. Reapplying the same source checksum performs a full
verification and imports zero rows.
The optional minicpm.inference
object is forwarded to MiniCPM's
session.init
payload. For a conversational voice agent, a listen_prob_scale
below the default 1.0
makes the duplex model more eager to speak; 0.65
is the current hardware-tested starting point.
MiniCPM output is buffered by response because its one-second audio deltas can
arrive slower than realtime. Short responses start when MiniCPM emits listen
; long responses start after six seconds are buffered, with a 12-second failsafe. Playback remains in 60 ms RTP packets while browser and SQLite audio telemetry retain their original 20 ms frame granularity. For example:
sqlite3 -header -column bridge/telemetry.sqlite3 \
"SELECT captured_at, direction, audio_peak, audio_rms
FROM telemetry_events WHERE kind='audio' ORDER BY id DESC LIMIT 30;"
For an end-to-end mobile-app audio check, send POST /test-tone
. The bridge queues a two-second 880 Hz tone through WebRTC and Bluetooth HFP. Another participant receiving the app's microphone audio should hear it; the tablet itself normally will not monitor its own microphone.
POST /test-tone?seconds=60
sends a sustained pulse/silence pattern for UDP
loss diagnosis. For a deterministic speech test that bypasses MiniCPM, send
one to 60 seconds of raw 8 kHz mono PCMU to POST /test-audio
. The body must be aligned to complete 160-byte (20 ms) frames; a 60-second fixture is exactly 480,000 bytes:
curl -X POST --data-binary @speech-60s.pcmu \
http://BRIDGE_LAN_IP:8080/test-audio
The hardware-validated active-SCO run delivered all 1,000 bundled RTP packets,
all 3,000 source frames, and all 480,000 decoded samples without PCM drops or
buffer overruns. See docs/debug-notes.md
for the full result and tuning rationale.
Connect the board's Micro-USB UART
port, then run:
pio run
pio run --target upload
pio device monitor
If upload auto-reset fails, hold BOOT
, tap RST
, release BOOT
, and retry
the upload. Do not hold BOOT
during normal audio operation because GPIO0 is also used by the Audio Kit hardware.
- Boot the board and open the serial monitor.
- If the phone/tablet cached an older pairing, forget
ESP32 Voice Agent
first. - Pair
ESP32 Voice Agent
and enable itsPhone calls,** Input device**, or equivalent communication-audio option. - Open the mobile voice app and select the Bluetooth headset route if the app exposes an audio-device picker.
- Watch for WebRTC state 7, agent state
listening
, and nonzeroSCO PCM rx=... B/s tx=... B/s
messages in the serial log.
Ordinary media playback is not a valid test. The firmware provides HFP/SCO for two-way speech, not an A2DP media sink. An app that only uses Android's media stream may continue playing through the tablet speaker and may not expose its microphone to this device.
Compatible mobile app <-> iOS/Android HFP/SCO <-> ESP32-A1S
|
WebRTC/PCMU
|
Go bridge + TURN
|
MiniCPM-o 4.5 WSS
The ES8388 codec and wired speaker are not in the voice-agent path. HFP audio comes directly from the ESP32 Bluetooth controller through Voice over HCI.
- The iOS/Android app must support a Bluetooth communication/headset route. The ESP32 cannot force an app that deliberately uses only media audio or the tablet's built-in microphone to switch routes.
- A2DP media playback is not implemented; HFP is used because the voice agent needs bidirectional audio.
- One ESP32 session is supported at a time.
- The laptop bridge must remain running on the configured LAN address.
- The ESP32 supports 2.4 GHz Wi-Fi only. Wi-Fi and Bluetooth share that radio; weak Wi-Fi can still cause call-audio underruns outside the tested conditions.
- The MiniCPM public audio session currently has a 600-second limit.
- MiniCPM can generate one second of output audio several seconds apart. The bridge rebuffers those responses, but it cannot make a slower-than-realtime model source both continuous and low-latency.
- WebRTC is narrowband PCMU because it matches CVSD HFP and keeps the original ESP32's CPU and memory use predictable.
- The firmware relies on ESP-IDF's legacy decoded-PCM HFP callbacks. The code is pinned to ESP-IDF 5.5.x because Espressif plans to remove that path.
- The default prompt forbids claims of being human, but server/model behavior should be verified before conversations with real people.