Your First Voice AI App Shouldn’t Start With a WebSocket

Smallest.ai recommends developers start voice AI projects with batch HTTP requests rather than WebSockets, proving transcript value before adding streaming complexity. The company's Pulse speech-to-text model achieves roughly 64 ms time-to-first-transcript in streaming mode, but a simple batch prototype using the pre-recorded STT API endpoint provides a faster path to evaluating accuracy, speaker diarization, and latency for production use.

The first successful voice feature usually looks unimpressive: one audio file goes in, and one useful transcript comes out. That is not a toy result. It is the shortest route to the questions that decide whether a voice product will work. Does the transcript preserve names and numbers? Are speaker turns separated? Are timestamps useful downstream? Can the system handle the audio your users will actually produce? It is tempting to begin with live microphones, persistent connections, partial transcripts, browser permissions, and animated interfaces. Those pieces feel like voice AI. They also make failures harder to isolate. A safer rule is simple: prove the transcript’s value first, then earn the complexity of streaming. Start with the product’s clock The important architectural choice is not Python versus another language. It is batch versus real time. If you are processing a voicemail, podcast, meeting recording, or archive, the entire file already exists. An HTTP request matches the problem: send the audio, wait for the result, and store the transcript. There is no benefit in pretending a completed file is a live conversation. A voice assistant, live-captioning tool, or call workflow runs on a different clock. It needs partial results while the speaker is talking. The system must receive small audio frames, interpret interim hypotheses, decide when an utterance is final, and recover when the connection drops. That is where a WebSocket belongs. Smallest.ai publishes a roughly 64 ms time-to-first-transcript figure for Pulse streaming. Treat that as a provider specification, not a guarantee for your application: network path, audio framing, region, load, and transcript-stability requirements all affect what users experience. The Pulse speech-to-text product page is the current product reference, while your own p50 and p95 traces should make the architecture decision. The smallest useful Python proof A batch prototype should stay intentionally boring. Put the API key in an environment variable, read a representative audio file, request only the metadata you need, and inspect the returned JSON. python import os import requests endpoint = "https://api.smallest.ai/waves/v1/stt/" params = { "model": "pulse", "language": "en", "word timestamps": "true", "diarize": "true", } headers = { "Authorization": f"Bearer {os.environ 'SMALLEST API KEY' }", "Content-Type": "application/octet-stream", } with open "meeting.wav", "rb" as audio: response = requests.post endpoint, params=params, headers=headers, data=audio, timeout=120, response.raise for status result = response.json print result "transcription" print result.get "words", :5 This uses the current unified endpoint documented in the pre-recorded STT API reference. In production, also handle timeouts, retries, rate limits, response-schema changes, and request identifiers. Do not log the authorization header or raw audio by default. That short loop gives you a useful evaluation surface. Test several accents and speaking styles. Use clean and noisy recordings. Include telephony audio if production will receive it. Check the JSON your next component must consume, not only the sentence printed to the terminal. Measure the transcript you need, not the demo you want A single successful file proves connectivity. A small evaluation set begins to prove usefulness. Use audio captured from the microphones, codecs, channels, and environments your users will have. Record failures involving names, numbers, abbreviations, product terms, and code-switching—not only average text quality. Inspect speaker labels and timestamp drift if downstream features depend on them. Track request latency at p50 and p95; one fast example says little about production behavior. Define what happens when the API times out, rejects a file, or returns an empty transcript. This stage also tells you whether streaming is necessary. If users upload recordings and return later, a persistent connection may add operational cost without improving the product. If the transcript drives a live response, batch processing will eventually reveal its limit. Streaming is not a faster POST request Moving to streaming changes the application, not merely the transport. A live client sends raw audio frames and receives partial and final transcript messages while the connection remains open. Sending and receiving must happen concurrently. Batch waits for the complete recording; streaming turns a moving signal into usable partial results. python import asyncio import json import os from urllib.parse import urlencode import websockets CHUNK SIZE = 4096 params = { "language": "en", "encoding": "linear16", "sample rate": "16000", "word timestamps": "true", } url = "wss://api.smallest.ai/waves/v1/stt/live?model=pulse&" + urlencode params headers = {"Authorization": f"Bearer {os.environ 'SMALLEST API KEY' }"} async def send audio ws, path: str : with open path, "rb" as audio: while chunk := audio.read CHUNK SIZE : await ws.send chunk await asyncio.sleep 0.05 await ws.send json.dumps {"type": "close stream"} async def receive transcripts ws : async for message in ws: event = json.loads message label = "final" if event.get "is final" else "partial" print label, event.get "transcript", "" if event.get "is last" : return async def transcribe file path: str : async with websockets.connect url, additional headers=headers, as ws: sender = asyncio.create task send audio ws, path receiver = asyncio.create task receive transcripts ws await asyncio.gather sender, receiver asyncio.run transcribe file "audio.pcm" The current real-time WebSocket documentation recommends 4096-byte chunks and a close stream control message when a one-off stream is complete. The sample above expects headerless, 16-bit linear PCM at 16 kHz mono. A WAV file normally includes a container header, so do not rename a WAV file to .pcm and assume the bytes are equivalent. The concurrency model matters more than the socket A production microphone flow should separate audio capture, network sending, transcript receiving, and UI updates. If one task blocks the others, buffers grow and the interface feels stale even when recognition is fast. Audio producer: reads fixed-size frames and applies bounded buffering. Network sender: writes frames, handles backpressure, and stops cleanly. Transcript receiver: distinguishes revisable partial text from final text. Application state: stores only stable transcript segments and exposes connection status. Recovery path: reconnects deliberately and prevents duplicate or out-of-order segments. Partial transcripts are a preview, not an append-only log. Replace the current partial segment when a new hypothesis arrives; persist it only when the server marks it final. Otherwise, readers see duplicated words and downstream systems process text that the recognizer later corrected. Keep the browser simple—and the key off it A browser can capture microphone audio and render transcript updates, but it should not receive a long-lived service API key. Put the authenticated upstream connection behind your server. The browser connects to your application, and your server authorizes the user, opens the provider connection, applies limits, and forwards only the events the interface needs. That boundary gives you a place to enforce session duration, input format, concurrency, logging policy, and abuse controls. It also creates a clean progression: file uploads can use a normal server route, while microphone audio uses a streaming proxy. Batch and live transcription remain two deliberate modes instead of one over-engineered pipeline. A transcript is rarely the final product Raw text is enough for a connectivity demo. Production workflows usually need structure. Word timestamps support captions and review tools. Speaker diarization separates participants. Redaction can reduce sensitive information passed downstream. Each feature should exist because a downstream workflow needs it, not because a parameter is available. If your product depends on speaker separation, read the speaker diarization implementation guide before choosing a transcript schema. For conversational systems, the STT–LLM–TTS latency-budget guide explains why a fast recognition stage does not guarantee a fast audible response. Know when the prototype has earned real time Move from batch to streaming when at least one requirement cannot be met by waiting for a complete file: The user needs words on screen while speaking. A conversational system must begin reasoning before the entire recording exists. Turn-taking, interruption, or live routing depends on partial results. The recording is long enough that upload-then-process creates unacceptable delay. When you cross that boundary, measure the complete path: audio capture, frame queueing, network transit, first usable partial, finalization, downstream processing, and—if the system speaks—first audible response. Optimizing only the model’s headline latency can hide the stage users are actually waiting on. Build outward from one trustworthy transcript A voice product does not become serious when it opens a WebSocket. It becomes serious when every added layer solves a problem the simpler version exposed. Start with representative audio. Produce a transcript. Verify the structure and failure modes. Add streaming only when the user’s clock demands it. Then keep the API key behind your server and measure the whole path under realistic load. If you want to test the same batch-to-streaming path with Smallest.ai, open the self-serve application and create an API key—no demo booking required. The complete Python implementation guide provides the broader walkthrough. What requirement in your product genuinely forces the architecture to become real time?