llm 0.37 Adds Multimodal Embeddings and a Cheaper Default — What That Means for Local Audio Pipelines

Simon Willison released llm 0.37 on 2 November, adding the gpt-6.1-sol and gpt-6.1-terra models, an `llm openai endpoint add` command for arbitrary OpenAI-compatible endpoints, and a switch of the default model to gpt-6-luna at half the previous default's price. The release also lets the library accept different content-types for binary payloads, enabling multimodal embedding models such as Gemini Embedding 2 to embed text, audio, images and video through one API. The writeup argues the change matters mainly for offline or post-turn retrieval work, since embedding calls measured in hundreds of milliseconds should stay off the hot path of live audio pipelines.

Simon Willison shipped llm 0.37 https://simonwillison.net/2026/Nov/2/llm/ on 2 November. The release adds gpt-6.1-sol and gpt-6.1-terra , a new llm openai endpoint add command for arbitrary OpenAI-compatible endpoints, and switches the default model to gpt-6-luna half the price of the previous default . The more interesting line for people who actually ship tools is the embeddings change: the library now accepts different content-types for binary payloads, which is required for models like Gemini Embedding 2 that can embed text, audio, images, and video in one API. That is a small API surface change. It is also the exact seam where most local real-time systems still fall over. An embedding call that accepts raw audio is useful for retrieval, speaker clustering, or “find the similar turn” later. It is almost never the thing you want on the hot path of a live call. Latency budgets for a dual-track interview or call assistant look roughly like this: A multimodal embedding model, even a fast one, is usually measured in hundreds of milliseconds once you count serialization, network or GPU queue , and the fact that you rarely want to embed every 200 ms chunk. You embed the finalized turn, or you embed offline. The streaming path stays lean. The practical pattern is the same one that shows up in any real agent loop once tool count or payload size leaves the toy regime: keep the expensive representation out of the critical path until you actually need it. None of the new embedding APIs change the fact that a Windows call has two independent audio streams that must stay separate: Mixing them early destroys speaker labels and forces the recognizer to handle two languages or two accents in one beam. The correct architecture is two independent pipelines that only meet at the UI or the final transcript store. On Windows the loopback path is still “open an input stream on a render device” via WASAPI. In Rust with cpal it looks like this: // Simplified — real code also handles format negotiation and device changes use cpal::traits::{DeviceTrait, HostTrait, StreamTrait}; fn open loopback host: &cpal::Host, device name: &str - Result<cpal::Stream, anyhow::Error { let device = host .output devices ? .find |d| d.name .ok .as deref == Some device name .ok or else || anyhow::anyhow "device not found" ?; let config = device.default output config ?; // cpal treats a render device opened as input as loopback let stream = device.build input stream &config.into , |data: & f32 , : &cpal::InputCallbackInfo| { // push to ring buffer owned by the STT session // never block here }, |err| eprintln "loopback error: {err}" , None, ?; stream.play ?; Ok stream } The callback must not allocate, must not lock a mutex that the UI or network thread might hold, and must tolerate the device disappearing mid-call. Those constraints are unchanged by any CLI or embedding update. Shared mode lets other apps keep playing sound; exclusive mode drops latency but steals the device. Most interview helpers stay in shared mode and accept the extra 20–40 ms because the alternative is “your headphones go silent the moment the tool starts.” Once the turns are finalized you can do useful offline or near-line work with the same CLI that developers already use for everything else: embed a finalized wav snippet content-type now respected llm embed -m gemini-embedding-2 --binary --content-type audio/wav turn incoming.wav or just chat with a cheaper default while you triage logs llm -m gpt-6-luna "summarize the last 10 turns, mark code-switching" The llm openai endpoint add command is the BYOK escape hatch. Point it at a local llama.cpp server, an Ollama endpoint, or a company proxy and the rest of your scripts keep working. That is the same pattern you want inside a desktop app: the STT and translation models are configurable, the capture and overlay code is not. AI-assisted PRs are mentioned in the release notes. Willison notes that most of the fixes in 0.37 came from contributors using models, and that triage with a stronger model kept the review load manageable. The same dynamic appears in any project that ships frequently: the volume of patches rises, the signal-to-noise depends on how good your local evaluation harness is. For a real-time audio tool the harness is “does the overlay stay under 200 ms end-to-end on a mid-range laptop while both tracks are live.” The models keep getting cheaper and more multimodal. The capture graph, the buffer discipline, and the decision of what stays on the hot path do not. I'm building Talkaside https://talkaside.com/ , a Windows app for job interviews and calls in another language. Drafted with AI assistance from this project's code and logs.