{"slug": "llm-0-37-adds-multimodal-embeddings-and-a-cheaper-default-what-that-means-for", "title": "llm 0.37 Adds Multimodal Embeddings and a Cheaper Default — What That Means for Local Audio Pipelines", "summary": "Simon Willison released llm 0.37 on 2 November, adding the gpt-6.1-sol and gpt-6.1-terra models, an `llm openai endpoint add` command for arbitrary OpenAI-compatible endpoints, and a switch of the default model to gpt-6-luna at half the previous default's price. The release also lets the library accept different content-types for binary payloads, enabling multimodal embedding models such as Gemini Embedding 2 to embed text, audio, images and video through one API. The writeup argues the change matters mainly for offline or post-turn retrieval work, since embedding calls measured in hundreds of milliseconds should stay off the hot path of live audio pipelines.", "body_md": "Simon Willison shipped [llm 0.37](https://simonwillison.net/2026/Nov/2/llm/) on 2 November. The release adds `gpt-6.1-sol` and `gpt-6.1-terra`, a new `llm openai endpoint add` command for arbitrary OpenAI-compatible endpoints, and switches the default model to `gpt-6-luna` (half the price of the previous default). The more interesting line for people who actually ship tools is the embeddings change: the library now accepts different content-types for binary payloads, which is required for models like Gemini Embedding 2 that can embed text, audio, images, and video in one API.\n\nThat is a small API surface change. It is also the exact seam where most local real-time systems still fall over.\n\nAn embedding call that accepts raw audio is useful for retrieval, speaker clustering, or “find the similar turn” later. It is almost never the thing you want on the hot path of a live call.\n\nLatency budgets for a dual-track interview or call assistant look roughly like this:\n\nA multimodal embedding model, even a fast one, is usually measured in hundreds of milliseconds once you count serialization, network (or GPU queue), and the fact that you rarely want to embed every 200 ms chunk. You embed the finalized turn, or you embed offline. The streaming path stays lean.\n\nThe practical pattern is the same one that shows up in any real agent loop once tool count or payload size leaves the toy regime: keep the expensive representation out of the critical path until you actually need it.\n\nNone of the new embedding APIs change the fact that a Windows call has two independent audio streams that must stay separate:\n\nMixing them early destroys speaker labels and forces the recognizer to handle two languages or two accents in one beam. The correct architecture is two independent pipelines that only meet at the UI or the final transcript store.\n\nOn Windows the loopback path is still “open an input stream on a render device” via WASAPI. In Rust with cpal it looks like this:\n\n```\n// Simplified — real code also handles format negotiation and device changes\nuse cpal::traits::{DeviceTrait, HostTrait, StreamTrait};\n\nfn open_loopback(host: &cpal::Host, device_name: &str) -> Result<cpal::Stream, anyhow::Error> {\n    let device = host\n        .output_devices()?\n        .find(|d| d.name().ok().as_deref() == Some(device_name))\n        .ok_or_else(|| anyhow::anyhow!(\"device not found\"))?;\n\n    let config = device.default_output_config()?;\n    // cpal treats a render device opened as input as loopback\n    let stream = device.build_input_stream(\n        &config.into(),\n        |data: &[f32], _: &cpal::InputCallbackInfo| {\n            // push to ring buffer owned by the STT session\n            // never block here\n        },\n        |err| eprintln!(\"loopback error: {err}\"),\n        None,\n    )?;\n    stream.play()?;\n    Ok(stream)\n}\n```\n\nThe callback must not allocate, must not lock a mutex that the UI or network thread might hold, and must tolerate the device disappearing mid-call. Those constraints are unchanged by any CLI or embedding update.\n\nShared mode lets other apps keep playing sound; exclusive mode drops latency but steals the device. Most interview helpers stay in shared mode and accept the extra 20–40 ms because the alternative is “your headphones go silent the moment the tool starts.”\n\nOnce the turns are finalized you can do useful offline or near-line work with the same CLI that developers already use for everything else:\n\n```\n# embed a finalized wav snippet (content-type now respected)\nllm embed -m gemini-embedding-2 --binary --content-type audio/wav turn_incoming.wav\n\n# or just chat with a cheaper default while you triage logs\nllm -m gpt-6-luna \"summarize the last 10 turns, mark code-switching\"\n```\n\nThe `llm openai endpoint add` command is the BYOK escape hatch. Point it at a local llama.cpp server, an Ollama endpoint, or a company proxy and the rest of your scripts keep working. That is the same pattern you want inside a desktop app: the STT and translation models are configurable, the capture and overlay code is not.\n\nAI-assisted PRs are mentioned in the release notes. Willison notes that most of the fixes in 0.37 came from contributors using models, and that triage with a stronger model kept the review load manageable. The same dynamic appears in any project that ships frequently: the volume of patches rises, the signal-to-noise depends on how good your local evaluation harness is. For a real-time audio tool the harness is “does the overlay stay under 200 ms end-to-end on a mid-range laptop while both tracks are live.”\n\nThe models keep getting cheaper and more multimodal. The capture graph, the buffer discipline, and the decision of what stays on the hot path do not.\n\nI'm building [Talkaside](https://talkaside.com/), a Windows app for job interviews and calls in another language.\n\n*Drafted with AI assistance from this project's code and logs.*", "url": "https://wpnews.pro/news/llm-0-37-adds-multimodal-embeddings-and-a-cheaper-default-what-that-means-for", "canonical_source": "https://dev.to/madnessbrains_i/llm-037-adds-multimodal-embeddings-and-a-cheaper-default-what-that-means-for-local-audio-5b8i", "published_at": "2026-11-03 03:12:05+00:00", "updated_at": "2026-11-03 03:22:10.309520+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-agents"], "entities": ["Simon Willison", "llm", "OpenAI", "Gemini Embedding 2", "gpt-6-luna", "gpt-6.1-sol", "gpt-6.1-terra", "WASAPI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-0-37-adds-multimodal-embeddings-and-a-cheaper-default-what-that-means-for", "markdown": "https://wpnews.pro/news/llm-0-37-adds-multimodal-embeddings-and-a-cheaper-default-what-that-means-for.md", "text": "https://wpnews.pro/news/llm-0-37-adds-multimodal-embeddings-and-a-cheaper-default-what-that-means-for.txt", "jsonld": "https://wpnews.pro/news/llm-0-37-adds-multimodal-embeddings-and-a-cheaper-default-what-that-means-for.jsonld"}}