{"slug": "giving-voice-agents-a-voice-that-adapts-to-context", "title": "Giving voice agents a voice that adapts to context", "summary": "Gradium's Voice Design tool lets developers generate natural text-to-speech voices from a text description, and a pipeline pairing it with TypeSafe AI's Jev decision model can classify an incoming message and design a matching voice on demand, Gradium reports. Jev classifies a message against developer-defined labels for about $0.00002 per request in the demo, with requests taking 240 to 850 milliseconds including the network round trip, and costs $0.042 per million input tokens on OpenRouter with output free. The approach targets cases a fixed voice catalog handles poorly, such as procedurally generated game characters, scene-changing narrators, and support agents that shift tone, though each Voice Design voice consumes one custom-voice slot shared with voice clones.", "body_md": "# Giving voice agents a voice that adapts to context\n\nMost voice APIs come either with a library of predefined voice presets or the ability to create a custom voice from an audio sample.\n\nWith [Voice Design](https://docs.gradium.ai/guides/voices/voice-design), you can now create natural, realistic voices from a text description (or \"prompt\"). Voice characteristics become output your code computes, from the same context it already uses to decide what the agent says.\n\nReading that context has also become inexpensive. A new class of decision models, such as [Jev](https://docs.typesafe.ai) from TypeSafe AI, classifies a message against labels you define for about $0.00002 per request (in the following demo), with no training. This opens use cases that a fixed catalog of voices cannot serve well:\n\n- a game that gives every procedurally generated character a voice of its own,\n- an interactive story whose narrator changes with the scene,\n- a support agent that answers inquiries with an appropriate tone.\n\nConsider this latter case. You are building an automated voice agent for the customer support line. Most requests are routine, like asking for an address change. Now one ticket arrives: *\"I have been charged twice and nobody has answered my emails.\"*\n\nA single voice answers both in the same tone, whether the caller wants a quick confirmation or needs reassurance. You could map each type of request to a prebuilt voice, but you would have to anticipate every case in advance, for every language the line supports, and find a preset with the right delivery for each. With Voice Design, you describe the delivery you need, such as \"calm and reassuring, warm rounded resonance\", and the agent creates the voices it is missing in the background as new cases appear.\n\nIn this article we build an end-to-end pipeline for that agent:\n\n1. Extract a few signals from an incoming message with Jev\n2. Map those signals to a Voice Design description\n3. Resolve the description to a voice the agent can speak with *now* , and design new voices in the background.\n4. Synthesize the reply with Gradium TTS.\n\nYou can follow along in the [live demo](https://gradium.ai/demos/voice-design-demo/) (bring your own [Gradium API key](https://studio.gradium.ai/platform/api-keys)), or run it yourself from the [GitHub code repository](https://github.com/gradium-ai/examples).\n\nPrerequisites:\n\n- Node.js 18+\n- Access to Jev, either directly through [TypeSafe AI](https://docs.typesafe.ai) or via[OpenRouter](https://openrouter.ai) (without a Jev key, a simple keyword classifier stands in)\n- A [Gradium API key](https://studio.gradium.ai/platform/api-keys) . Each voice converted from Voice Design uses one custom-voice slot on your account, shared with voice clones, so check your allowance before generating a library.\n\nNote: the live demo is a simplified version of the pipeline described below. It creates each voice the first time it's needed and reuses it afterwards, so the first reply in a new context takes a few extra seconds. The full pipeline avoids that wait by preparing voices ahead of time.\n\n## The pipeline\n\nThe solid path runs on every turn and never calls Voice Design. The dotted path runs outside the conversation and grows the library over time.\n\n## Step 1: Extract context with a decision model\n\nThe first step turns an incoming message into a few structured signals. This is a classification task, and decision models such as [Jev](https://docs.typesafe.ai) are particularly well suited to it: they answer typed questions about an input and return a probability for each option and a confidence score. Three properties make it a good fit for a voice pipeline:\n\n- Fast. In our tests, a request took between 240 and 850 milliseconds including the network round trip. That is short enough to run on every incoming message.\n- Cheap. Jev costs $0.042 per million input tokens on [OpenRouter](https://openrouter.ai/typesafe/jev-1.13) , and output is free.\n- No training required. The labels and their descriptions are written in plain language in the request. Changing a label set only means editing an object in your code.\n\nJev is available through OpenRouter as `typesafe/jev-1.13`. Because the request carries an API key, this code runs on the server. Tone and energy become two choice questions sent in a single request:\n\n``` js\nconst QUESTIONS = {\n  tone: {\n    type: \"choice\",\n    instructions: \"How does the writer of this message feel?\",\n    criteria: {\n      frustrated: \"Annoyed, upset, or dissatisfied.\",\n      neutral: \"Matter-of-fact, no strong emotion.\",\n      enthusiastic: \"Excited, happy, or eager.\",\n    },\n  },\n  energy: {\n    type: \"choice\",\n    instructions: \"How urgent is this request?\",\n    criteria: {\n      relaxed: \"Can be handled at a normal pace.\",\n      urgent: \"Needs prompt attention.\",\n    },\n  },\n};\n\nconst DEFAULTS = { tone: \"neutral\", energy: \"relaxed\" };\nconst MIN_CONFIDENCE = 0.6;\n\nexport async function extractContext(message, { language, register }) {\n  const res = await fetch(\"https://openrouter.ai/api/alpha/decisions\", {\n    method: \"POST\",\n    headers: {\n      Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,\n      \"Content-Type\": \"application/json\",\n    },\n    body: JSON.stringify({ model: \"typesafe/jev-1.13\", state: message, questions: QUESTIONS }),\n  });\n  if (!res.ok) throw new Error(`Jev request failed: ${res.status}`);\n  const { answers } = await res.json();\n\n  const context = { language, register };\n  for (const [field, answer] of Object.entries(answers)) {\n    context[field] = answer.confidence >= MIN_CONFIDENCE ? answer.choice : DEFAULTS[field];\n  }\n  return context;\n}\n```\n\nFor a support line, the message *\"I have been charged twice and nobody has answered my emails.\"* comes back as:\n\n```\n{\n  \"language\": \"en\",\n  \"register\": \"formal\",\n  \"tone\": \"frustrated\",\n  \"energy\": \"urgent\"\n}\n```\n\nThe three example messages resolve as expected, with the same results over repeated runs (values in parentheses are the probabilities of the selected options):\n\n| Message | tone | energy | \n|---|---|---|\n| \"I have been charged twice and nobody has answered my emails.\" | frustrated (1.00) | urgent (1.00) | \n| \"Hello, I moved last month and would like to update the delivery address on my account.\" | neutral (1.00) | relaxed (0.99) | \n| \"Thank you so much for your support!!!\" | enthusiastic (1.00) | relaxed (1.00) | \n\nThree rules to keep the extraction reliable:\n\n- Keep the label sets small and closed. Every value has to map to a voice later, so each new label multiplies the number of voices you need.\n- Fall back to a default below a confidence threshold. Jev reports a confidence score separate from the option probabilities. When it is low, a safe default such as \"neutral\" is better than a guess.\n- Prefer known properties over inferred ones. `language` and`register` are passed in rather than classified: the language comes from the user's account settings, and the register from the product, since a support line answers formally even when the customer writes casually.\n\nNote that the classifier here could be replaced by local models if messages cannot leave the browser or your infrastructure.\n\n## Step 2: Turn signals into a voice description\n\nWe recommend describing a voice with concrete attributes: gender, age range, accent, pitch, pace, energy, timbre, manner, and intended use. Some of those should vary with context, but not all. Split the description in two:\n\n- **Persona** : who the agent is. Gender, age, accent. Chosen by the product, fixed per language.\n- **Delivery** : how the agent sounds in this interaction. Pace, energy, manner, resonance. Derived from context.\n\n``` js\nconst PERSONA = {\n  en: \"A female voice, 30 to 40, neutral American accent, mid pitch\",\n  fr: \"A female voice, 30 to 40, Parisian accent, mid pitch\",\n  de: \"A female voice, 30 to 40, standard German accent, mid pitch\",\n};\n\nconst DELIVERY = {\n  tone: {\n    frustrated: \"calm and reassuring, warm rounded resonance\",\n    neutral: \"clear and direct, balanced resonance\",\n    enthusiastic: \"bright and upbeat, lively resonance\",\n  },\n  energy: {\n    relaxed: \"unhurried pacing, medium energy\",\n    urgent: \"steady, efficient pacing, focused energy\",\n  },\n  register: {\n    formal: \"Ideal for a professional support line.\",\n    casual: \"Ideal for a friendly in-app assistant.\",\n  },\n};\n\nexport function voicePrompt({ language, register, tone, energy }) {\n  const persona = PERSONA[language];\n  if (!persona) throw new Error(`No persona for language \"${language}\"`);\n\n  const prompt =\n    `${persona}: ${DELIVERY.tone[tone]}, ${DELIVERY.energy[energy]}. ` +\n    DELIVERY.register[register];\n\n  if (prompt.length > 500) throw new Error(\"Voice description exceeds 500 characters\");\n  return prompt;\n}\n```\n\nFor the double-charge message, this produces:\n\n*\"A female voice, 30 to 40, neutral American accent, mid pitch: calm and reassuring, warm rounded resonance, steady, efficient pacing, focused energy. Ideal for a professional support line.\"*\n\nNotice how the agent's voice **responds to** the user's tone instead of mirroring it: a frustrated customer gets a calm voice, and an urgent request gets efficient pacing rather than rushed delivery. The mapping encodes what the agent should do about the user's state, which is a product decision you can review in code.\n\nAnother important property is **that the user's message never reaches the prompt**; only values from closed lists do. That makes every possible prompt enumerable, and guarantees that nothing a user types can steer the voice beyond the options you allowed.\n\n## Step 3: Resolve a voice without waiting for one\n\nVoice Design doesn't return a production voice directly. It returns *candidates* you audition before converting one to a permanent `voice_id` that can be used for TTS.\n\nSo the context-to-voice pipeline needs to create a library of approved voices keyed by spec, and a resolver that always returns one of them immediately.\n\nThis is why the spec space is deliberately left small: 3 personas x 2 registers x 3 tones x 2 energy levels = 36 combinations (60 if you cover all five languages Voice Design supports), and in practice you'd approve only the ones your product uses. That's few enough to design and audition by hand before launch.\n\nFor combinations you didn't anticipate, the resolver picks the closest approved voice in the same language and flags the miss:\n\n``` js\nconst WEIGHTS = { tone: 2, energy: 1, register: 1 };\n\nexport const specKey = (s) => `${s.language}:${s.register}:${s.tone}:${s.energy}`;\n\nexport function resolveVoice(spec, library) {\n  const exact = library.get(specKey(spec));\n  if (exact) return { voiceId: exact.voiceId, exact: true };\n\n  let best = null;\n  for (const entry of library.values()) {\n    if (entry.spec.language !== spec.language) continue;\n    const distance = Object.entries(WEIGHTS)\n      .reduce((d, [field, w]) => d + (entry.spec[field] === spec[field] ? 0 : w), 0);\n    if (!best || distance < best.distance) best = { ...entry, distance };\n  }\n  if (!best) throw new Error(`No approved voice for \"${spec.language}\"`);\n  return { voiceId: best.voiceId, exact: false };\n}\n```\n\nTone carries more weight than energy or register because getting it wrong is the most audible mistake: a bright voice answering a complaint is worse than a slightly slow one answering an urgent question.\n\nThe per-turn handler never awaits Voice Design:\n\n```\nexport function voiceForTurn(spec, library, designQueue) {\n  const { voiceId, exact } = resolveVoice(spec, library);\n  if (!exact) designQueue.add(spec); // designed later\n  return voiceId;\n}\n```\n\n### Designing voices in the background\n\nThe specs added to `designQueue` are processed by a worker: a background job, separate from the conversation, that takes one spec at a time and turns it into an approved voice. For each spec, the worker generates candidate voices from the spec's prompt, waits until they are ready, then auditions them and converts the one that passes. The first two steps look like this:\n\n``` js\nconst BASE = \"https://api.gradium.ai/api\";\nconst headers = {\n  \"x-api-key\": process.env.GRADIUM_API_KEY,\n  \"Content-Type\": \"application/json\",\n};\n\nasync function generateCandidates(spec, n = 3) {\n  const res = await fetch(`${BASE}/voice-generator/generate`, {\n    method: \"POST\",\n    headers,\n    body: JSON.stringify({ prompt: voicePrompt(spec), language: spec.language, n_samples: n }),\n  });\n  if (!res.ok) throw new Error(`generate failed: ${res.status}`);\n  const { embeddings } = await res.json();\n  return embeddings.map((e) => e.embedding_id);\n}\n\nasync function waitUntilReady(id, timeoutMs = 120_000, intervalMs = 500) {\n  const deadline = Date.now() + timeoutMs;\n  while (Date.now() < deadline) {\n    const res = await fetch(`${BASE}/voice-generator/embeddings?embedding_id=${id}`, { headers });\n    const { embeddings } = await res.json();\n    if (embeddings[0]?.ready) return;\n    await new Promise((r) => setTimeout(r, intervalMs));\n  }\n  throw new Error(`Candidate ${id} not ready after ${timeoutMs} ms`);\n}\n```\n\nGeneration typically takes about 3 seconds, so polling every 500 ms adds at most half a second to it. The timeout is required: a request with an out-of-range generation setting still returns 201, but its candidates never become ready. The two-minute bound follows the [Voice Design guide](https://docs.gradium.ai/guides/voices/voice-design).\n\nOnce candidates are ready, audition them on a short line (100 characters max) through the regular TTS endpoint, then convert the one you approve:\n\n``` js\nasync function convert(candidateId, spec) {\n  const res = await fetch(`${BASE}/voices/from-embedding`, {\n    method: \"POST\",\n    headers,\n    body: JSON.stringify({\n      voxium_embedding_id: candidateId,\n      name: specKey(spec),\n      description: voicePrompt(spec),\n    }),\n  });\n  if (!res.ok) throw new Error(`convert failed: ${res.status}`);\n  const { uid } = await res.json();\n  return uid; // permanent voice_id, usable over REST and WebSocket\n}\n```\n\nWhether a human approves the candidate or the worker converts the first one automatically is a product choice. For a support line, listen first, because the generated voice may not match the prompt exactly. For a game spawning hundreds of background characters, auto-conversion may be acceptable. Keep the custom-voice allowance in mind either way.\n\n### When generation latency is fine\n\nGeneration latency is only a problem inside a conversational turn. There are moments where a few seconds go unnoticed:\n\n- Before launch, designing the whole spec space.\n- At session start, when context is known before anyone speaks: a CRM record, a selected language, a character-creation screen. The agent greets in a default voice while the tailored one is prepared.\n- Between sessions, when the queue of cache misses is processed.\n\nThe design is the same in all three cases: voice selection runs on the hot path, voice creation runs wherever a few seconds are free.\n\n## Step 4: Speak\n\nWith a permanent `voice_id`, synthesis is an ordinary TTS request:\n\n``` js\nasync function speak(text, voiceId) {\n  const res = await fetch(`${BASE}/post/speech/tts`, {\n    method: \"POST\",\n    headers,\n    body: JSON.stringify({ text, voice_id: voiceId, output_format: \"wav\", only_audio: true }),\n  });\n  if (!res.ok) throw new Error(`TTS failed: ${res.status}`);\n  return res.arrayBuffer();\n}\n```\n\nA live agent would use the WebSocket endpoint with the same `voice_id`.\n\n### Keep voice changes deliberate\n\nA few rules to keep in mind for a clean context-aware voice:\n\n- **Pin the voice per session** . Resolve once at the start of a conversation, or when something meaningful changes: the user switches language, sets a preference, or the conversation moves from small talk to a billing dispute. An agent that changes its voice every turn sounds like a relay of different people.\n- **Never infer a persona from the user** . The classifier only decides how the agent delivers its replies. Don't derive the agent's gender, age, or accent from what you guess about the user. Persona belongs to the product and, where offered, to the user's explicit settings.\n- **Let explicit preferences win** . If a user has chosen a voice, that choice overrides anything the classifier says.\n\n## Voice is now a function of context\n\nThe pipeline in this article is deliberately simple: one message associated with four signals, which delivers a clean mapping. Real conversations are messier, and a production agent needs more context than a single message and firmer rules about when to change voice. But the core takeaway holds: once a voice can be described in text, your application can decide how it should sound.\n\nA common complaint about voice agents is that they sound robotic. Part of that comes from the voice model, and part comes from using one tone for everything: to confirm an address change and to answer someone who was charged twice.\n\nVoice Design moves that choice into your code. The agent's voice becomes a mapping from context to voice, built from ordinary application logic you can test, review, and change. Adding a language or a new kind of conversation means adding a description, not recording a voice or searching a preset catalog.", "url": "https://wpnews.pro/news/giving-voice-agents-a-voice-that-adapts-to-context", "canonical_source": "https://gradium.ai/blog/voice-that-adapts-to-context", "published_at": "2026-09-29 00:00:00+00:00", "updated_at": "2026-10-20 17:48:14.034972+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-products", "generative-ai"], "entities": ["Gradium", "Voice Design", "TypeSafe AI", "Jev", "OpenRouter", "Gradium TTS", "Node.js"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/giving-voice-agents-a-voice-that-adapts-to-context", "markdown": "https://wpnews.pro/news/giving-voice-agents-a-voice-that-adapts-to-context.md", "text": "https://wpnews.pro/news/giving-voice-agents-a-voice-that-adapts-to-context.txt", "jsonld": "https://wpnews.pro/news/giving-voice-agents-a-voice-that-adapts-to-context.jsonld"}}