Beyond Generative LLMs: Architecting Sub-10ms Voice AI with Felona Voice

The promise of conversational AI has always been a natural, instantaneous interaction. Yet, for many developers building voice agents, the reality has often been a frustrating battle against latency, unpredictable responses, and escalating cloud costs. We've largely relied on large language models (LLMs) to power these interactions, leveraging their incredible generative capabilities. But what if the very architecture of generative LLMs is fundamentally misaligned with the requirements of real-time, structured voice interactions?

Enter Felona Voice, an open-source, ultra-low-latency voice agent framework for TypeScript that dares to challenge this paradigm. It redefines how we build voice AI by moving beyond the auto-regressive token generation of LLMs, introducing a groundbreaking architecture powered by Joint Embedding Vectors (JEV) and stateful conversational transition graphs (VoiceGraph).

Traditional voice agents often follow a simple, yet costly and slow, loop:

While LLMs excel at open-ended conversations and creative text generation, this architecture introduces several critical challenges for structured voice agents:

These issues highlight an architectural mismatch: we're using a powerful, general-purpose text generator for a task that often requires precise intent recognition and deterministic action execution.

Felona Voice tackles these challenges head-on by fundamentally re-thinking the core of voice agent intelligence. Instead of generating a response from scratch, Felona Voice focuses on matching user intent to predefined actions using Joint Embedding Vectors (JEV).

Here's how it works:

VoiceGraph provides the crucial context. It's a stateful conversational transition graph that guides the agent. Based on the current state and the highest JEV similarity match, Felona Voice deterministically decides the next action. This architecture delivers mind-blowing performance:

Felona Voice is designed with developers in mind, offering a fluent builder API in TypeScript:

import { createAgent } from "felona-voice";

// Define your voice agent
const agent = createAgent("Concierge")
  .system("You are an intelligent voice concierge. Your goal is to assist users with hotel-related requests.")
  .action(
    "book_table",
    "Book a restaurant reservation; I want to reserve a table; Can I get a booking?",
    async (ctx) => {
      console.log("User wants to book a table.");
      // In a real app, you'd integrate with a booking system
      return "Certainly, I can help you book a table. What time and for how many people?";
    }
  )
  .action(
    "request_room_service",
    "Order room service; I need food delivered to my room; Can I get something to eat?",
    async (ctx) => {
      console.log("User wants room service.");
      return "Of course, what would you like to order from room service?";
    }
  )
  .fallback("I'm sorry, I didn't quite catch that. How can I assist you with your hotel stay?");

// Simulate user interaction
async function runInteraction() {
  console.log("Agent: " + (await agent.interact("Hello, I'd like to reserve a table for dinner.")));
  console.log("Agent: " + (await agent.interact("I need some food delivered to my room.")));
  console.log("Agent: " + (await agent.interact("Tell me a joke."))); // This should hit fallback
}

runInteraction();

This API allows you to define actions, their descriptions (which form the basis for JEV matching), and their associated logic. Felona Voice also offers:

This architectural shift translates directly into tangible benefits for your bottom line and user experience:

Let's consider an enterprise-scale voice agent handling 100,000 conversational turns per month. If each LLM inference for intent resolution costs, conservatively, $0.04:

With Felona Voice, the core intent resolution using JEV is an in-memory operation, incurring $0.00 in API fees. While you still pay for ASR and TTS services (which Felona Voice integrates with), the most expensive and slowest part of the LLM loop—the generative inference—is eliminated for structured intent handling.

This means that for the intent resolution component, you achieve a 100% cost saving. When factoring in the overall cost of ASR, TTS, and intent resolution, Felona Voice can easily reduce your monthly infrastructure bills by 90-95% compared to a heavily LLM-dependent architecture, especially for high-volume, structured interactions.

This architecture is a game-changer for applications requiring:

Felona Voice represents a significant leap forward in voice AI. By recognizing the architectural limitations of generative LLMs for structured, real-time interactions and instead leveraging the power of Joint Embedding Vectors and deterministic VoiceGraphs, it delivers unparalleled speed, cost efficiency, and reliability.

It's time to build voice agents that truly feel natural and responsive, without breaking the bank or sacrificing control. Join the movement towards a more efficient and effective voice AI future.

🌟 Star the repository on GitHub: github.com/mohitjoer/felona_voice

📦 Install via npm: npm install felona-voice

📖 Explore full documentation: felona-voice.mohitjoe.tech/docs

This article was originally published on felona-voice.mohitjoe.tech.