{"slug": "building-ultra-low-latency-voice-ai-applications-on-aws", "title": "Building ultra-low-latency voice AI applications on AWS", "summary": "Cartesia, an AWS Partner specializing in real-time voice AI, is using purpose-built State Space Models (SSMs) and AWS infrastructure to deliver sub-100ms text-to-speech and speech-to-text at global scale, according to an AWS machine-learning blog post. Cartesia's SSM-based approach, deployed in its Sonic speech-generation model and Ink transcription model, avoids the quadratic compute growth of transformer attention and maintains a constant memory footprint with native token-by-token streaming. The company argues this architecture resolves the quality-versus-latency trade-off that limits transformer-based text-to-speech in contact centers, telehealth, gaming and accessibility applications.", "body_md": "## [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Building ultra-low-latency voice AI applications on AWS\n\nConversational AI is transforming how humans interact with technology. From virtual assistants and AI-powered contact centers to real-time translation and accessibility tools, voice-first interfaces are rapidly becoming the default mode of interaction for millions of users worldwide.\n\nYet delivering a truly natural voice experience remains one of the hardest challenges in AI infrastructure. Users expect human-like responsiveness. Excess latency can break conversational flow and erode trust. To feel natural, AI-generated speech must arrive within the same temporal window as human turn-taking.\n\nThis post introduces how [Cartesia](https://www.cartesia.ai/), an AWS Partner specializing in real-time voice AI, uses purpose-built State Space Models (SSMs) and AWS infrastructure to deliver sub-100ms text-to-speech and speech-to-text at global scale. Developers can build voice experiences that feel indistinguishable from human conversation.\n\n## The rise of conversational AI and latency expectations\n\nThe demand for real-time voice AI is growing rapidly across industries:\n\n- **Contact centers:** AI agents handling millions of customer interactions require instantaneous speech generation to maintain natural dialogue.\n- **Telehealth:** Remote patient consultations demand low-latency voice for accurate, empathetic communication.\n- **Gaming and virtual worlds:** Immersive experiences require real-time voice synthesis for non-player characters (NPCs) and player interactions.\n- **Accessibility:** Screen readers and voice interfaces must respond instantly to serve users with visual impairments.\n- **Inbound and outbound sales:** AI sales agents must respond without awkward pauses to qualify leads, handle objections, and keep prospects on the line.\n\nTraditional transformer-based text-to-speech (TTS) models struggle with streaming latency because they process entire sequences before producing output. This architectural limitation creates a fundamental tension: higher quality often means higher latency. A new class of models, State Space Models, resolves this trade-off.\n\n## AI for voice: State Space Models (SSM)\n\nState Space Models (SSMs) take a fundamentally different approach to processing sequences than transformers, the architecture behind most of today’s AI. Transformers use a mechanism called attention, which compares every element in a sequence to every other element. As a result, the amount of computation grows quadratically as the sequence gets longer. SSMs instead treat a sequence as a system that evolves continuously over time, processing it step by step and carrying a compact internal state forward. This can be converted into a form computers handle efficiently, keeping the cost low for certain long sequences. This efficiency makes SSMs well suited to real-time, streaming data like live speech.\n\nKey advantages of SSMs for audio processing include:\n\n- High-quality long-range dependencies: Structured SSMs are a form of linear recurrence, but their state parameterization preserves information over long audio sequences, avoiding the long-range signal decay that limits classic recurrent neural networks (RNNs) such as long short-term memory (LSTM) networks.\n- Constant memory footprint: Unlike transformers that cache all previous tokens, SSMs maintain a fixed-size hidden state.\n- Native streaming support: The recurrent structure processes input token-by-token, so it supports true streaming without architectural modifications.\n\nThese properties make SSMs uniquely suited for real-time voice applications where both quality and latency are non-negotiable.\n\n## How it works: SSMs for real-time voice\n\nCartesia puts these SSM advantages to work in two production voice models: Sonic for speech generation and Ink for transcription.\n\n### Introducing Cartesia\n\nCartesia is a real-time multimodal intelligence company that has built a voice AI application, purpose-built on State Space Model architecture. Founded by researchers who pioneered SSM advances, Cartesia’s application is designed for streaming inference at the edge of human perception.\n\nThe application features two core models, Sonic for text-to-speech and Ink for speech-to-text, both optimized for sub-100ms first-byte latency while maintaining studio-quality output.\n\n### Sonic (text-to-speech)\n\nSonic is Cartesia’s flagship streaming text-to-speech model, delivering ultra-realistic speech synthesis with sub-100ms latency:\n\n- **First audio byte in approximately 90ms:** Users hear the voice respond almost instantaneously.\n- **Ultra-realistic voice quality:** Natural prosody, emotion, and expressiveness across over 44 languages.\n- **Voice cloning with minimal data:** Create natural-sounding custom voices from 10-second audio samples.\n- **Streaming-native architecture:** Audio begins generating before the full text input is received.\n- **Context-aware synthesis:** Maintains consistent tone and style across long passages.\n\nSonic is recommended for teams that want the fastest time-to-production with minimal operational overhead. Call the API and receive streaming audio in real time.\n\n### Ink (speech-to-text)\n\nInk is Cartesia’s streaming speech-to-text model, purpose-built for conversational AI applications:\n\n- **Native turn detection:** Automatically identifies when a speaker has finished their turn, alleviating awkward pauses without a separate voice activity detection (VAD) or end pointing model.\n- **Context-aware end pointing:** Uses semantic understanding to distinguish between mid-sentence pauses and true conversation boundaries.\n- **Streaming transcription:** Emits partial transcripts in real time as the user speaks, so a downstream large language model (LLM) can begin forming its response before the turn ends.\n- **Noise robustness:** Maintains accuracy in noisy environments common in real-world deployments.\n- **Multi-speaker support:** Handles overlapping speech and speaker transitions gracefully.\n\nTogether, Sonic and Ink provide the complete voice AI stack needed to build conversational agents that feel human.\n\n## Deploying Cartesia on AWS\n\nIf you need data sovereignty, latency control, or compliance, you can deploy Cartesia models directly into your own AWS account. Sonic is delivered as an Amazon SageMaker AI model package through AWS Marketplace, which you deploy to a SageMaker AI endpoint running entirely within your virtual private cloud (VPC). Inference data doesn’t leave your account.\n\nThe deployment patterns in this section cover Sonic. Ink is currently available through the Cartesia API, which you can purchase through the [Cartesia Voice AI Platform](https://aws.amazon.com/marketplace/pp/prodview-bcf67mlwzvkce) listing in AWS Marketplace. Two primary patterns are available for Sonic.\n\n### Option 1: Amazon SageMaker AI (AWS Marketplace model package)\n\nDeploying on Amazon SageMaker AI is the fastest path to a managed, in-VPC Sonic endpoint. Sonic ships as an AWS Marketplace model package, so there’s a one-time subscribe step that grants your account an AWS Region-specific model-package ARN. You then create the SageMaker AI endpoint from that ARN, either from the console or with the SageMaker or Boto3 SDK. The following steps follow Cartesia’s official deployment notebook.\n\n#### Prerequisites\n\nRun from an Amazon SageMaker Notebook Instance or SageMaker Studio.\n\nThe AWS Identity and Access Management (IAM) execution role needs `AmazonSageMakerFullAccess`.\n\nTo subscribe, your role (and you) need `aws-marketplace:Subscribe`, `aws-marketplace:Unsubscribe`, and `aws-marketplace:ViewSubscriptions`. Otherwise, the account must already be subscribed to the listing.\n\nBefore deployment, choose a supported instance type from the model package’s `SupportedRealtimeInferenceInstanceTypes` `field:`\n\n- `ml.g6e.xlarge` or larger\n- `ml.g7e.2xlarge` or larger\n- `ml.g7.2xlarge` or larger\n- `ml.p5.48xlarge`\n- `ml.p5en.48xlarge`\n\nIf your chosen instance type’s applied value is below your current usage plus your planned endpoint instance count, you may need to request a service quota increase. See [SageMaker AI quotas](https://docs.aws.amazon.com/general/latest/gr/sagemaker.html#limits_sagemaker) for details.\n\n#### Deploy Sonic 3.6 to a SageMaker endpoint\n\nIn Amazon SageMaker JumpStart, you can subscribe to its AWS Marketplace model package, create a SageMaker model, and host it on a real-time endpoint. The following four steps show the deployment path.\n\n1. In SageMaker AI, open **AWS Marketplace resources** , then**All products** . Search for**Sonic 3.6 SageMaker** and choose the Cartesia listing.\n2. Choose **View purchase options** , review the pricing and terms, and subscribe. The subscription grants your AWS account access to the model package. It doesn’t deploy resources.\n3. Open **Deployments & inference > Deployable models** , choose**Create model** , select**Use a model package subscription from AWS Marketplace** , and select**Sonic 3.6 SageMaker** . Choose an execution role, then create the model.\n4. Create a real-time endpoint from the model with a supported GPU instance, such as `ml.g6e.2xlarge` . The endpoint first appears with**Creating** status. Wait for**InService** before invoking it.\n\nThe model-package ARN is Region-specific. To verify the Sonic 3.6 package in US East (N. Virginia) with the SageMaker API, run:\n\nAfter the endpoint reaches **InService**, invoke it with the SageMaker Runtime `InvokeEndpointWithResponseStream` API and process the returned audio stream in your application.\n\n### Option 2: Self-managed on EC2 GPU instances\n\nSome teams have existing GPU infrastructure, strict networking requirements, or a need to co-locate Cartesia with other services. For these cases, you can run the model container yourself on Amazon Elastic Compute Cloud (Amazon EC2) GPU instances. This gives you full control over the runtime, scaling, and network path. It uses Cartesia’s container image, obtained from Cartesia, rather than the AWS Marketplace model package.\n\n**Obtain the container image.** Request the Sonic container image and licensing terms from Cartesia. Replicate it into your own private Amazon Elastic Container Registry (Amazon ECR) repository so your compute plane can pull it with IAM-scoped credentials rather than reaching out to an external registry at scale.\n\n**Choose your GPU instance family.** Sonic runs on NVIDIA GPU instances. The G6 (L4) and G6e (L40S) families are the optimal balance for real-time TTS. The earlier G5 (A10G) also works. Match GPU memory and vCPU to your target concurrency and prefer a single-GPU instance per model replica for simplest scaling.\n\n**Provision the compute plane (Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS)).** For ECS: register a GPU task definition (set the container’s GPU resource requirement, for example, `resourceRequirements` with type GPU) and run it as a service behind an Application Load Balancer. For EKS: create a GPU node group (or Karpenter provisioner) using the EKS GPU-optimized AMI, install the NVIDIA device plugin, and deploy the container as a Deployment fronted by a Service/Ingress.\n\n**Configure networking for low latency.** Place inference nodes in a cluster placement group to minimize inter-node latency, keep replicas in a single Availability Zone for the tightest path, and use VPC endpoints for ECR/S3 pulls. For hybrid or on-premises callers, terminate traffic over AWS Direct Connect or a private Network Load Balancer to avoid public-internet hops.\n\n**Load model weights efficiently.** Store model weights on a shared, low-latency volume (Amazon Elastic File System (Amazon EFS) or Amazon FSx for Lustre) mounted across replicas so new pods/tasks start without each re-downloading multi-GB artifacts. Warm the model at container start and expose a health check so the load balancer only routes to ready replicas.\n\n**Tune batching and concurrency.** Set the container’s max batch size and concurrent-stream limits to match your traffic profile. Higher batch size improves GPU utilization but adds first-byte latency, so tune against your sub-100ms target. Benchmark `streams-per-GPU` to find the replica capacity you will scale on.\n\n**Implement custom auto scaling.** Scale on a signal that reflects real load, active audio-stream count or GPU utilization, rather than CPU. On Amazon ECS, use AWS Application Auto Scaling with a custom Amazon CloudWatch metric. On Amazon EKS, use HPA/KEDA driven by a GPU or stream-count metric, with Karpenter (or Cluster Autoscaler) adding GPU nodes as replicas grow.\n\n**Add observability and resiliency.** Emit latency (time-to-first-byte), stream counts, and GPU metrics to CloudWatch. Run at least two replicas across nodes for availability. Set rolling-update or draining policies so deployments don’t drop in-flight audio streams.\n\nThis approach is recommended when integrating Cartesia into an existing machine learning (ML) system, meeting networking or compliance requirements not available through managed services, or optimizing cost with Reserved or Spot GPU capacity. The trade-off is that you own provisioning, scaling, patching, and monitoring that SageMaker would otherwise handle for you.\n\n**Learn more:** See [Amazon ECS task definitions for GPU workloads](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-gpu.html), [managing NVIDIA GPU devices on Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/device-management-nvidia.html), and [getting started with GPU-accelerated EC2 instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/gpu-instances-started.html).\n\n### Choosing the right deployment pattern\n\nAWS offers two ways to run Sonic, each suited to a different starting point.\n\nThe managed path, using the Amazon SageMaker AI model package from AWS Marketplace, is the default for most teams: you subscribe, deploy to an in-VPC endpoint in minutes, and let SageMaker handle scaling and operations.\n\nThe self-managed path, running the model container yourself on EC2 GPU instances, is designed for teams with specific constraints: existing GPU infrastructure, strict networking requirements, or a need to co-locate Cartesia with other services. For these teams, full control over the runtime, scaling, and network path matters more than time to deployment.\n\nThe following table summarizes how the two compare.\n\n| **Criteria** | **SageMaker (Marketplace)** | **EC2 GPU (Self-Managed)** | \n| Time to Deployment | Minutes after subscribing | Hours to days | \n| Compute | Real-time endpoint. See `SupportedRealtimeInferenceInstanceTypes` for supported instances. | EC2 GPU instances | \n| Operational Control | Managed by AWS