Artificial Intelligence #
Usage-based metering for bidirectional streaming solves a software billing problem for real-time speech AI. Customers deploying streaming models on Amazon SageMaker AI bidirectional streaming pay for the endpoint infrastructure that hosts the model. Separately, model providers listing on AWS Marketplace need a way to charge for what customers consume during a live stream. Today, we’re announcing usage-based metering for Amazon SageMaker bidirectional (bidi) streaming inference, so model providers can report actual usage for the AWS Marketplace software charge. Amazon SageMaker AI endpoint infrastructure charges still apply.
Deepgram, the real-time speech AI company, is the launch partner for this capability, bringing usage-based Speech-to-Text (STT) and Text-to-Speech (TTS) billing to AWS Marketplace. This post builds on a series of innovations between Deepgram and AWS. First, we introduced bidirectional streaming for real-time inference on Amazon SageMaker AI, which supports long-lived bidirectional streams between clients and model servers. Then, Deepgram enhanced Amazon SageMaker AI support with AWS Identity and Access Management (AWS IAM) temporary delegation, simplifying secure authentication for enterprise deployments, and expanded observability with enhanced metrics. Now, with usage-based metering, we further strengthen the foundation for production real-time AI, aligning billing with how customers consume streaming and real-time AI workloads.
Why streaming broke the existing billing model #
With AWS Marketplace software metering for SageMaker AI real-time request-response inference, model providers can report a unit count in an HTTP response header. AWS Marketplace uses that quantity for the software charge. This works for request-response APIs where the billable quantity is known when the response is sent.
Bidirectional streaming is different. A client streams audio and receives transcriptions through the SageMaker AI API using an HTTP/2 event stream. SageMaker AI connects to the model server over WebSockets. A session can last 30 seconds or 30 minutes. Usage accumulates as the session runs, and the final total isn’t known until it ends. A response header alone can’t report periodic usage during that live session.
Hourly software pricing can charge customers while a model is available, including quiet periods. With usage-based metering, providers can align the AWS Marketplace software charge with reported usage. SageMaker AI endpoint infrastructure charges remain separate and continue to apply while the endpoint is provisioned.
How companion metadata streaming works #
The solution introduces a second WebSocket connection alongside the existing inference stream. When a bidirectional streaming session starts, SageMaker AI establishes two connections to the model server:
- Data connection : carries inference request and response data, the existing bidirectional stream.
- Metadata connection : a companion WebSocket dedicated to metering data.
Model servers opt in by returning one header on the WebSocket upgrade response:
X-Amzn-SageMaker-Metadata-Stream-Supported: true
SageMaker AI then opens a companion WebSocket to the model server at /invocations-bidirectional-stream-metadata. Both connections carry the same X-Amzn-SageMaker-Request-Id header, which the model server uses to correlate metering records with the active inference session.
The model server sends metering data as JSON text frames on the metadata connection. Send records at a frequency that matches the billing model: incrementally as usage accrues (recommended), either as each unit is consumed or as periodic batches of units consumed since the previous record. Send any final record as soon as the session ends. Each frame contains exactly one metering record, rather than an array of records. The metadata stream has a rate quota of one message per second per connection, with a maximum message size of 512 bytes.
We recommend sending metering records incrementally as usage accrues. This reduces the amount of usage left unreported if a session ends unexpectedly. Transport delivery alone does not establish that usage has been billed. Validate reported units against AWS Marketplace reports and buyer charges.
Metering messages use the inference.count dimension, and ConsumedUnits supplies the quantity for that dimension. If the product charges per second of audio, a record reporting 12 minutes of metered audio carries 720 seconds as ConsumedUnits: 720. The dimension name doesn’t mean one unit per request. The provider defines which audio counts as metered usage, and the AWS Marketplace offer maps the unit to its price. The listing shows one price per unit, labeled per request, with up to three decimal places. The ClientToken field supports idempotency. If a duplicate token arrives on the same connection, SageMaker AI drops the duplicate message silently.
The metadata connection runs over TCP, which provides ordered, reliable delivery at the transport layer. If it fails to establish or drops mid-session, inference continues by default because X-Amzn-SageMaker-Metadata-Stream-Required defaults to false. Set this header to true when the provider must stop inference if the metadata stream becomes unavailable. This choice trades inference availability for metering continuity. It doesn’t guarantee that malformed or dropped records are billed.
What this means for customers #
If you deploy real-time AI models for transcription, contact centers, voice agents, or captioning, the new metering model changes how the AWS Marketplace software charge is calculated. SageMaker AI endpoint infrastructure charges still apply. Transparent, per-unit billing through AWS Marketplace means usage flows directly into your AWS bill. You get consolidated AWS billing, negotiated rates through private offers, and no separate billing relationship with the model provider to manage.
Pay for audio processed according to the product’s unit definition. If the product charges per second of audio, 12 minutes of metered audio equals 720 billable seconds and ConsumedUnits: 720. An open connection lasting 12 minutes doesn’t, by itself, define the billable audio quantity. The provider’s counting rules determine the metered usage, whereas endpoint infrastructure charges remain separate.
Model providers can offer negotiated per-unit rates or fixed-price contracts through AWS Marketplace private offers.
We recommend discussing directly with your AWS account representative or partner representative.
What this means for model providers #
If you’re a model provider listing on AWS Marketplace, you can configure your model server to report usage during streaming inference sessions through companion metadata streaming.
The SageMaker Marketplace Onboarding Agent Skill accelerates onboarding with an opt-in inference metering path. Its starter templates advertise metadata-stream support and build validated, incremental ConsumedUnits frames for the companion WebSocket. Review and production-harden the code before submission.
Align billing with your pricing model by defining what one unit represents and reporting units as the session progresses. AWS Marketplace charges one per-unit price for real-time inference.
The following three-step walkthrough describes how to integrate your model server to report metering data during bidirectional streaming inference sessions.
Step 1: Opt in using data WebSocket response headers
When SageMaker AI opens the data WebSocket connection to your model server, include these headers in your WebSocket upgrade (101) response:
| | Header | Required | Values | Description |
| 1 | X-Amzn-SageMaker-Metadata-Stream-Supported | Yes (to opt in) | TRUE | Signals that your model server supports the companion metadata stream. If absent or not true , no metadata connection is attempted. Without the metadata stream, the session is not metered for the model provider’s software charge. |
| 2 | X-Amzn-SageMaker-Metadata-Stream-Required | No | true /false | This header only takes effect when X-Amzn-SageMaker-Metadata-Stream-Supported istrue . If true, inference stops when the metadata connection fails. If false or absent, inference continues in data-only mode. Default: false. Choose true when metering continuity takes priority over inference availability. This setting doesn’t guarantee billing for malformed or dropped records. |
| 3 | X-Amzn-SageMaker-Metadata-Invocation-Path | No | Custom path string | Overrides the default metadata WebSocket path. Default: /invocations-bidirectional-stream-metadata . |
Step 2: Accept the metadata WebSocket connection
After your model server returns X-Amzn-SageMaker-Metadata-Stream-Supported: true, SageMaker AI opens a second WebSocket connection to your model server at the default path /invocations-bidirectional-stream-metadata (or the custom path you specified by using X-Amzn-SageMaker-Metadata-Invocation-Path).
SageMaker AI includes the header X-Amzn-SageMaker-Request-Id on both the data and metadata WebSocket handshakes. Your model server must do the following:
- Accept the WebSocket upgrade on the metadata path.
- Use
X-Amzn-SageMaker-Request-Idto correlate the metadata connection with the corresponding data connection for the same inference session.
Step 3: Send metering messages
Send metering data as a JSON object in a WebSocket text frame on the metadata connection. Each frame must contain exactly one metering record. In this example, the product charges per second of audio, and the record reports 12 minutes of metered audio as 720 units:
Message fields:
| | Field | Required | Type | Constraints | Description |
| 1 | Metering | Yes | Object | — | Container for the metering record. |
| 2 | Metering.Dimension | Yes | String | Must be inference.count | The billing dimension. Must be inference.count . |
| 3 | Metering.ConsumedUnits | Yes | Number | 0 or greater | The units consumed since your previous record. Each record is billed separately. A value of 0 is accepted but not billed. |
| 4 | Metering.ClientToken | No | String | Max 64 characters | A unique token (for example, a UUID) for idempotency. If the same ClientToken is sent multiple times on the same connection, only the first occurrence is recorded. |
Send metering records incrementally as usage accrues to reduce unreported usage if a session ends unexpectedly. Reconcile the reported quantities with AWS Marketplace reports and buyer charges to verify billing, including interrupted-session tests.
Constraints and limits
| | Limit | Value | Notes | | 1 | Maximum message size | 512 bytes | Total size of the JSON text frame | | 2 | Maximum send rate | 1 message per second per connection | Exceeding this rate might result in dropped messages | | 3 | Maximum Dimension length | 128 characters | — | | 4 | Maximum ClientToken length | 64 characters | — | | 5 | Messages per frame | Exactly 1 | Don’t batch multiple metering records in a single frame | | 6 | Frame type | Text frames only | Binary frames are not supported for metering |
Error handling and failure modes
The Required header controls what happens when the metadata connection becomes unavailable. Set it to true to stop inference, or use the default false to let inference continue in data-only mode. This is a choice between metering continuity and inference availability. Malformed messages can still be dropped with either setting.
| | Scenario | Required = true | Required = false (default) | | 1 | Metadata WebSocket fails to connect | Data stream is terminated | Data stream continues normally (data-only mode) | | 2 | Metadata WebSocket | | |