{"slug": "build-a-model-improvement-flywheel-with-amazon-sagemaker-hyperpod-data-capture", "title": "Build a model improvement flywheel with Amazon SageMaker HyperPod Data Capture", "summary": "Amazon SageMaker HyperPod added Data Capture, a built-in capability that records live inference inputs and outputs to Amazon Simple Storage Service (Amazon S3) at three independent tiers with minimal latency impact and zero impact on model availability, AWS announced. The three tiers capture full request/response payloads at the SageMaker endpoint, Application Load Balancer access-log fields, and full payloads plus event metadata at the model pod via the HyperPod inference operator's NGINX, OpenTelemetry collector and S3 uploader sidecars. AWS's walkthrough uses a synthetic home-appliance support router to show captured traffic, Amazon Athena queries surfacing a new-product blind spot, SQL labeling and a fine-tune that teaches the router the new product, with accuracy measured before and after.", "body_md": "## [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Build a model improvement flywheel with Amazon SageMaker HyperPod Data Capture\n\nFoundation models deployed on Amazon SageMaker HyperPod meet inputs that no pre-curated dataset covers: domain-specific jargon, unexpected formats, new products, and changing user behavior. Without a record of production inference data, teams can’t detect drift, find failure patterns, or use real interactions to improve the model.\n\nAmazon SageMaker HyperPod now provides Data Capture, a built-in capability that records live inference inputs and outputs to Amazon Simple Storage Service (Amazon S3) at three independent tiers, with minimal latency impact and zero impact on model availability.\n\nIn this post, we walk through one complete example on a single SageMaker HyperPod cluster. A home-appliance support router captures its production traffic, Amazon Athena queries surface a new-product blind spot, a SQL policy labels the captured records, and a fine-tune on that data teaches the router the new product. We measure accuracy before and after, on data that could not have existed before the model was deployed. The inference operator serves the model on one node, and the training operator runs the fine-tuning on another.\n\nWe use synthetic data for the walkthrough. The scripts, manifests, SQL, synthetic datasets, and expected outputs are available in the [accompanying GitHub repository](https://github.com/awslabs/awsome-distributed-ai/tree/main/examples/use-cases/sagemaker-hyperpod-data-capture-flywheel).\n\n## Three-tier Data Capture architecture\n\nHyperPod Data Capture provides three independent tiers, each at a different point in the request path. You can turn on any combination in the same `dataCapture` block.\n\n|  | **Tier 1: SageMaker endpoint** | **Tier 2: Load balancer** | **Tier 3: Model pod** | \n| What’s captured | Full request/response payloads | Application Load Balancer (ALB) access-log fields: request line, status codes, timings, client IP, user agent | Full request/response payloads plus event metadata | \n| Managed by | SageMaker native data capture | ALB access logs | HyperPod inference operator (NGINX + OTel collector + S3 uploader sidecars) | \n| Best for | Compliance, Model Monitor, audit | Traffic and error-rate monitoring at low cost | Debugging, analytics, training-data collection | \n| Configuration | `sagemakerEndpoint:` — sampling %, input/output, AWS KMS | `loadBalancer: { enabled: true }` | `modelPod:` — sampling %, input/output, buffer, max payload size, AWS KMS, content types | \n\nTiers 1 and 3 both store full payloads, so enabling both stores every record twice. Use Tier 1 for compliance and Model Monitor, Tier 2 when you only need traffic metadata, and Tier 3 when you intend to *use* the payloads. This post uses Tier 3 only. Because Tier 3 records complete payloads, which can include names, addresses, and order numbers, protect that data before it becomes training data. Use sampling, AWS Key Management Service (AWS KMS) encryption (`kmsKeyId`), S3 lifecycle rules, and personally identifiable information (PII) redaction (for example, with Amazon Comprehend).\n\n## Prerequisites\n\n- A SageMaker HyperPod cluster orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS), with the inference operator installed ([Setting up your HyperPod clusters for model deployment](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-setup.html) ) and the training operator EKS add-on`amazon-sagemaker-hyperpod-training-operator` installed ([Installing the training operator](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-eks-operator-install.html) ).\n- **Two available NVIDIA GPUs** , one for serving and one for training. We used an`ml.g6e.8xlarge` for serving and an`ml.g6e.xlarge` for training (one 48 GB NVIDIA L40S each). The serving pod holds its GPU for the life of the endpoint, so with only one GPU the training pod stays`Pending` .\n- `kubectl` configured for the cluster.\n- An S3 bucket in the same Region.\n\nConfirm both operators are installed. The command returns two rows:\n\n## Solution overview\n\nThe following diagram shows the five steps of the walkthrough on one SageMaker HyperPod cluster: capture every request and response to Amazon S3, analyze the captures with Athena, label them with a SQL policy, fine-tune with the training operator on a second GPU, and redeploy side by side with capture still on.\n\nThe inference operator adds three sidecars to the model pod. Every request that passes through the NGINX proxy is copied to an OpenTelemetry collector, batched, and written to S3 through a Mountpoint-S3 volume. Athena reads those files in place. The training operator runs the fine-tune as a `HyperPodPyTorchJob` on a second GPU, and the new model is served by a second endpoint that also has capture on.\n\n### Step 0: Set your variables\n\nEvery later step reads these. Edit the first block, then paste the whole thing:\n\n### Step 1: Stage the base model in S3\n\nWe use `Qwen2.5-3B-Instruct` because it is small enough to fine-tune in about a minute and fits on a single L40S.\n\n### Step 2: IAM permissions\n\nFour identities need access. For the policy statements, see [Data capture for inference on HyperPod](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-data-capture.html).\n\n- **The Mountpoint-S3 CSI driver role.** Tier 3 capture writes through a Mountpoint-S3 volume, and the driver authenticates as its own IAM roles for service accounts (IRSA) role, not as the pod’s service account. It needs`s3:PutObject` ,`s3:AbortMultipartUpload` , and`s3:DeleteObject` on your capture bucket, plus`kms:Decrypt` ,`kms:GenerateDataKey` , and`kms:DescribeKey` if you set`kmsKeyId` . If this is missing, the endpoint stays healthy and requests succeed, but no captures appear. See[Mountpoint for Amazon S3 CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html) .\n- **The inference operator execution role.** It needs`s3:PutObject` on your capture bucket, plus`kms:Decrypt` and`kms:GenerateDataKey` if you set`kmsKeyId` . The capture files themselves are written through the CSI driver role, so when no captures appear, check that role first.\n- **The training pod’s service account (**`$TRAIN_SA`**).** Read access to the base model and training data, and write access for the merged model, through[EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) .\n- **Athena.** Read access to the capture prefix, plus a query-results location.\n\n## Enabling Data Capture\n\nThe next three steps deploy the router with Tier 3 capture on, check that the pod runs the capture sidecars, and confirm that records reach Amazon S3.\n\n### Step 3: Deploy the model with Tier 3 capture enabled\n\nThis is the whole feature: the `dataCapture.modelPod.enabled: true` block. Everything else is an ordinary vLLM deployment.\n\nThe preceding `modelPod` block sets 100% sampling and captures both input and output, which are the defaults. To tune it, add any of these optional fields under `modelPod`:\n\n### Step 4: Wait for the pod\n\nA capture-enabled pod runs **four containers**: the model, `sidecar-reverse-proxy` (NGINX), `otel-collector`, and `s3-uploader`. The model container and the pod’s `app` label are both named after `modelName`.\n\nIn our run, the pod was ready about five minutes after `kubectl apply`, most of it spent downloading the weights.\n\n### Step 5: Send a request and confirm captures land in S3\n\nSend an inference request through the capture proxy on port **8081**, then, after the flush interval, list the capture prefix. The model itself listens on port 8000. Requests sent there bypass the proxy and are not captured.\n\nYou see one or more `.jsonl` objects under `{s3Uri}/{uniqueID}/pod/{variant}/{YYYY}/{MM}/{DD}/{HH}/`. The `uniqueID` is generated per endpoint. Each line is one inference in the SageMaker data-capture format shown next.\n\n### The captured record format\n\nThe request and response bodies are stored **base64-encoded**, as the `encoding` field shows, so reading them back requires base64-decoding and then parsing JSON:\n\nDecoded, `endpointInput.data` is the request body the client sent and `endpointOutput.data` is the response the model returned.", "url": "https://wpnews.pro/news/build-a-model-improvement-flywheel-with-amazon-sagemaker-hyperpod-data-capture", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/build-a-model-improvement-flywheel-with-amazon-sagemaker-hyperpod-data-capture/", "published_at": "2026-11-13 18:01:15+00:00", "updated_at": "2026-11-13 18:23:19.170328+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "mlops", "ai-tools"], "entities": ["Amazon SageMaker HyperPod", "Amazon Web Services", "Amazon Simple Storage Service", "Amazon Athena", "Amazon Elastic Kubernetes Service", "Amazon Key Management Service", "NVIDIA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/build-a-model-improvement-flywheel-with-amazon-sagemaker-hyperpod-data-capture", "markdown": "https://wpnews.pro/news/build-a-model-improvement-flywheel-with-amazon-sagemaker-hyperpod-data-capture.md", "text": "https://wpnews.pro/news/build-a-model-improvement-flywheel-with-amazon-sagemaker-hyperpod-data-capture.txt", "jsonld": "https://wpnews.pro/news/build-a-model-improvement-flywheel-with-amazon-sagemaker-hyperpod-data-capture.jsonld"}}