{"slug": "batch-write-and-discover-records-in-amazon-sagemaker-feature-store", "title": "Batch write and discover records in Amazon SageMaker Feature Store", "summary": "Amazon Web Services (AWS) announced two new APIs for Amazon SageMaker Feature Store: BatchWriteRecord, which writes up to 25 records across multiple feature groups in a single call, and ListRecords, which enables enumeration of record identifiers in both Standard and In-Memory storage tiers. The APIs address throughput bottlenecks in high-volume feature pipelines and provide a way to recover records in the In-Memory tier, which previously had no discovery mechanism.", "body_md": "[Artificial Intelligence](/blogs/machine-learning/)\n\n# Batch write and discover records in Amazon SageMaker Feature Store\n\n[Amazon SageMaker Feature Store](/sagemaker/ai/feature-store/) is a fully managed, purpose-built repository to store, share, and manage features for machine learning (ML) models. It provides low-latency online serving for real-time inference, an offline store for historical retention and training feature data, and supports both [streaming](/blogs/machine-learning/using-streaming-ingestion-with-amazon-sagemaker-feature-store-to-make-ml-backed-decisions-in-near-real-time/) and [batch](/blogs/machine-learning/scale-ml-feature-ingestion-using-amazon-sagemaker-feature-store/) ingestion patterns.\n\nAs ML platforms mature, two operational gaps surface repeatedly. First, teams running high-throughput feature pipelines must call [PutRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_PutRecord.html) (which writes a single feature record to the online store) in a loop. This means one API call per record, per feature group, which creates connection overhead and poor throughput. A fraud-detection pipeline ingesting 10,000 records per second across five feature groups must sustain 50,000 individual API calls per second only to keep features current. A second challenge is that teams using the In-Memory storage tier have no way to browse or enumerate records stored in the online store. If record identifiers are lost through a bug or pipeline failure, those records become permanently unrecoverable. There is no offline store for the In-Memory tier to fall back on, no [Amazon Athena](/athena/) query to run, and no API to discover what exists.\n\nToday, we are announcing [two new APIs](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_Operations_Amazon_SageMaker_Feature_Store_Runtime.html) for Amazon SageMaker Feature Store:\n\n[BatchWriteRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_BatchWriteRecord.html)— Write up to 25 records across multiple feature groups in a single API call, with partial-success semantics, per-record time-to-live (TTL) control, and the same EventTime-based ordering guarantees as PutRecord.[ListRecords](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_ListRecords.html)— Enumerate record identifiers within a feature group using pagination. Works with both Standard ([Amazon DynamoDB](/dynamodb/)-backed) and In-Memory ([Redis](/elasticache/)-backed) storage tiers.\n\nIn this post, we walk through each API with code examples you can use to get started.\n\n## Prerequisites\n\nTo follow along with the examples in this post, you need:\n\n- An\n[AWS account](/free/)with permissions to create Amazon SageMaker AI resources. - An\n[Amazon SageMaker AI execution role](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html)with access to[Amazon Simple Storage Service (Amazon S3)](/s3/)and[AWS Glue](/glue/), and permissions to interact with Feature Store data plane APIs. The following AWS Identity and Access Management (IAM) policy shows the minimum required permissions: [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)(latest version) or[SageMaker Python SDK v3.8.0](https://github.com/aws/sagemaker-python-sdk/releases/tag/v3.8.0)or later.- One or more existing feature groups with records ingested (if you are new to Feature Store, see the\n[end-to-end workshop notebook](https://github.com/aws-samples/amazon-sagemaker-feature-store-end-to-end-workshop/blob/main/01-module-feature-store-foundations/m1_nb1_introduction_to_feature_store.ipynb)).\n\n## BatchWriteRecord\n\nThe BatchWriteRecord API tackles the throughput limits of single-record ingestion. The following sections explain the problem it solves and how it works.\n\n### The challenge with single-record ingestion\n\nThe existing [PutRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_PutRecord.html) API in Feature Store writes one record to one feature group per call. Each call performs a conditional write: the record is persisted as the “latest” version only if its EventTime, included in the request, is newer than the existing record. If the condition fails, the record is still written as a historical version for the offline store.\n\nThis design provides strong ordering guarantees, but at scale it forces an N×M calling pattern (N records × M feature groups), creating connection overhead and tail latency that limit throughput.\n\n### How BatchWriteRecord works\n\n[BatchWriteRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_BatchWriteRecord.html) accepts up to 25 entries in a single request, targeting one or more feature groups simultaneously. Each record succeeds or fails independently. This is a partial-success API, meaning individual record failures do not fail the entire request.\n\nThe API preserves the same EventTime-based ordering as [PutRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_PutRecord.html):\n\n- If the incoming record’s EventTime is newer than the existing record, it becomes the latest version in the online store.\n- If not, the record is written as a historical version to the offline store (for feature groups with offline storage).\n- Records that fail for other reasons (authentication/validation errors, service throttling) are returned in the response with error details and the original record.\n- The requests that are unprocessed will be returned in response as\n[UnprocessedEntries](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_BatchWriteRecord.html#API_feature_store_BatchWriteRecord_ResponseSyntax)which can be retried.\n\n### Request structure\n\nThe response returns only the records that failed:\n\nRecords not listed in [Errors](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_BatchWriteRecord.html#API_feature_store_BatchWriteRecord_ResponseSyntax) or [UnprocessedEntries](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_BatchWriteRecord.html#API_feature_store_BatchWriteRecord_ResponseSyntax) succeeded. Your application should retry only the failed records using exponential backoff for retriable errors.\n\n### Code example: Batch ingestion with Boto3\n\n### Code example: Writing across multiple feature groups\n\nYou can target multiple feature groups in a single request. Records are grouped by feature group and processed independently:\n\nA failure in one feature group does not affect records destined for other feature groups.\n\n### TTL (Time-to-Live) support\n\n[BatchWriteRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_BatchWriteRecord.html) supports [TTL](https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store-time-to-live.html) at three levels of precedence, shown in the following priority order:\n\n- Record-level TTL — Set with TtlDuration on individual entries. Takes highest priority.\n- Request-level TTL — A default TtlDuration at the top level of the request, applied to entries without a record-level TTL.\n- Feature-group-level TTL — The TTL configured on the feature group itself, applied when neither record-level nor request-level TTL is set.\n\n### Key considerations\n\nMaximum 25 entries per request. This limit applies to the total number of entries across all feature groups in a single request.\n\nPartial-success semantics: Unlike transactional APIs, BatchWriteRecord does not roll back successful writes if some records fail. Design your retry logic to re-submit only the records returned in Errors.\n\nSimilar IAM model as PutRecord: The caller must have sagemaker:BatchWriteRecord and sagemaker:PutRecord permission on the Amazon Resource Name (ARN) of each target feature group. Per-feature-group authorization is checked before processing.\n\nEventTime ordering is preserved: BatchWriteRecord uses conditional writes to maintain the same latest-record-wins semantics as PutRecord. A stale record cannot overwrite a newer one in the online store.\n\nTargetStores flexibility: Each entry can independently target OnlineStore, OfflineStore, or both (defaults to the feature group’s enabled stores), giving you fine-grained control over where each record lands.\n\n## ListRecords\n\nThe ListRecords API closes the gap in record discovery for both storage tiers. The following sections explain the problem it solves and how it works.\n\n### The challenge with record discovery\n\nFeature Store supports [PutRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_PutRecord.html), [GetRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_GetRecord.html), and [DeleteRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_DeleteRecord.html), but all require the caller to know the exact record identifier. There is no API to browse or enumerate records within a feature group.\n\nFor the Standard tier, the workaround is querying the offline store by using [Amazon Athena](/athena/). This requires offline store configuration, adds cost, and is not real-time.\n\nFor the In-Memory tier, the situation is critical. There is no corresponding offline store by default. If record identifiers are lost, those records are completely unrecoverable. You cannot discover them, and you cannot delete them. This leads to phantom data, wasted storage costs, and potential compliance risks when data subjects request deletion.\n\n### How ListRecords works\n\n[ListRecords](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_ListRecords.html) enumerates record identifiers within a feature group using pagination. It returns only active, non-deleted, non-expired records that are ready to be used with [GetRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_GetRecord.html) or [DeleteRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_DeleteRecord.html).\n\nThe API works with both storage tiers:\n\n- Standard tier (\n[Amazon DynamoDB](/dynamodb/)): Scans the online store, returning identifier of the latest version of each record. Soft-deleted and expired records are automatically excluded. - In-Memory tier (\n[Redis](/elasticache/)): Scans keys and filters out soft-deleted records and internal system keys. Returns record identifiers extracted from key names.\n\n### Request and response structure\n\nRequest body:\n\nInitial call\n\nOr\n\nResponse:\n\nWhen NextToken is absent in the response, pagination is complete.\n\n### Code example: Enumerate all records in a feature group\n\n### Code example: Clean up orphaned records\n\nA common use case is identifying and deleting records that are no longer needed. This is critical for In-Memory tier feature groups, where orphaned records persist indefinitely:\n\n### Pagination behavior\n\n- Page size: Configurable through MaxResults (default 10, maximum 100).\n- Token format: Opaque, encrypted string. Do not parse or construct tokens. Pass them through unchanged.\n- Ordering: Results are not guaranteed to be in any particular order.\n- Concurrent writes: If records are written or deleted during pagination, you may observe duplicates or gaps. This is documented behavior.\n- Token scope: Tokens are tied to a specific feature group and account and cannot be reused across either.\n\n### Key considerations\n\nRecord identifiers only. The current release returns record identifiers without feature values. Use [GetRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_GetRecord.html) or [BatchGetRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_BatchGetRecord.html) to retrieve full records for the identifiers you need.\n\nAutomatic filtering. The API excludes soft-deleted records, expired records (Standard tier TTL), and internal system keys (In-Memory tier). You see only active, retrievable records.\n\nIAM permission. The caller must have sagemaker:ListRecords permission on the feature group ARN.\n\nBoth tiers supported. ListRecords works identically from the caller’s perspective regardless of whether the feature group uses Standard or In-Memory storage.\n\n## Putting it together\n\nThese two APIs complement each other naturally. Consider a compliance workflow that verifies complete data deletion for a user across multiple feature groups:\n\n## Cleanup\n\nTo avoid ongoing charges, delete feature groups you created while following this walkthrough. For In-Memory tier feature groups, use [ListRecords](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_ListRecords.html) to enumerate records and [DeleteRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_DeleteRecord.html) to remove them before deleting the feature group.\n\n## Conclusion\n\n[BatchWriteRecord](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_BatchWriteRecord.html) and [ListRecords](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_ListRecords.html) provide key enhancements in the data plane of Amazon SageMaker Feature Store. BatchWriteRecord reduces the API call volume for high-throughput ingestion by up to 25x while preserving the EventTime-based ordering guarantees that keep your online store correct. ListRecords unlocks record discovery and lifecycle management. This is critical for In-Memory tier customers who previously had no way to enumerate or clean up their data.\n\nTogether, these APIs support patterns that were previously difficult or impossible: bulk ingestion pipelines with fewer connections and lower latency, compliance workflows that can verify complete data deletion, and operational tooling that can browse feature group contents in real time.\n\nFor more information, see the [Feature Store documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store.html), the [Feature Store API reference](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_Operations_Amazon_SageMaker_Feature_Store_Runtime.html), the [offline store configuration documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_OfflineStoreConfig.html#sagemaker-Type-OfflineStoreConfig-DisableGlueTableCreation), and the [What’s New announcement](/about-aws/whats-new/2026/07/amzn-sgm-feature-store-batch-write-list/).\n\nFor background on Feature Store capabilities, explore these related posts:\n\n[Understanding the Key Capabilities of Amazon SageMaker Feature Store](/blogs/machine-learning/understanding-the-key-capabilities-of-amazon-sagemaker-feature-store/).[Accelerate ML Feature Pipelines with new capabilities in Amazon SageMaker Feature Store](/blogs/machine-learning/accelerate-ml-feature-pipelines-with-new-capabilities-in-amazon-sagemaker-feature-store/).[Using Streaming Ingestion with Amazon SageMaker Feature Store](/blogs/machine-learning/using-streaming-ingestion-with-amazon-sagemaker-feature-store-to-make-ml-backed-decisions-in-near-real-time/).", "url": "https://wpnews.pro/news/batch-write-and-discover-records-in-amazon-sagemaker-feature-store", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/batch-write-and-discover-records-in-amazon-sagemaker-feature-store/", "published_at": "2026-08-28 19:31:05+00:00", "updated_at": "2026-08-28 19:49:31.322075+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure", "ai-tools"], "entities": ["Amazon Web Services", "Amazon SageMaker Feature Store", "BatchWriteRecord", "ListRecords", "PutRecord", "Amazon DynamoDB", "Redis", "SageMaker Python SDK"], "alternates": {"html": "https://wpnews.pro/news/batch-write-and-discover-records-in-amazon-sagemaker-feature-store", "markdown": "https://wpnews.pro/news/batch-write-and-discover-records-in-amazon-sagemaker-feature-store.md", "text": "https://wpnews.pro/news/batch-write-and-discover-records-in-amazon-sagemaker-feature-store.txt", "jsonld": "https://wpnews.pro/news/batch-write-and-discover-records-in-amazon-sagemaker-feature-store.jsonld"}}