{"slug": "orchestrating-sagemaker-ai-endpoint-auto-scaling-with-aws-cdk", "title": "Orchestrating SageMaker AI endpoint auto scaling with AWS CDK", "summary": "AWS published a reusable AWS CDK construct that uses CloudFormation custom resources to deregister and re-register SageMaker AI endpoint auto scaling automatically during endpoint updates, fixing deployments that otherwise fail or silently lose auto scaling. The construct replaces the native AWS::ApplicationAutoScaling::ScalableTarget and ScalingPolicy resources with three custom resources ordered via node.add_dependency: a deregister step, the CDK-managed endpoint update, then scalable-target registration and scaling-policy attachment, which wait for the endpoint to reach InService and poll every 30 seconds with a 1-hour total_timeout for registration. The problem matters because manually deregistering via the AWS CLI can leave an endpoint running without auto scaling and no error, or push a stack into UPDATE_ROLLBACK_FAILED requiring manual recovery in the CloudFormation console.", "body_md": "## [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Orchestrating SageMaker AI endpoint auto scaling with AWS CDK\n\nUpdating the Amazon Sagemaker AI real-time endpoint configuration (changing the instance type, instance count, or environment variables) causes deployments through AWS CloudFormation or the AWS Cloud Development Kit (AWS CDK) to fail. SageMaker AI requires you to deregister the scalable target before updating the endpoint, but when you deregister it, CloudFormation can’t track the AWS Application Auto Scaling resources.\n\nIn this post, we show you how to use a reusable CDK construct with AWS CloudFormation custom resources to orchestrate deregister and register auto scaling automatically during endpoint updates. You can update your endpoint configuration and auto scaling parameters in a single deployment without manual intervention.\n\n## Why this problem occurs\n\nWhen you deploy a SageMaker AI endpoint with native CloudFormation auto scaling resources, then change the endpoint configuration and redeploy, several failure modes emerge. The specific failure depends on what you change and how you work around the issue.\n\n### Changing the endpoint configuration with auto scaling registered\n\nCloudFormation tries to update the endpoint while the scalable target is still registered. SageMaker AI rejects the update with the following error:\n\nSageMaker AI blocks the endpoint update before CloudFormation reaches the auto scaling resources.\n\n### Manually deregistering auto scaling, then changing only the endpoint configuration\n\nIf you deregister the scalable target through the AWS Command Line Interface (AWS CLI) and redeploy with only an endpoint configuration change, the deployment succeeds. However, CloudFormation doesn’t register auto scaling even though it is in the CloudFormation template. Because the auto scaling parameters in the CloudFormation template haven’t changed, CloudFormation doesn’t register the scalable target or re-apply the scaling policy. The endpoint is now running without auto scaling, and there is no error to indicate anything went wrong.\n\n### Manually deregistering auto scaling, then changing auto scaling parameters\n\nIf you deregister the scalable target through the AWS CLI and redeploy with auto scaling parameter changes, CloudFormation tries to update the ScalableTarget resource that no longer exists:\n\nThis failure is notable because the rollback also fails (UPDATE_ROLLBACK_FAILED), leaving the stack in a broken state that requires manual intervention in the CloudFormation console to recover.\n\n## Solution overview\n\nYou can use this CDK construct to replace the native `AWS::ApplicationAutoScaling::ScalableTarget` and `AWS::ApplicationAutoScaling::ScalingPolicy` CloudFormation resources with three custom resources. These custom resources deregister and register auto scaling around the endpoint update with explicit dependency ordering, enforced by `node.add_dependency` calls in CDK. The full sequence is:\n\n1. **Deregister** (custom resource): Removes the scalable target before the endpoint update begins. On first deploy, no scalable target exists yet, so the handler handles the error gracefully and continues.\n2. **Endpoint update** (CDK-managed): CDK updates the endpoint resource normally, swapping to the new endpoint configuration.\n3. **Register Scalable Target** (custom resource): Waits for the endpoint to reach`InService` status, then registers the scalable target.\n4. **Attach Scaling Policy** (custom resource): Waits for the scalable target to be queryable, then applies the target tracking policy.\n\nRegister Scalable Target and Attach Scaling Policy use the CDK provider’s asynchronous completion pattern. Each custom resource has an `on_event` handler that returns immediately and an `is_complete` handler. The provider’s waiter re-invokes the `is_complete` handler on an interval (every 30 seconds) until its readiness check passes. The provider’s `total_timeout` (1 hour for Register Scalable Target, 15 minutes for Attach Scaling Policy) bounds the wait, so it isn’t limited by the timeout of any single Lambda invocation.\n\nA `ConfigHash` property on each custom resource forces CloudFormation to re-invoke the Lambda functions whenever a configuration value changes. The endpoint configuration and auto scaling parameters are defined together because a single `ConfigHash` drives the auto scaling orchestration.\n\nWhile the construct defines three custom resources, the deployed stack contains more than three Lambda functions. Each custom resource is backed by a CDK `cr.Provider`, and the provider framework generates its own helper Lambda functions. A synchronous provider (Deregister) adds one framework function. An asynchronous provider (Register Scalable Target and Attach Scaling Policy) adds three (`onEvent`, `isComplete`, and `onTimeout`) plus an AWS Step Functions state machine that drives the waiter. As a result, the deployed stack contains three custom resources, two Step Functions state machines, and twelve Lambda functions. Five of the resources hold the construct’s own handler code (one for Deregister, and an `on_event` and `is_complete` pair for each asynchronous resource), with the remaining seven generated by the provider framework. You don’t write or maintain the framework-generated functions, but they are present in the stack and worth accounting for when you review the deployed resources.\n\n## Walkthrough\n\nIn this walkthrough, you deploy a SageMaker AI endpoint with auto scaling using the provided example stack, then verify that configuration updates work correctly.\n\nThe walkthrough includes the following steps:\n\n1. Clone the repository and install dependencies.\n2. Configure your endpoint and auto scaling settings.\n3. Deploy and verify.\n4. Test an update.\n\nThe source code is available in the [linked GitHub repository](https://github.com/aws-samples/sample-sagemaker-endpoint-autoscaling).\n\n### Prerequisites\n\nFor this walkthrough, you need the following:\n\n- An AWS account with permissions for Amazon SageMaker AI, AWS Lambda, AWS Identity and Access Management (IAM), Application Auto Scaling, and Amazon CloudWatch.\n- [AWS CDK v2](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) installed.\n- [Python 3.12](https://www.python.org/downloads/) or later.\n- [uv](https://docs.astral.sh/uv/getting-started/installation/) for Python dependency management.\n- A SageMaker AI model artifact in Amazon Simple Storage Service (Amazon S3) and a container image in Amazon Elastic Container Registry (Amazon ECR), or a SageMaker AI built-in algorithm container.\n\n### Clone the repository and install dependencies\n\n1. Clone the repository:\n2. Install dependencies:\n\n### Configure your endpoint and auto scaling settings\n\nThe endpoint and auto scaling settings are defined together in config/`config.yaml` because a single `config_hash` drives the auto scaling orchestration. A change to either section triggers the full sequence.\n\nEdit config/`config.yaml` with your values:\n\nThe endpoint parameters map to the SageMaker AI `CreateModel` and `CreateEndpointConfig` APIs. The auto scaling parameters map to the Application Auto Scaling `RegisterScalableTarget` and `PutScalingPolicy` APIs.\n\n### Deploy and verify\n\n1. Bootstrap your AWS environment if you haven’t already:\n2. Deploy the stack:\n\nCloudFormation creates the SageMaker AI model, endpoint configuration, and endpoint, then runs the three custom resources in sequence to register auto scaling.\n\n1. Verify that the deployment registered the scalable target:\n2. Verify that the deployment applied the scaling policy:\n\n### Test an update\n\nChange a value in config/`config.yaml`, for example, update the instance type from `ml.m5.large` to `ml.m5.xlarge`, and redeploy:\n\nAfter the deployment completes, verify that auto scaling is still registered and has the updated values:\n\n### Integrating with an existing stack\n\nUse the example stack if you’re starting fresh or want a reference implementation. Use the construct directly if you already have a CDK stack with SageMaker AI resources.\n\nTo integrate with an existing stack, copy the `sagemaker_autoscaling/constructs/` directory into your project and add the construct alongside your endpoint resources:\n\n## How the construct works\n\nThis section walks through the key implementation details of `SagemakerAutoscalingConstruct`.\n\n### Deregister\n\nThe Deregister custom resource calls `deregister_scalable_target` before the endpoint update. It handles `ObjectNotFoundException` for first deployments where no scalable target exists:\n\nThe dependency is established in CDK:\n\n### Register Scalable Target\n\nEven though CloudFormation treats the endpoint resource as complete once the update is accepted, the scalable target can only be registered after the endpoint reaches `InService`. Rather than block a single Lambda invocation on that wait, Register Scalable Target uses the provider’s asynchronous completion pattern. The `on_event` handler returns immediately, and the provider’s waiter re-invokes the `is_complete` handler on `query_interval` until the endpoint is `InService`. The provider’s `total_timeout` (1 hour) bounds the wait, not the timeout of any single Lambda invocation:\n\nOn stack deletion, the `on_event` handler deregisters the scalable target to clean up.\n\n### Attach Scaling Policy\n\nThere is a brief window after `RegisterScalableTarget` returns where the target isn’t yet queryable. Attach Scaling Policy uses the same asynchronous completion pattern: the `on_event` handler returns immediately, and the `is_complete` handler reports completion only once the scalable target is queryable, at which point it applies the policy. The provider’s `total_timeout` (15 minutes) bounds the wait:\n\nOn stack deletion, the `on_event` handler deletes the scaling policy to clean up.\n\n### Cleaning up\n\nTo avoid incurring future charges, delete the resources:\n\nThis removes the SageMaker AI endpoint, endpoint configuration, model, Lambda functions, IAM roles, Amazon CloudWatch alarms created by the target tracking scaling policy, and deregisters the scalable target.\n\n## Conclusion\n\nIn this post, we showed you how to use a reusable CDK construct to deregister and register auto scaling automatically to orchestrate SageMaker AI endpoint updates. The construct uses three CloudFormation custom resources with explicit dependency ordering and a `ConfigHash` property to keep auto scaling intact across deployments.\n\nTo get started, clone the `sample-sagemaker-endpoint-autoscaling` repository and follow the setup instructions. The repository includes an example stack you can deploy directly, or you can integrate the construct into your existing CDK stack.\n\nFor more information about Application Auto Scaling for SageMaker AI, see [Automatically scale Amazon SageMaker AI models](https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling.html). For more information about CloudFormation custom resources, see Custom resources in the [AWS CloudFormation User Guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html).", "url": "https://wpnews.pro/news/orchestrating-sagemaker-ai-endpoint-auto-scaling-with-aws-cdk", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/orchestrating-sagemaker-ai-endpoint-auto-scaling-with-aws-cdk/", "published_at": "2026-10-23 16:30:10+00:00", "updated_at": "2026-10-23 16:44:19.112861+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "developer-tools", "artificial-intelligence"], "entities": ["Amazon SageMaker AI", "AWS CDK", "AWS CloudFormation", "AWS Application Auto Scaling", "AWS CLI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/orchestrating-sagemaker-ai-endpoint-auto-scaling-with-aws-cdk", "markdown": "https://wpnews.pro/news/orchestrating-sagemaker-ai-endpoint-auto-scaling-with-aws-cdk.md", "text": "https://wpnews.pro/news/orchestrating-sagemaker-ai-endpoint-auto-scaling-with-aws-cdk.txt", "jsonld": "https://wpnews.pro/news/orchestrating-sagemaker-ai-endpoint-auto-scaling-with-aws-cdk.jsonld"}}