Govern models with MLflow and Amazon SageMaker AI Model Registry sync: Part 2

AWS published a guide for governing ML models across accounts using MLflow and Amazon SageMaker AI Model Registry sync, detailing hub-and-spoke and hybrid topologies for regulated environments. The post extends Part 1's single-account setup to cross-account governance, introducing administrator and model owner personas, and includes working notebooks in a GitHub repository.

Artificial Intelligence /blogs/machine-learning/ Govern models with MLflow and Amazon SageMaker AI Model Registry sync: Part 2 Governing models across accounts is the natural next step once automatic model registration is in place. In Part 1 /blogs/machine-learning/govern-models-with-mlflow-and-amazon-sagemaker-ai-model-registry-sync-part-1/ we introduced how managed MLflow on Amazon SageMaker AI /sagemaker-ai/ synchronizes registered models into the SageMaker AI Model Registry https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry.html . We walked through a single-account setup where AWS Identity and Access Management IAM condition keys separate the data scientist and governance officer personas. Larger organizations, however, separate development from production at the account level. They run multiple development accounts and a central governance function. In regulated environments, they also impose a hard requirement that development workloads cannot write into production-grade accounts. In this post, we extend the same building blocks to two cross-account governance topologies. The first is a hub-and-spoke pattern that centralizes governance by sharing one MLflow app across development accounts with AWS Resource Access Manager AWS RAM . The second is a hybrid pattern for regulated environments that keeps development accounts fully isolated from the governance hub. We close by showing how an approved model moves from the registry to a deployed endpoint through continuous integration and continuous delivery CI/CD , and we compare the topologies to help you choose one. Working notebooks are available in the accompanying GitHub repository https://github.com/aws-samples/genai-ml-platform-examples/tree/main/operations . The personas Part 1 /blogs/machine-learning/govern-models-with-mlflow-and-amazon-sagemaker-ai-model-registry-sync-part-1/ introduced the data scientist works in a Jupyter notebook, interacts only with MLflow and the governance officer works in the Amazon SageMaker Studio Models UI . Cross-account topologies add two more. The administrator performs the one-time cross-account setup: AWS RAM shares, bucket policies, and destination groups, using the console or CLI. In the hybrid topology, a model owner in each development account approves models locally before they are promoted to the hub. Each topology that follows is organized by these persona workflows. Prerequisites Before starting this walkthrough, make sure you have the following: - Completion of Part 1. Familiarity with automatic model registration, the lifecycle staging construct, and IAM condition key gates from Part 1 /blogs/machine-learning/govern-models-with-mlflow-and-amazon-sagemaker-ai-model-registry-sync-part-1/ . - Two AWS accounts , a development spoke account and a governance hub account. Configure a named AWS Command Line Interface AWS CLI profile for each for example, mlops-spoke and mlops-hub . For setup details, see the Account and profile setup REPO LINK account-and-profile-setup section in the accompanying repository. - The accompanying AWS CloudFormation stack deployed into both accounts. The template cfn/sagemaker-studio-mlflow.yaml REPO LINK provisions a SageMaker AI Studio domain, user profile, execution role, and a Managed MLflow app with AutoModelRegistrationEnabled . Deploy one stack per account. See Provision the environment REPO LINK provision-the-environment-cloudformation in the repository README for the exact commands. For the full provisioning walkthrough, including CloudFormation deployment, reading stack outputs, and setting environment variables, refer to the accompanying repository REPO LINK . Topology 1: Hub-and-spoke central governance For larger organizations with multiple teams, it is common to have multiple separate development accounts with a central governance account. This topology provides workload isolation for development within each development account, with shared services offered through a centralized account hosting the MLflow app and the central Model Registry. The centralized hub account shares the MLflow app with one or more spoke development accounts using AWS RAM. This pattern extends the approach described in Centralize model governance with SageMaker Model Registry and AWS RAM sharing /blogs/machine-learning/centralize-model-governance-with-sagemaker-model-registry-resource-access-manager-sharing/ . The following figure illustrates the workflow: 1. The hub creates an AWS RAM resource share for the MLflow app and the spoke accepts the invitation. Using external principals lets the share work even when the accounts are not in the same organization. 2. A data scientist in the spoke registers a model against the shared MLflow app. With automatic registration, you get the Model Package Group and version in the hub account, created synchronously with the register call. 3. The hub attaches a resource policy to the group and shares it back to the spoke with the AllowDeploy managed permission, so the spoke can describe and deploy the model. 4. The governance officer validates the synced metrics and lineage and approves the model centrally in the hub. 5. The approval event triggers a CI/CD pipeline, and a machine learning ML engineer deploys the approved model to an Amazon SageMaker endpoint in the spoke account. Administrator: Set up cross-account access one-time The administrator prepares the hub and the sharing before either persona starts working. Prerequisite: model artifacts live in the hub’s MLflow artifact store, an S3 bucket in the hub account. Cross-account S3 access is needed on both sides: - Resource side: a bucket policy on the hub artifact bucket granting the spoke account s3:GetObject , s3:PutObject , s3:ListBucket , and s3:GetBucketLocation . Without it, the data scientist’s log model call fails with an S3 access error before registration happens. - Identity side: the spoke execution role needs s3:GetObject on the hub bucket for deploy-time artifact reads. AmazonSageMakerFullAccess covers buckets whose name contains sagemaker . If the spoke role is scoped tighter, add the hub bucket explicitly. With the prerequisite in place, the administrator: 1. Creates an AWS RAM resource share for the hub MLflow app, which the spoke accepts. 2. After the first registration, attaches a resource policy to the synced Model Package Group and RAM-shares it back to the spoke with the AllowDeploy managed permission. In the hub AWS RAM console both shares show as Active: Data scientist: Register from the spoke, in a notebook The workflow is identical to Part 1, with one difference: the notebook points the MLflow tracking URI at the hub app’s Amazon Resource Name ARN . Automatic registration creates the Model Package Group and version in the hub , synchronously with the register call. The candidate appears in the hub registry pending approval, with the metrics, evaluation card, and lineage from the spoke’s run, as shown in the following figure: Two points to plan for. First, automatic registration appends a short hash suffix to the group name, so my-model becomes my-model-<hash . Discover the group with list model package groups NameContains=... , and reference it cross-account by its full ARN rather than its name. Second, lineage is recorded in the hub, because the sync executes there. The governance officer sees the full lineage graph in the hub’s Studio, but lineage entities are not shared back to the spoke by the AWS RAM share. Cross-account lineage sharing is not automatic. Governance officer: Approve centrally, in the hub Studio UI The governance officer works in the hub’s Studio Models view, one control point for candidates from every spoke. They review the synced metrics and lineage, promote to production, and set the approval status to Approved . The following figure shows the current lifecycle status of the registered version in the SageMaker AI Model Registry. What happens after approval is covered in the From approval to deployment section. Topology 2: Hub-and-spoke hybrid governance Some regulated organizations treat the hub as a production-grade account and do not want data scientists writing into it, even indirectly. For these customers, we recommend a hybrid topology that keeps each development account self-contained. The following figure illustrates the workflow: 1. A data scientist registers a model against the development account’s own MLflow app. 2. Automatic sync registers it into the development Model Registry, local to that account. The hub is untouched. 3. The development account’s model owner reviews and approves the model in the development registry, which triggers the copy workflow. 4. The workflow copies the approved model into the hub cross-account: it replicates the model artifacts into a hub-owned bucket, rewrites the inference specification, and registers the package in the hub’s shared destination group. 5. The governance officer in the hub re-validates the copied model and approves it as the central governance record. 6. An ML engineer deploys the locally approved model to an endpoint in the spoke, typically through a CI/CD pipeline. No cross-account artifact is needed at deploy time. Administrator: Expose the hub destination group one-time The hub administrator creates a destination Model Package Group, attaches a resource policy allowing the development account to CreateModelPackage into it, and RAM-shares it with the AllowRegister managed permission. The administrator also creates or designates a hub-owned artifact bucket that the copy workflow writes model artifacts into. The bucket policy grants the development account s3:PutObject and s3:ListBucket so the copy can run under the development account’s role. The AWS managed framework images used in this post are public per-Region, so no container image replication is needed. If your container image lives in a private repository in the development account, replicate the image to the hub or add an Amazon Elastic Container Registry Amazon ECR repository policy. Data scientist: Register in the development account, in a notebook Identical developer experience to Part 1, entirely inside the development account: the data scientist registers a model in the development MLflow app, which automatically syncs the registered model into the development Model Registry. The hub is untouched. Development account model owner: Approve locally The model owner reviews the candidate in the development registry and approves it. This local approval is the trigger for promotion to the hub, as shown in the following figure: Approval-triggered copy, then governance officer re-validates in the hub The approval triggers a workflow an Amazon EventBridge rule on the Model Package state change in production that copies the approved package into the hub. The copy step replicates the model artifacts into a hub-owned bucket, rewrites the inference specification to point at the hub copy, and recreates the package in the hub with a pointer back to the source. The result is self-contained: the hub package has no runtime dependency on the development account. The copy step performs three actions. It replicates the model artifacts from the development bucket into the hub-owned bucket, and rewrites the inference specification so it points at the hub copy. Finally, it registers the package in the hub destination group with CustomerMetadataProperties recording the source package ARN and account. The result is self-contained: the hub package has no runtime dependency on the development account. See the repository for the actual implementation. The governance officer sees the copied package in the hub’s Studio, with provenance metadata pointing back to the source, re-validates it, and approves it independently. An example of the metadata a governance officer can see is shown in the following figure. From approval to deployment Across both topologies, the approval in the registry is the gate, not the deployment itself. The approval status change emits an event to Amazon EventBridge /eventbridge/ , and you consume that event to trigger a CI/CD deployment pipeline: an Amazon EventBridge rule matching ModelApprovalStatus: Approved starts a pipeline that creates or updates the endpoint from the approved Model Package. Where the pipeline runs and where the endpoint lands are organizational choices. A common arrangement runs the pipeline in a shared-services account and deploys into a dedicated deployment account. Simpler setups run both in the development spoke account, which is what the accompanying samples demonstrate. Whichever account deploys needs read access to the model artifacts. In the central topology, that means the hub artifact bucket see the preceding administrator prerequisite . In the hybrid topology, the hub copy is already self-contained. One consideration when introducing a separate deployment account: model lineage stays in the account where the sync recorded it. Plan your governance reviews around the hub or development registry rather than expecting lineage to follow the model into the deployment account. Choosing a topology The topologies trade account isolation against operational simplicity. The following table includes the single-account setup from Part 1 for comparison. | Consideration | Single-account Part 1 | Hub-and-spoke central | Hub-and-spoke hybrid | | Accounts | One | Hub plus spokes | Hub plus spokes | | Who writes to the hub | Not applicable | Spoke, through the shared MLflow app | No one. Only approved copies | | Governance boundary | IAM role | Account plus AWS RAM | Account plus approval-triggered copy | | Best fit | Small teams, early projects | Central governance with shared tooling | Regulated environments needing hub isolation | | Operational cost | Lowest | Moderate | Highest | Across the three, the building blocks are the same: automatic registration produces the synchronized metadata, the staging construct expresses the lifecycle, and IAM condition keys and resource tags enforce the approval gate. What changes is where the account boundary sits and how a model crosses it. Clean up To avoid ongoing charges, remove the resources created during this walkthrough. The accompanying repository includes cleanup scripts for each topology that automate most of these steps. If you prefer to clean up manually, remove resources in the following order to respect dependencies: 1. Delete the Amazon SageMaker AI endpoint and endpoint configuration in the spoke account. The endpoint incurs per-instance-hour charges while running. 2. Delete the Amazon SageMaker AI model created in the spoke for deployment. 3. Remove the AWS RAM resource shares. Delete the Model Package Group share hub → spoke and the MLflow app share hub → spoke . In the spoke account, the corresponding shared resources disappear automatically after the share is deleted. 4. Delete Model Package versions and the Model Package Group in the hub account. Remove all versions first, then delete the group. 5. Delete the MLflow registered model in the hub’s MLflow app. 6. Remove the cross-account bucket policy on the hub’s artifact-store bucket, or delete the bucket if it was created solely for this walkthrough. 7. Delete the CloudFormation stacks in both accounts to remove the Amazon SageMaker AI domains, MLflow apps, execution roles, and artifact-store buckets. If stack deletion stalls on the Studio domain, delete any running Studio applications and spaces in the domain first, then retry. Conclusion Cross-account governance builds on the same automatic model registration introduced in Part 1 /blogs/machine-learning/govern-models-with-mlflow-and-amazon-sagemaker-ai-model-registry-sync-part-1/ : the sync produces synchronized metadata, the staging construct expresses the lifecycle, and IAM condition keys and resource tags enforce the approval gate. The hub-and-spoke central topology gives governance officers one control point for candidates from every development account, at the cost of allowing spokes to write into the hub through the shared MLflow app. The hybrid topology removes even that write path for regulated environments: development accounts stay self-contained, and only approved, self-contained copies cross into the hub. In both cases, approval events flow to Amazon EventBridge, where a CI/CD pipeline turns the governance decision into a deployment. To get started, review the documentation for automatic model registration https://docs.aws.amazon.com/sagemaker/latest/dg/mlflow-track-experiments-model-registration.html , then work through the cross-account notebooks in the GitHub repository https://github.com/aws-samples/genai-ml-platform-examples/tree/main/operations/sagemaker-mlflow-model-registry-goverance . For more information, refer to the following resources: Acknowledgement Special thanks to Rahul Kharse and Siamak Nariman for their contribution.