Production MLOps at GS Caltex with Amazon SageMaker Canvas, part 1

GS Caltex, the South Korean energy and chemical company, automated the handoff of models built in Amazon SageMaker Canvas into Amazon SageMaker Pipelines, cutting model deployment time from about a week to a few hours, according to an AWS blog post. GS Caltex runs batch inference every 30 minutes instead of continuously running endpoints, which the company reports has meaningfully lowered its inference costs. Process engineers build and validate models in SageMaker Canvas without code, while ML engineers configure SageMaker Pipelines workflows for deployment and scheduled inference, with Amazon EventBridge triggering each run and results stored in Amazon S3.

Artificial Intelligence https://aws.amazon.com/blogs/machine-learning/ Production MLOps at GS Caltex with Amazon SageMaker Canvas, part 1 GS Caltex https://www.gscaltex.com/en uses Amazon SageMaker Canvas https://aws.amazon.com/sagemaker/canvas/ and Amazon SageMaker Pipelines https://aws.amazon.com/sagemaker/ai/pipelines/ to connect process engineers’ models to production workflows. The South Korean energy and chemical company divides model development and operations between engineers with complementary skills. Process engineers build and validate models in SageMaker Canvas without writing code, drawing on their process knowledge. Machine learning ML engineers configure SageMaker Pipelines workflows for deployment and scheduled inference. Moving assets from SageMaker Canvas into SageMaker Pipelines still required integration work. ML engineers had to set data paths for production, supply the model configuration, and connect preprocessing and inference jobs. GS Caltex automated this handoff while reusing the assets created in SageMaker Canvas. GS Caltex reports that its ML engineers now deploy models within a few hours, compared with about a week before. The company runs batch inference every 30 minutes, reducing the need for continuously running endpoints. GS Caltex reports that this has meaningfully lowered its inference costs. This post explains how GS Caltex connects SageMaker Canvas to SageMaker Pipelines and reuses validated preprocessing flows and models in production. Solution overview The machine learning operations MLOps workflow has two paths: preparing a new model version for deployment, and running batch inference with the deployed model. From a validated model to a repeatable workflow You can use SageMaker Canvas to prepare data visually, build models, and generate batch predictions. You can also register validated models in Amazon SageMaker Model Registry from SageMaker Canvas. GS Caltex built a separate handoff process to turn these assets into executable SageMaker Pipelines steps. Process engineers understand the variables and data behind a process and assess whether a model’s predictions are useful. ML engineers turn the resulting assets into production jobs and manage model versions and execution history. | Area | What is prepared in SageMaker Canvas | What is managed in the production workflow | | Data preparation | A preprocessing flow built with Data Wrangler | Preprocessing jobs in Amazon SageMaker Processing | | Model development | A trained model and evaluation results | Model versions and approval status in Model Registry | | Prediction | Validation of model predictions | Batch inference every 30 minutes using an approved model | | Operations | Development and validation of individual models | Step execution history, stored results, and failure handling | Reference architecture The following diagram shows how process engineers hand off SageMaker Canvas assets to ML engineers for deployment and scheduled inference. ML engineers use Amazon Simple Storage Service Amazon S3 for preprocessing flows and Model Registry for model versions, then schedule pipeline runs through Amazon EventBridge. ML engineers receive the preprocessing flow and model from SageMaker Canvas and review the data formats and prediction results. After that review, they approve the model in Model Registry and deploy it. This preparation takes place when a new model version is deployed. Subsequent inference runs reuse the deployed model. Amazon EventBridge starts each scheduled run. SageMaker Pipelines prepares the input data from Amazon Simple Storage Service Amazon S3 , runs preprocessing and SageMaker Batch Transform https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html , and stores the results in S3. ML engineers can inspect step status and Amazon CloudWatch logs to monitor the workflow. The production pipeline reuses the preprocessing logic and trained model created in SageMaker Canvas. ML engineers configure the input and output paths, set the order of execution, and write the code that connects approval and scheduling to the workflow. Prerequisites Before you begin, prepare the following: - A model built and evaluated in SageMaker Canvas https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-custom-models.html , along with its Data Wrangler preprocessing flow https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-data-prep.html - CSV data for inference and an S3 bucket to store the files. - A SageMaker execution role https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html and an Amazon EventBridge role with permission to start the pipeline https://docs.aws.amazon.com/sagemaker/latest/dg/pipeline-eventbridge.html - A Python environment with Boto3 and SageMaker Python SDK 2.x installed. The full reference implementation is available in the GitHub repository https://github.com/aws-samples/sample-canvas-to-pipelines-mlops . The code examples in this post focus on the connections between components. The repository uses a sample loan classification model, so adjust the configuration to match your model’s input and output formats. Technical implementation The following steps cover preprocessing reuse, model approval and deployment, pipeline construction, and scheduled inference. 1. Run the SageMaker Canvas preprocessing flow with SageMaker Processing A preprocessing flow created with Data Wrangler in SageMaker Canvas records the data sources and transformation steps. To set up the reference implementation, export data to Amazon S3 https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-export-data.html using SageMaker Processing, then retrieve the job settings and the .flow file used by that job. In the export settings, turn off Auto job configuration and select SageMaker Processing . After the export starts, open the Processing job details on the Amazon SageMaker AI console. Record the container image URI, the S3 URI of the input named flow , the execution role, and the output settings. Use these values to configure the pipeline step that runs the .flow file. The SageMaker execution role must have permissions to read the required S3 data, write the results, and access the container image. Choose the instance type and count based on the volume of data you need to process. Keep the preprocessing logic validated by the process engineer, and adjust the data sources, paths, and other settings for the production environment. Check the container image and output node against the original job configuration. | Item to check | What to do | | Input location | Check the S3 URI referenced by the data source in the .flow file. Uploading raw data to another location does not update this reference. | | Container image | Use the image from the Processing job created during the S3 export, and pin a version that reproduces the validated transformations. | | Output ID | Use the output node and output name from the exported configuration. If the required format is {node id}.{output name} , match it exactly. | | Learned transformation values | Preserve values learned during training, such as scaling and encoding parameters. Do not refit them on each inference batch. | | Validation | Compare the columns, data types, and transformed values produced by SageMaker Canvas and Processing for the same input. | Some .flow files contain execution metadata such as internal metadata.dw job.container uri . Check the actual Processing job settings rather than assuming that every export contains this field. If you move the workflow to another AWS Region, verify that the container image is available there. The reference implementation separates intermediate and final outputs by execution under runs/<execution-id / . Its PrepareInput step writes data to a fixed S3 location referenced by the .flow file. Prevent executions that share that input path from overlapping. To isolate input data for each execution as well, add a preparation step that updates the source URI in a copy of the .flow file. Declaring a SageMaker Pipelines parameter does not automatically replace a URI inside the file. 2. Handle differences between training and inference inputs The target column is one difference between training and inference data. A flow built with training data can contain transformations that expect the target column in the input schema. Inference data does not contain that known outcome. The reference implementation adds a PrepareInput step before preprocessing. If the target column is missing, this step adds it with a placeholder value. After preprocessing, DropTargetColumn removes the column. Match the placeholder’s data type and column position to the training schema, and pass only the expected features to the model in their original order. Use this approach only when the placeholder target does not affect feature values. A flow that filters rows by the target, derives features from it, or applies target encoding needs additional treatment. In those cases, separate the transformations and learned state required for inference. Validation should cover column order and count, data types, missing-value handling, and row count. After removing the target, check that the remaining columns match the feature order used during training. 3. Register, approve, and deploy the model Register the validated SageMaker Canvas model version in Model Registry https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-register-model.html . The registered version can include the model files, inference configuration, and evaluation results. ML engineers review this information before changing the approval status. The model file, such as model.tar.gz , must be used with the correct inference container and environment settings. A model package in Model Registry groups this information together. In this batch architecture, deployment creates a SageMaker model resource from an approved package for Batch Transform to reference. The following example checks a package that has already been approved, then creates the model resource. Supply model package arn , model name , and sagemaker role arn during deployment. This example uses a unique model name for each version. Model Registry records approval status, but it does not enforce an approval check on every inference request. Check that status in the deployment code and retain the selected package ARN and model name in the pipeline configuration. If approval is revoked, update the schedules and execution configurations that use the model. Registration and deployment take place when a model version is released. The pipeline that runs every 30 minutes creates Batch Transform jobs using that model, so it does not need to register the model again on each run. 4. Connect preprocessing, inference, and result assembly The inference pipeline has five steps. | Step | Function | Input and output | | PrepareInput | Add a placeholder target column if needed | New inference data - data with the target column supplied | | PreprocessNewData | Run the SageMaker Canvas preprocessing flow | Prepared input and .flow - transformed data | | DropTargetColumn | Remove the target column and CSV header | Transformed data - feature CSV without a header | | BatchPredict | Run batch inference with the approved model | Feature CSV - prediction files | | AssembleResults | Combine the preprocessed data with predictions | Preprocessed data and prediction files - final result CSV | Configure each job before connecting the steps. In the following example, prepare args , preprocess args , drop target args , and assemble args hold the step arguments returned by run calls on processors that use a PipelineSession . The processors and transformer share the pipeline session object. Configure the transformer with the previously deployed model name .