Deploying Kubeflow as an Azure ML Alternative

A developer published a guide for deploying Kubeflow on Kubernetes as a self-hosted alternative to Azure Machine Learning, covering installation via Kustomize manifests, notebook environments, pipeline orchestration, distributed training with the Trainer v2 API, model serving through KServe, and hyperparameter tuning with Katib. The walkthrough maps each Azure ML capability to its open-source Kubeflow counterpart, arguing that self-hosting eliminates per-minute compute charges and keeps data within the operator's own cluster. It also addresses RBAC, object storage integration, and migration considerations for existing Azure ML workflows.

Azure Machine Learning is Microsoft's cloud-native machine learning platform that provides experiment tracking, managed compute, pipelines, a model registry, and model serving as hosted services within the Azure ecosystem, but it ties teams to Azure-specific APIs, managed compute pricing, and Microsoft's tooling. Kubeflow https://www.kubeflow.org/ is an open-source machine learning platform built on Kubernetes that provides self-hosted alternatives to Azure ML capabilities through modular, portable components, letting organizations retain complete control over infrastructure, data residency, scalability, and operational costs. This guide deploys Kubeflow on a Kubernetes cluster as a self-managed replacement for Azure Machine Learning, covering installation with Kustomize manifests, notebook setup, ML pipeline orchestration, distributed training with the Trainer v2 API, model serving through KServe, hyperparameter optimization using Katib, RBAC and access management, object storage integration, and migration considerations for existing Azure ML workflows. By the end, you'll have a self-hosted ML platform running notebooks, pipelines, distributed training, model serving, and automated hyperparameter tuning. Azure ML and Kubeflow provide comparable ML platform capabilities, but they differ in deployment and operational models. Azure ML delivers these as fully managed services within the Azure ecosystem, while Kubeflow provides equivalent open-source components that run on any Kubernetes cluster. The following table maps each Azure ML feature to its Kubeflow counterpart. | Azure Machine Learning | Kubeflow Equivalent | Description | |---|---|---| | Azure ML Notebooks | Kubeflow Notebooks | Interactive development environments with JupyterLab, VS Code, and RStudio | | Azure ML Jobs | Kubeflow Trainer | Distributed training for PyTorch, DeepSpeed, MLX, JAX, and XGBoost workloads | | Azure ML Pipelines | Kubeflow Pipelines KFP | Directed Acyclic Graph DAG based ML workflow orchestration | | Azure ML Model Registry | Kubeflow Model Registry | Versioned model artifact management with metadata tracking | | Azure ML Endpoints | KServe | Serverless model serving with autoscaling and canary deployments | | Azure ML Experiments | Katib | Automated hyperparameter tuning with multiple search algorithms | Self-hosting with Kubeflow eliminates per-minute compute charges, keeps all data within your own cluster, runs on any cloud provider or on-premises hardware, and allows complete customization of every component. Before you begin, you need to: StorageClass that is configured in your cluster for provisioning persistent volumes. Kubeflow uses Kustomize to deploy its components as Kubernetes resources. The official kubeflow/manifests repository contains all component manifests that are organized under common/ for shared infrastructure services such as Istio, cert-manager, and Dex, and under applications/ for Kubeflow-specific applications such as Pipelines, Notebooks, and KServe. The following steps clone the Kubeflow manifests repository and deploy all components to the cluster. 1. Verify the Kubernetes cluster connection: bash $ kubectl cluster-info 2. Check the Kubernetes server version: bash $ kubectl version Verify that the Server Version field shows version 1.31 or later. 3. Clone the official Kubeflow manifests repository: bash $ git clone https://github.com/kubeflow/manifests.git 4. Switch to the manifests directory: bash $ cd manifests 5. Check out the latest stable release tag: bash $ git checkout 26.03 6. Deploy all Kubeflow components: The command uses a bounded retry loop that attempts the installation up to 5 times, which accommodates the time that Kubernetes CRDs and webhooks need to register before dependent resources apply. The loop exits automatically after a successful apply or after reaching the retry limit. bash $ for i in 1 2 3 4 5; do kustomize build example | kubectl apply --server-side --force-conflicts -f - && break || { echo "Attempt $i failed, retrying in 30s..."; sleep 30; }; done The first one or two attempts may output errors about CRDs or webhooks not being established. These errors are expected and resolve on subsequent attempts after the CRDs register. The loop exits automatically when the apply succeeds, which typically happens on the second or third attempt. The full installation takes approximately 10 to 15 minutes after the final successful apply for all pods to reach a Running state. The --server-side --force-conflicts flags are required because some Kubeflow CRDs exceed the annotation size limit that standard kubectl apply supports. The default installation uses the email user@example.com and password 12341234 . Change these credentials before exposing Kubeflow to any network. See the Set Up Access Control section later in this article for instructions. After the deployment completes, verify that all Kubeflow components are running and the CRDs are registered. 1. Check that all pods in the kubeflow namespace reach a Running state: bash $ kubectl get pods -n kubeflow --field-selector=status.phase =Succeeded Verify that all listed pods display a Running status with all containers ready. If any pods show CrashLoopBackOff or Pending , check their logs with kubectl logs -n kubeflow POD-NAME and verify that the cluster meets the minimum resource requirements. 2. Check that the Istio ingress gateway service is running: bash $ kubectl get svc istio-ingressgateway -n istio-system Verify that the service appears in the output. 3. Check that Kubeflow and its component CRDs are registered: bash $ kubectl get crd | grep -E "kubeflow|kserve|katib|istio|knative|trainer" | wc -l The output displays the count of registered CRDs across Kubeflow and its components. A count of 40 or more indicates a complete installation. Kubeflow uses profiles to provide namespace-level isolation for each user. The default installation does not automatically provision a user namespace, so you need to create one manually. 1. Create a new file called user-profile.yaml : bash $ nano user-profile.yaml 2. Add the following configuration: apiVersion: kubeflow.org/v1 kind: Profile metadata: name: kubeflow-user-example-com spec: owner: kind: User name: user@example.com Save and close the file. 3. Apply the profile manifest: bash $ kubectl apply -f user-profile.yaml This command creates an isolated namespace called kubeflow-user-example-com with default Role-Based Access Control RBAC policies and a service account for the default user. 4. Verify that the namespace exists: bash $ kubectl get namespace kubeflow-user-example-com 5. Verify that the default service account exists: The service account takes a few seconds to provision after the profile is created. Wait 10 seconds before running this command. bash $ kubectl get serviceaccount default-editor -n kubeflow-user-example-com Kubeflow components such as Notebooks, Pipelines, and the Model Registry require persistent storage. The cluster needs a default StorageClass to dynamically provision Persistent Volume Claims PVCs . 1. Verify that a default StorageClass exists: bash $ kubectl get storageclass The default StorageClass shows default next to its name. If no default exists, set one by annotating an existing StorageClass . Replace STORAGE-CLASS-NAME with the name of an existing StorageClass from the output above. bash $ kubectl patch storageclass STORAGE-CLASS-NAME -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}' Kubeflow Notebooks provides managed JupyterLab, VS Code, and RStudio environments that run as Kubernetes pods with direct access to cluster resources, GPUs, and persistent storage. This component serves as a self-hosted alternative to Azure Machine Learning notebooks and compute instances. Set up port forwarding and log in to the Kubeflow Central Dashboard. 1. Set up port forwarding to access the Kubeflow Central Dashboard: bash $ kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80 http://localhost:8080 in a web browser. The Kubeflow login screen appears. Click user@example.com 12341234 kubeflow-user-example-com from the namespace dropdown at the top. Launch a new notebook server from the Kubeflow dashboard. ml-workspace in the 0.5 and 1 . 5Gi . This volume persists data across notebook restarts. Running state. The status indicator turns green when the notebook is ready. Open the JupyterLab interface and verify that ML libraries are accessible. Click Connect next to the notebook server name. A new tab opens with the JupyterLab interface. Click Python 3 ipykernel under the Notebook section in the launcher to create a new notebook. Paste the following code into a cell and press Shift+Enter to run it. python import numpy as np from sklearn.ensemble import RandomForestClassifier from sklearn.model selection import train test split X = np.random.randn 1000, 10 y = X :, 0 + X :, 1 0 .astype int X train, X test, y train, y test = train test split X, y, test size=0.2 model = RandomForestClassifier n estimators=100 model.fit X train, y train print f"Accuracy: {model.score X test, y test :.4f}" The cell outputs an accuracy score such as Accuracy: 0.9750 . Kubeflow Pipelines is a workflow orchestration platform for building, automating, and managing machine learning pipelines as Directed Acyclic Graphs DAGs . Each step in a pipeline executes within its own containerized environment, improving reproducibility, portability, and experiment versioning across ML workflows. It acts as an open-source, self-hosted alternative to Azure Machine Learning pipelines and workflow orchestration features. Install the KFP SDK, define a three-step pipeline, compile it, and submit a run through the internal API. The following steps use both the notebook terminal and the cluster management terminal where specified. 2. Create the pipeline definition file: bash $ nano sample pipeline.py 3. Add the following configuration: python from kfp import dsl, compiler @dsl.component base image="python:3.11-slim" def preprocess - str: import json data = {"samples": 1000, "features": 10, "status": "preprocessed"} return json.dumps data @dsl.component base image="python:3.11-slim" def train input data: str - str: import json data = json.loads input data result = {"model": "random forest", "accuracy": 0.95, "input": data} return json.dumps result @dsl.component base image="python:3.11-slim" def evaluate input data: str : import json result = json.loads input data print f"Model: {result 'model' }, Accuracy: {result 'accuracy' }" @dsl.pipeline name="sample-ml-pipeline" def ml pipeline : preprocess task = preprocess train task = train input data=preprocess task.output evaluate input data=train task.output compiler.Compiler .compile ml pipeline, "pipeline.yaml" print "Pipeline compiled successfully" 4. Compile the pipeline to generate the YAML definition: bash $ python3 sample pipeline.py 5. Switch to the cluster management terminal and create the authorization policy manifest: This policy allows the notebook namespace to call the Kubeflow Pipelines API through the Istio service mesh. bash $ nano allow-pipeline-access.yaml 6. Add the following configuration: apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: allow-notebook-to-pipeline namespace: kubeflow spec: selector: matchLabels: app: ml-pipeline rules: - from: - source: namespaces: "kubeflow-user-example-com" 7. Apply the authorization policy: bash $ kubectl apply -f allow-pipeline-access.yaml 8. Switch to the notebook terminal and upload the compiled pipeline to Kubeflow Pipelines through the internal API: bash $ curl -s -F "uploadfile=@pipeline.yaml" -H "kubeflow-userid: user@example.com" http://ml-pipeline.kubeflow.svc.cluster.local:8888/apis/v2beta1/pipelines/upload The command returns a JSON response that contains the pipeline id . Note this value for the next step. 9. Create an experiment to organize pipeline runs: bash $ curl -s -X POST -H "Content-Type: application/json" -H "kubeflow-userid: user@example.com" http://ml-pipeline.kubeflow.svc.cluster.local:8888/apis/v2beta1/experiments -d '{"display name":"default","namespace":"kubeflow-user-example-com"}' The command returns a JSON response that contains the experiment id . Note this value for the next step. 10. Start a pipeline run: Replace PIPELINE-ID and EXPERIMENT-ID with the values from the previous steps. bash $ curl -s -X POST -H "Content-Type: application/json" -H "kubeflow-userid: user@example.com" http://ml-pipeline.kubeflow.svc.cluster.local:8888/apis/v2beta1/runs -d '{"display name":"test-run","experiment id":"EXPERIMENT-ID","pipeline version reference":{"pipeline id":"PIPELINE-ID"},"runtime config":{}}' The pipeline run status is visible from the command line and from the Kubeflow dashboard. The dashboard provides a graph view that shows each step's completion state. 1. Verify that the workflow completed: bash $ kubectl get workflows -n kubeflow-user-example-com Verify that the STATUS column shows Succeeded . preprocess , train , and evaluate steps each marked with a green checkmark when the run completes successfully. Kubeflow Trainer v2 provides a unified TrainJob API for running distributed training jobs across frameworks including PyTorch, DeepSpeed, MLX, JAX, and XGBoost. The Trainer uses ClusterTrainingRuntime resources that define pre-configured runtime environments, which separates infrastructure configuration from training logic. Kubeflow Trainer replaces Azure ML Jobs with native Kubernetes-based distributed training. Create and deploy a distributed PyTorch training job that uses the torch-distributed runtime. Run the following commands from the cluster management terminal. 1. Create the TrainJob manifest: bash $ nano trainjob.yaml apiVersion: trainer.kubeflow.org/v1alpha1 kind: TrainJob metadata: name: pytorch-training namespace: kubeflow-user-example-com spec: runtimeRef: name: torch-distributed trainer: image: ghcr.io/kubeflow/katib/pytorch-mnist-cpu:v0.19.0 numNodes: 2 resourcesPerNode: requests: cpu: "500m" memory: "1Gi" limits: cpu: "1" memory: "2Gi" Save and close the file. The runtimeRef field references the torch-distributed ClusterTrainingRuntime, which configures the PyTorch distributed training environment. The numNodes field specifies the number of training nodes that Kubeflow provisions for the job. 3. Apply the training manifest: bash $ kubectl apply -f trainjob.yaml Check the training job status from the cluster management terminal. 1. Verify the TrainJob status: bash $ kubectl get trainjob -n kubeflow-user-example-com The STATE column shows Complete when training finishes successfully. KServe https://kserve.github.io/website/ provides a Kubernetes CRD called InferenceService for deploying, scaling, and managing ML model endpoints. It supports serverless inference with autoscaling from zero, canary rollouts, and multi-model serving across frameworks including TensorFlow, PyTorch, scikit-learn, XGBoost, and ONNX. KServe also supports deploying models directly from Hugging Face Hub using the hf:// URI schema and from the Kubeflow Model Registry using the model-registry:// protocol. KServe is included in the Kubeflow installation and replaces Azure ML Endpoints. Deploy a pre-trained scikit-learn model and expose it as a serving endpoint. Run the following commands from the cluster management terminal. 1. Create the model serving manifest: bash $ nano sklearn-iris.yaml apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: sklearn-iris namespace: kubeflow-user-example-com annotations: sidecar.istio.io/inject: "false" spec: predictor: model: modelFormat: name: sklearn storageUri: "gs://kfserving-examples/models/sklearn/1.0/model" resources: requests: cpu: 100m memory: 256Mi limits: cpu: "1" memory: 1Gi 3. Apply the InferenceService manifest: bash $ kubectl apply -f sklearn-iris.yaml 4. Wait for the InferenceService to become ready: bash $ kubectl get inferenceservice sklearn-iris -n kubeflow-user-example-com -w The READY column changes to True when the model is loaded and serving. Press Ctrl+C to stop watching. Navigate to KServe Endpoints in the Kubeflow dashboard sidebar to view the deployed model. Send a test inference request to verify that the model is serving predictions. 1. Run the following command from the Kubeflow notebook terminal, which has direct access to the cluster-internal service endpoint: bash $ curl -s --max-time 30 -H "Content-Type: application/json" http://sklearn-iris-predictor-00001-private.kubeflow-user-example-com.svc.cluster.local/v1/models/sklearn-iris:predict -d '{"instances": 6.8, 2.8, 4.8, 1.4 , 6.0, 3.4, 4.5, 1.6 }' The response returns predicted class labels. {"predictions": 1, 1 } Katib https://www.kubeflow.org/docs/components/katib/ is the Kubeflow component that provides automated hyperparameter tuning and neural architecture search. It supports multiple search algorithms including random search, grid search, Bayesian optimization, Tree-structured Parzen Estimator TPE , and CMA Evolution Strategy. Katib replaces Azure ML Experiments and Automatic Model Tuning with a Kubernetes-native solution. Katib experiments are defined and submitted through the Katib Python SDK https://www.kubeflow.org/docs/components/katib/getting-started/ from a JupyterLab notebook cell. The SDK creates the experiment resource on the cluster and manages trial pod configuration and metrics collection automatically. 1. Open a terminal in JupyterLab by clicking File New Terminal and install the Katib Python SDK: bash $ pip install kubeflow-katib python import kubeflow.katib as katib def objective parameters : import time time.sleep 5 result = 4 int parameters "a" - float parameters "b" 2 print f"result={result}" parameters = { "a": katib.search.int min=10, max=20 , "b": katib.search.double min=0.1, max=0.2 } katib client = katib.KatibClient namespace="kubeflow-user-example-com" name = "tune-experiment" katib client.tune name=name, objective=objective, parameters=parameters, objective metric name="result", objective type="maximize", algorithm name="random", max trial count=4, parallel trial count=2, resources per trial={"cpu": "1", "memory": "1Gi"}, The tune method creates a Katib experiment that runs 4 trials 2 in parallel using random search. The cell output includes a Katib Experiment tune-experiment link here line. Click here to open the experiment directly in the Katib Experiments tab and monitor trial progress. The experiment status turns green when all trials complete. katib client.wait for experiment condition name=name print katib client.get optimal hyperparameters name Kubeflow uses Dex as its OpenID Connect OIDC identity provider and Istio for network-level authorization. Each user gets an isolated namespace, which is called a profile, with its own resources, secrets, and RBAC policies. The Dex ConfigMap uses hashFromEnv: DEX USER PASSWORD to read the password hash from an environment variable rather than storing it directly in the ConfigMap. To change the default password, update the Secret that provides this environment variable to the Dex pod. 1. Install the bcrypt Python package to generate a password hash: bash $ pip install bcrypt 2. Generate a bcrypt hash for the new password. Replace YOUR-SECURE-PASSWORD with the password you want to set: python $ python3 -c "import bcrypt; print bcrypt.hashpw b'YOUR-SECURE-PASSWORD', bcrypt.gensalt .decode " Copy the output hash for use in the next step. 3. Update the dex-passwords Secret with the new hash. Replace GENERATED-BCRYPT-HASH with the hash output from the previous step: bash $ kubectl create secret generic dex-passwords -n auth \ --from-literal=DEX USER PASSWORD='GENERATED-BCRYPT-HASH' \ --dry-run=client -o yaml | kubectl apply -f - The command outputs a warning about a missing annotation. This is expected because dex-passwords was created by Kubeflow without --save-config . Verify that the output ends with secret/dex-passwords configured . 4. Restart the Dex deployment to apply the changes: bash $ kubectl rollout restart deployment dex -n auth staticPasswords list or connector entries in the Dex ConfigMap. See the Each additional user needs a profile that follows the same manifest structure as the default user profile. Replace the metadata.name and owner.name fields with the new user's details, then apply the manifest with kubectl apply -f . To give an existing user access to another user's namespace without creating a separate profile, navigate to the target namespace in the Kubeflow dashboard namespace dropdown. Click Manage Contributors in the left sidebar and enter the user's email address. ML workflows generate large artifacts including trained models, pipeline outputs, datasets, and logs. Kubeflow uses SeaweedFS as its default S3-compatible object storage backend for artifact persistence. KServe also supports S3-compatible storage for loading model artifacts. The default Kubeflow installation deploys SeaweedFS in the kubeflow namespace with pre-configured credentials. Verify that the storage deployment is running. 1. Check the SeaweedFS pod status: bash $ kubectl get pods -n kubeflow -l app=seaweedfs 2. For production deployments, replace SeaweedFS with an external S3-compatible object storage service: Replace YOUR-ACCESS-KEY and YOUR-SECRET-KEY with the access key and secret key for your storage service, then update the mlpipeline-minio-artifact secret with the new credentials. bash $ kubectl create secret generic mlpipeline-minio-artifact -n kubeflow --from-literal=accesskey=YOUR-ACCESS-KEY --from-literal=secretkey=YOUR-SECRET-KEY --dry-run=client -o yaml | kubectl apply -f - Configure KServe to access SeaweedFS for loading model artifacts stored in the cluster. 1. Create a storage secret for KServe model storage: bash $ nano s3-storage-secret.yaml 2. Add the following configuration. Replace ACCESS-KEY and SECRET-ACCESS-KEY with any strong keyword: apiVersion: v1 kind: Secret metadata: name: s3-storage-secret namespace: kubeflow-user-example-com annotations: serving.kserve.io/s3-endpoint: "seaweedfs.kubeflow:8333" serving.kserve.io/s3-usehttps: "0" type: Opaque stringData: AWS ACCESS KEY ID: "ACCESS-KEY" AWS SECRET ACCESS KEY: "SECRET-ACCESS-KEY" 3. Apply the storage secret: bash $ kubectl apply -f s3-storage-secret.yaml Migrating from Azure ML to Kubeflow involves exporting existing assets and mapping each Azure ML component to its Kubeflow equivalent. Notebooks transfer without format changes since both platforms use the standard Jupyter notebook format. Training scripts, pipelines, and experiments require rewriting to replace Azure ML SDK calls with Kubeflow and KFP SDK equivalents. Export Notebooks: Download Azure ML Studio notebooks as .ipynb files from the console or by using the Azure CLI. Upload them directly to Kubeflow Notebook servers, since both platforms use standard Jupyter notebook format. Update any azure.ai.ml SDK calls that rely on Azure ML-specific APIs. Convert Training Scripts: Azure ML training scripts that use azure.ai.ml SDK job and command patterns such as command ... , MLClient ... , ScriptRunConfig ... , or framework-specific job configurations need to be converted into standard framework training scripts for Kubernetes-based execution. Replace Azure ML-specific environment variables, datastore mounts, and output paths such as AZUREML MODEL DIR , ./outputs , and Azure ML input/output bindings with Kubernetes volume mount paths. Package the training code into container images and reference them in TrainJob manifests. Migrate Pipelines: Azure ML Pipelines are defined using the azure.ai.ml SDK and need rewriting with the KFP SDK. Azure ML Pipelines use Python-based DAG definitions via the azure.ai.ml SDK; replace it with KFP's @dsl.pipeline decorated functions. Export Models: Download trained model artifacts from Azure Machine Learning model registries, datastores, or blob storage using the Azure CLI or azure.ai.ml SDK. Upload them to the object storage backend connected to Kubeflow and update the storageUri field in KServe InferenceService manifests to reference the new storage location. KServe supports the same model formats commonly used in Azure ML deployments, including TensorFlow SavedModel, TorchScript, ONNX, and scikit-learn pickle models, without requiring additional conversion. Migrate Experiments: Export Azure ML experiment tracking data using the MLflow SDK's mlflow.artifacts.download artifacts API call. For hyperparameter tuning, recreate tuning jobs as Katib experiments with equivalent search spaces and objective metrics using the Katib Python SDK. For the full guide with additional tips, visit the original article on Vultr Docs https://docs.vultr.com/how-to-deploy-kubeflow-as-an-azure-ml-alternative .