INFERNO: Kubernetes-Native AI Inference Operator - 10-Day Expert Learning Plan

A developer outlines a 10-day expert learning plan for building a Kubernetes-native AI inference operator using Kubebuilder. The plan covers required Go and Kubernetes knowledge, the problem of boilerplate in operator development, and the workflow of generating CRDs, controllers, and deployment manifests.

Project: Kubernetes-Native AI Inference Operator Duration: 10 days Level: Expert Goal: Build a working Kubernetes controller that manages AI inference workloads Go Knowledge Required: - Basic syntax variables, functions, structs - Interfaces - Error handling with error type - Goroutines basics - Package management with go mod If you're weak on these , take 1 day to review: Quick Go refresher resources - https://gobyexample.com/ 30 min - Read: https://tour.golang.org/methods/1 interfaces, 20 min - Practice: Write a simple HTTP server Kubernetes Knowledge Required: - What Pods, Deployments, Services are - How to use kubectl get, apply, describe, logs - YAML basics - What a namespace is If weak, do: Quick k8s refresher - kubectl cheatsheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/ - Deploy a simple app: kubectl apply -f nginx.yaml - Check it: kubectl get pods, kubectl logs - Clean up: kubectl delete -f nginx.yaml The Problem It Solves: Building Kubernetes operators is hard. You need: - API groups, versions, kinds - CRD manifests - Reconciliation loops - RBAC rules - Deployment manifests - Testing setup All of this is boilerplate . Kubebuilder is a scaffolding tool that generates all this for you. What Kubebuilder Does: kubebuilder init Create project skeleton kubebuilder create api Generate API + controller template make manifests Generate CRD YAML make install Install CRD into cluster make run Run controller locally make deploy Deploy controller to cluster What Kubebuilder Generates: my-operator/ ├── api/v1alpha1/ Your API definitions the YAML structure │ ├── myresource types.go Go structs that become YAML │ └── myresource webhook.go Validation webhooks advanced ├── controllers/ Your business logic │ └── myresource controller.go The reconciliation loop ├── config/ │ ├── crd/ Kubernetes CRD manifests generated │ ├── manager/ Operator deployment YAML generated │ ├── rbac/ RBAC rules generated │ └── samples/ Example CRs you create ├── main.go Operator entry point ├── Dockerfile Container image definition ├── Makefile Build commands └── go.mod Go dependencies The Workflow: You Write: Kubebuilder Generates: ┌──────────────────────────┐ ┌──────────────────────────┐ │ 1. API types Go structs │ ────→ │ 1. CRD YAML manifests │ │ 2. Controller logic │ │ 2. RBAC rules │ │ 3. Reconcile function │ │ 3. OpenAPI docs │ │ 4. Tests │ │ 4. Deployment manifest │ └──────────────────────────┘ └──────────────────────────┘ Key Files You'll Edit: — Define what users can write in YAML api/v1alpha1/myresource types.go — Write the reconciliation logic controllers/myresource controller.go — Example YAML for testing config/samples/ .yaml Everything else is generated or boilerplate. This is the heart of every Kubernetes operator: ┌─────────────────────────────────────────────┐ │ Kubernetes API Server │ │ stores all resources │ └────────────────┬────────────────────────────┘ │ │ Watch: "Tell me when anything changes" ↓ ┌─────────────────────────────────────────────┐ │ Controller your code │ │ │ │ func r MyReconciler Reconcile ... { │ │ 1. Read the current CR │ │ 2. Check what's needed │ │ 3. Create/update Deployments, Services │ │ 4. Update the CR's status │ │ 5. Return no change or Requeue │ │ } │ └────────────────┬────────────────────────────┘ │ │ Create/Update Resources ↓ ┌─────────────────────────────────────────────┐ │ Kubernetes Cluster │ │ Deployments, Services, etc. │ └─────────────────────────────────────────────┘ When does Reconcile get called? - Someone creates/updates/deletes an InferenceService - Kubernetes notifies the controller: "Something changed " - Controller calls Reconcile - Controller reads the CR and creates/updates resources What should Reconcile do? func r MyReconciler Reconcile ctx, req Result, error { // 1. Read the resource cr := &MyResource{} r.Get ctx, req.NamespacedName, cr // 2. Check: Does desired state == actual state? deployment := &appsv1.Deployment{} err := r.Get ctx, deploymentName, deployment if err = nil { // 3a. Desired state missing: CREATE it newDeployment := r.constructDeployment cr r.Create ctx, newDeployment } else { // 3b. Desired state exists: UPDATE it r.Update ctx, deployment } // 4. Update status cr.Status.Ready = true r.Status .Update ctx, cr // 5. Tell k8s "I'm done, check again later" return Result{RequeueAfter: 10 time.Second}, nil } The Goal: Make actual state match desired state. Forever. This plan assumes you have basic Go and Kubernetes knowledge. If not, take 1-2 days to refresh using the links above. You'll build INFERNO in phases: Days 1-3 : Kubernetes controller fundamentals & Kubebuilder setup Days 4-6 : Core operator logic & InferenceService CRD Days 7-9 : Reconciliation, Deployment management, and testing Day 10 : Polish, integration testing, and documentation Each day has theory + hands-on code. By Day 10, you'll have a working operator that can deploy, scale, and manage fake inference workloads. Struct tags — How Go metadata works json:"field" , kubebuilder:validation: Interfaces — The client.Client interface implements Get, Create, Update Goroutines & channels — Controller-runtime uses these under the hood Error handling — Pattern of if err = nil { return err } Package management — Using import for external libraries - Understand how Kubernetes controllers work watch → reconcile → act - Learn the controller-runtime library - Set up your first Kubebuilder project - Understand CRDs Custom Resource Definitions Kubernetes Controller Pattern: ┌─────────────────────────────────┐ │ Watch: Is desired = actual? │ │ If yes → Reconcile │ │ Update cluster state │ │ Requeue if needed │ └─────────────────────────────────┘ controller-runtime : The library that manages this loop Reconciliation : The core function that makes changes Leader election : How operators stay safe in HA setups Finalizers : Cleanup when resources are deleted Create project structure go mod init github.com/yourusername/inferno kubebuilder init --domain inferno.io --repo github.com/yourusername/inferno Create the InferenceService API kubebuilder create api --group inference --version v1alpha1 --kind InferenceService api/v1alpha1/inferenceservice types.go — Your CRD schema controllers/inferenceservice controller.go — Your reconciliation logic config/crd/ — Kubernetes CRD manifests config/manager/ — Operator deployment manifests - Project initialized with kubebuilder - InferenceService CRD created empty, will fill tomorrow - Controller file exists and compiles - Understand the reconciliation loop by reading generated code Kubebuilder Book https://book.kubebuilder.io/ — Read chapters 1-3 controller-runtime Overview https://pkg.go.dev/sigs.k8s.io/controller-runtime - Watch: "How Kubernetes Controllers Work" 10 min explainer on YouTube Struct field tags — How json:"field" controls YAML marshalling Kubebuilder validation tags — +kubebuilder:validation:Required , Enum= , Min= Pointer vs value types — When to use int32 vs int Slice operations — Working with arrays TypeName Type embedding — How metav1.TypeMeta and metav1.ObjectMeta inherit fields - Design your CRD schema the YAML users will write - Understand Go struct tags for validation & OpenAPI - Implement status subresource - Write validation rules The user wants to write: apiVersion: inference.inferno.io/v1alpha1 kind: InferenceService metadata: name: llama spec: model: uri: s3://models/llama-3 framework: vllm version: "1" replicas: min: 1 max: 5 resources: requests: cpu: "4" memory: "16Gi" gpu: "1" autoscaling: enabled: true targetConcurrency: 10 traffic: - version: "1" weight: 100 You need to translate this into Go structs. Edit api/v1alpha1/inferenceservice types.go : package v1alpha1 import corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" // ModelSpec defines the model to deploy type ModelSpec struct { // URI is the location of the model s3://, http://, local path // +kubebuilder:validation:Required URI string json:"uri" // Framework is the inference runtime vllm, triton, etc // +kubebuilder:validation:Enum=vllm;triton;fake Framework string json:"framework" // Version is the model version tag // +kubebuilder:validation:Required Version string json:"version" } // ResourceRequirements mirrors k8s resource requests type ResourceRequirements struct { CPU string json:"cpu,omitempty" Memory string json:"memory,omitempty" GPU string json:"gpu,omitempty" } // AutoscalingSpec defines HPA behavior type AutoscalingSpec struct { Enabled bool json:"enabled" TargetConcurrency int json:"targetConcurrency,omitempty" MinReplicas int json:"minReplicas,omitempty" MaxReplicas int json:"maxReplicas,omitempty" } // TrafficWeight defines canary traffic split type TrafficWeight struct { Version string json:"version" Weight int json:"weight" // 0-100 } // InferenceServiceSpec defines the desired state type InferenceServiceSpec struct { Model ModelSpec json:"model" Replicas ReplicaConfig json:"replicas" Resources ResourceRequirements json:"resources,omitempty" Autoscaling AutoscalingSpec json:"autoscaling,omitempty" Traffic TrafficWeight json:"traffic,omitempty" } // ReplicaConfig defines scaling bounds type ReplicaConfig struct { Min int json:"min" Max int json:"max" } // InferenceServiceStatus defines the observed state type InferenceServiceStatus struct { // Phase: Pending, Running, Failed Phase string json:"phase,omitempty" // ReadyReplicas: how many pods are ready ReadyReplicas int json:"readyReplicas" // Message for debugging Message string json:"message,omitempty" // LastUpdateTime when the controller last reconciled LastUpdateTime metav1.Time json:"lastUpdateTime,omitempty" // Conditions for events Conditions metav1.Condition json:"conditions,omitempty" } // +kubebuilder:object:root=true // +kubebuilder:subresource:status // +kubebuilder:resource:shortName=isvc;scope=Namespaced type InferenceService struct { metav1.TypeMeta json:",inline" metav1.ObjectMeta json:"metadata,omitempty" Spec InferenceServiceSpec json:"spec,omitempty" Status InferenceServiceStatus json:"status,omitempty" } // +kubebuilder:object:root=true type InferenceServiceList struct { metav1.TypeMeta json:",inline" metav1.ListMeta json:"metadata,omitempty" Items InferenceService json:"items" } : Generate validation, OpenAPI docs, RBAC +kubebuilder: annotations Status subresource : Separate spec desired from status actual Validation tags : Required , Enum , Min , Max — enforced at API server Conditions : Standard k8s pattern for tracking async operations - Complete inferenceservice types.go with all structs - Run make generate to create CRD manifests - CRD installs without errors: kubectl apply -f config/crd/ - Can create a test InferenceService YAML Kubebuilder CRD Tutorial https://book.kubebuilder.io/cronjob-tutorial/api-design.html Go API Conventions https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md - Kubernetes API reference for metav1.Condition Context — How context.Context controls cancellation and timeouts The — client interface Get , Create , Update methods Error wrapping — if err = nil patterns Named return types — ctrl.Result, error convention Defer statements — Cleanup code not used yet, but will on Day 7 Type assertions — Checking error types apierrors.IsNotFound err - Understand the reconciliation function signature - Learn how to watch resources and trigger reconciliation - Implement basic reconciliation logic - Set up Deployment creation from InferenceService The controller's main job: Watch InferenceService CRs Reconcile : For each CR, ensure a Deployment exists with the right spec Update status with what it found Requeue if needed e.g., "check again in 10s" Edit controllers/inferenceservice controller.go : package controllers import "context" "fmt" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/controller-runtime/pkg/predicate" inferencev1alpha1 "github.com/yourusername/inferno/api/v1alpha1" // InferenceServiceReconciler reconciles an InferenceService object type InferenceServiceReconciler struct { client.Client Scheme runtime.Scheme } // +kubebuilder:rbac:groups=inference.inferno.io,resources=inferenceservices,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=inference.inferno.io,resources=inferenceservices/status,verbs=get;update;patch // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete func r InferenceServiceReconciler Reconcile ctx context.Context, req ctrl.Request ctrl.Result, error { log := log.FromContext ctx // Step 1: Fetch the InferenceService var inferenceService inferencev1alpha1.InferenceService if err := r.Get ctx, req.NamespacedName, &inferenceService ; err = nil { if apierrors.IsNotFound err { log.Info "InferenceService not found, ignoring" return ctrl.Result{}, nil } log.Error err, "Failed to fetch InferenceService" return ctrl.Result{}, err } log.Info "Reconciling InferenceService", "name", inferenceService.Name // Step 2: Create or update Deployment deployment := &appsv1.Deployment{} deploymentName := types.NamespacedName{ Name: inferenceService.Name, Namespace: inferenceService.Namespace, } if err := r.Get ctx, deploymentName, deployment ; err = nil { if apierrors.IsNotFound err { // Create new Deployment deployment = r.constructDeployment &inferenceService if err := controllerutil.SetControllerReference &inferenceService, deployment, r.Scheme ; err = nil { log.Error err, "Failed to set controller reference" return ctrl.Result{}, err } if err := r.Create ctx, deployment ; err = nil { log.Error err, "Failed to create Deployment" return ctrl.Result{}, err } log.Info "Created Deployment", "deployment", deployment.Name } else { log.Error err, "Failed to fetch Deployment" return ctrl.Result{}, err } } else { // Update existing Deployment deployment = r.constructDeployment &inferenceService if err := r.Update ctx, deployment ; err = nil { log.Error err, "Failed to update Deployment" return ctrl.Result{}, err } log.Info "Updated Deployment", "deployment", deployment.Name } // Step 3: Update status inferenceService.Status.Phase = "Running" inferenceService.Status.ReadyReplicas = int deployment.Spec.Replicas inferenceService.Status.LastUpdateTime = &metav1.Time{Time: time.Now } if err := r.Status .Update ctx, &inferenceService ; err = nil { log.Error err, "Failed to update status" return ctrl.Result{}, err } log.Info "Reconciliation complete" return ctrl.Result{}, nil } // constructDeployment builds the Deployment spec from InferenceService func r InferenceServiceReconciler constructDeployment isvc inferencev1alpha1.InferenceService appsv1.Deployment { replicas := int32 isvc.Spec.Replicas.Min labels := map string string{ "app": isvc.Name, "operator": "inferno", } deployment := &appsv1.Deployment{ ObjectMeta: metav1.ObjectMeta{ Name: isvc.Name, Namespace: isvc.Namespace, }, Spec: appsv1.DeploymentSpec{ Replicas: &replicas, Selector: &metav1.LabelSelector{ MatchLabels: labels, }, Template: corev1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: labels, }, Spec: corev1.PodSpec{ Containers: corev1.Container{ { Name: "inference-server", Image: "inference-server:fake", // TODO: vLLM later Ports: corev1.ContainerPort{ { ContainerPort: 8000, Name: "http", }, }, Env: corev1.EnvVar{ { Name: "MODEL URI", Value: isvc.Spec.Model.URI, }, { Name: "MODEL VERSION", Value: isvc.Spec.Model.Version, }, }, Resources: corev1.ResourceRequirements{ Requests: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse isvc.Spec.Resources.CPU , corev1.ResourceMemory: resource.MustParse isvc.Spec.Resources.Memory , }, }, }, }, }, }, }, } return deployment } func r InferenceServiceReconciler SetupWithManager mgr ctrl.Manager error { return ctrl.NewControllerManagedBy mgr . For &inferencev1alpha1.InferenceService{} . Owns &appsv1.Deployment{} . // Watch Deployments we create WithEventFilter predicate.GenerationChangedPredicate{} . // Ignore status-only updates Complete r } : Called when an InferenceService changes or a watched resource changes Reconcile : Links Deployment to InferenceService for cleanup SetControllerReference : Watch Deployments we create; if one changes, re-reconcile the InferenceService Owns Status subresource : Updated separately with r.Status .Update RBAC : The +kubebuilder:rbac: comments generate Kubernetes RBAC rules - inferenceservice controller.go implements Reconcile - constructDeployment creates valid k8s Deployment specs - Controller compiles: make build - RBAC rules generate: make manifests - Understand the reconciliation loop flow Shell commands from Go — How to run kubectl and kind commands Logging — Using log.FromContext ctx and log.Info Testing with real Kubernetes — Integration testing concepts Environment setup — How KUBECONFIG and kubectl contexts work - Set up a local Kubernetes cluster with kind - Deploy your operator locally - Manually test reconciliation - Debug with logs and kubectl Create a local test environment where you can: - Deploy your operator - Create an InferenceService CR - Watch the controller reconcile it into a Deployment - Verify everything works Create a kind cluster kind create cluster --name inferno-dev Verify it works kubectl get nodes kubectl get pods -A Install your operator CRDs make install Deploy the operator make deploy Check it's running kubectl get deployment -n inferno-system kubectl logs -n inferno-system deployment/inferno-controller-manager -f Create config/samples/inference v1alpha1 inferenceservice.yaml : apiVersion: inference.inferno.io/v1alpha1 kind: InferenceService metadata: name: test-llama spec: model: uri: s3://models/llama-3 framework: fake version: "1" replicas: min: 1 max: 3 resources: cpu: "2" memory: "8Gi" gpu: "0" autoscaling: enabled: false traffic: - version: "1" weight: 100 Deploy and observe: Apply the CR kubectl apply -f config/samples/inference v1alpha1 inferenceservice.yaml Watch the controller work kubectl logs -n inferno-system deployment/inferno-controller-manager -f Check if Deployment was created kubectl get deployments kubectl get inferenceservices kubectl describe inferenceservice test-llama Check the status kubectl get inferenceservice test-llama -o yaml View controller logs kubectl logs -n inferno-system deployment/inferno-controller-manager --tail=100 -f Describe what happened kubectl describe inferenceservice test-llama kubectl describe deployment test-llama Check events kubectl get events --all-namespaces | grep inferno Interactive debugging if needed kubectl exec -it deployment/inferno-controller-manager -n inferno-system -- bash - kind cluster running locally - Operator deployed: make deploy - InferenceService CR created - Deployment automatically created from the CR - Status updated correctly - Logs show clean reconciliation Helper functions — Extracting common code into constructService Go interfaces as contracts — Services follow same pattern as Deployments Type conversion — Using intstr.FromString for port types Map creation — Building label maps for selectors - Create and manage Kubernetes Services - Expose inference endpoints - Handle traffic routing - Understand label selectors When an InferenceService is created, the operator should also create a Kubernetes Service so clients can reach the inference server. InferenceService user writes this ↓ Controller ├→ Deployment pods run the server └→ Service exposes the server Update constructDeployment and add a new constructService in your controller: // In Reconcile , after creating/updating Deployment: // Step 2b: Create or update Service service := &corev1.Service{} serviceName := types.NamespacedName{ Name: inferenceService.Name, Namespace: inferenceService.Namespace, } if err := r.Get ctx, serviceName, service ; err = nil { if apierrors.IsNotFound err { service = r.constructService &inferenceService if err := controllerutil.SetControllerReference &inferenceService, service, r.Scheme ; err = nil { log.Error err, "Failed to set service controller reference" return ctrl.Result{}, err } if err := r.Create ctx, service ; err = nil { log.Error err, "Failed to create Service" return ctrl.Result{}, err } log.Info "Created Service", "service", service.Name } else { log.Error err, "Failed to fetch Service" return ctrl.Result{}, err } } // constructService builds a Service for the InferenceService func r InferenceServiceReconciler constructService isvc inferencev1alpha1.InferenceService corev1.Service { labels := map string string{ "app": isvc.Name, "operator": "inferno", } service := &corev1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: isvc.Name, Namespace: isvc.Namespace, }, Spec: corev1.ServiceSpec{ Selector: labels, Type: corev1.ServiceTypeClusterIP, Ports: corev1.ServicePort{ { Name: "http", Port: 80, TargetPort: intstr.FromString "http" , }, }, }, } return service } Deploy updated operator make deploy Create an InferenceService kubectl apply -f config/samples/inference v1alpha1 inferenceservice.yaml Verify Service was created kubectl get svc test-llama kubectl describe svc test-llama Test from inside the cluster kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \ curl http://test-llama/health - Service created automatically with InferenceService - Service selects the right Pods - Labels and selectors match Deployment - Service exposed at DNS name <name .<namespace .svc.cluster.local - RBAC updated for Services: make manifests Conditional creation — When to create vs skip resources Pointer arithmetic — Working with int32 values Default values — Setting sensible defaults in helpers Complex struct composition — HPA specs have deeply nested structs - Create HorizontalPodAutoscaler HPA from the InferenceService - Handle min/max replicas - Understand metric-based scaling - Integrate with KEDA future enhancement When autoscaling.enabled: true , the operator should create an HPA that scales the Deployment: InferenceService spec: autoscaling: enabled: true targetConcurrency: 10 minReplicas: 1 maxReplicas: 5 ↓ HPA created ↓ Metrics Server watches CPU/memory ↓ Scales Deployment 1→5 based on load In your controller, add: // After Service creation, in Reconcile : if inferenceService.Spec.Autoscaling.Enabled { hpa := &autoscalingv2.HorizontalPodAutoscaler{} hpaName := types.NamespacedName{ Name: inferenceService.Name, Namespace: inferenceService.Namespace, } if err := r.Get ctx, hpaName, hpa ; err = nil { if apierrors.IsNotFound err { hpa = r.constructHPA &inferenceService if err := controllerutil.SetControllerReference &inferenceService, hpa, r.Scheme ; err = nil { return ctrl.Result{}, err } if err := r.Create ctx, hpa ; err = nil { log.Error err, "Failed to create HPA" return ctrl.Result{}, err } log.Info "Created HPA", "hpa", hpa.Name } else { log.Error err, "Failed to fetch HPA" return ctrl.Result{}, err } } } // constructHPA builds an HPA for the InferenceService func r InferenceServiceReconciler constructHPA isvc inferencev1alpha1.InferenceService autoscalingv2.HorizontalPodAutoscaler { minReplicas := int32 isvc.Spec.Replicas.Min maxReplicas := int32 isvc.Spec.Replicas.Max // Default to CPU-based scaling if no custom metric cpuUtilization := int32 80 hpa := &autoscalingv2.HorizontalPodAutoscaler{ ObjectMeta: metav1.ObjectMeta{ Name: isvc.Name, Namespace: isvc.Namespace, }, Spec: autoscalingv2.HorizontalPodAutoscalerSpec{ ScaleTargetRef: autoscalingv2.CrossVersionObjectReference{ APIVersion: "apps/v1", Kind: "Deployment", Name: isvc.Name, }, MinReplicas: &minReplicas, MaxReplicas: maxReplicas, Metrics: autoscalingv2.MetricSpec{ { Type: autoscalingv2.ResourceMetricSourceType, Resource: &autoscalingv2.ResourceMetricSource{ Name: corev1.ResourceCPU, TargetAverageUtilization: &cpuUtilization, }, }, }, }, } return hpa } Also update RBAC: // +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete Deploy make deploy Create an InferenceService with autoscaling cat <<EOF | kubectl apply -f - apiVersion: inference.inferno.io/v1alpha1 kind: InferenceService metadata: name: autoscale-test spec: model: uri: s3://models/test framework: fake version: "1" replicas: min: 1 max: 5 resources: cpu: "1" memory: "1Gi" autoscaling: enabled: true targetConcurrency: 10 minReplicas: 1 maxReplicas: 5 EOF Check HPA was created kubectl get hpa kubectl describe hpa autoscale-test HPA needs metrics-server; kind doesn't have it by default, so skip load testing for now - HPA created when autoscaling.enabled: true - Min/max replicas respected - CPU-based scaling configured - HPA updates when InferenceService spec changes - RBAC includes HPA permissions Receiver methods — The r Reconciler pattern object-oriented Go Slice manipulation — Appending, finding in arrays of conditions Boolean logic — Conditional field setting Time handling — time.Now and metav1.Time types Defer for cleanup — Resource management patterns - Implement proper status tracking - Use Kubernetes Conditions pattern - Handle error states gracefully - Implement retry logic with exponential backoff Track the lifecycle of an InferenceService: Creating → Pending → Ready ✓ ↓ error Failed → Retry Update your InferenceServiceStatus to track conditions: // In controllers/inferenceservice controller.go func r InferenceServiceReconciler updateStatus ctx context.Context, isvc inferencev1alpha1.InferenceService, phase string, message string error { log := log.FromContext ctx isvc.Status.Phase = phase isvc.Status.Message = message isvc.Status.LastUpdateTime = &metav1.Time{Time: time.Now } // Add condition condition := metav1.Condition{ Type: "Ready", Status: metav1.ConditionTrue, ObservedGeneration: isvc.Generation, Reason: phase, Message: message, LastTransitionTime: metav1.Time{Time: time.Now }, } if phase = "Running" { condition.Status = metav1.ConditionFalse } // Upsert condition replace if exists, append if not found := false for i, c := range isvc.Status.Conditions { if c.Type == condition.Type { isvc.Status.Conditions i = condition found = true break } } if found { isvc.Status.Conditions = append isvc.Status.Conditions, condition } if err := r.Status .Update ctx, isvc ; err = nil { log.Error err, "Failed to update status" return err } return nil } // In Reconcile , use it like: if err := r.updateStatus ctx, &inferenceService, "Running", "Deployment is ready" ; err = nil { return ctrl.Result{}, err } // For errors, retry with backoff: if err = nil { r.updateStatus ctx, &inferenceService, "Error", err.Error return ctrl.Result{RequeueAfter: 10 time.Second}, nil // Retry in 10s } kubectl apply -f config/samples/inference v1alpha1 inferenceservice.yaml Watch status in real-time kubectl get inferenceservice -o wide -w View conditions kubectl get inferenceservice test-llama -o yaml | grep -A 10 conditions Create a bad spec to trigger error kubectl apply -f - <<EOF apiVersion: inference.inferno.io/v1alpha1 kind: InferenceService metadata: name: bad-service spec: model: uri: s3://models/test framework: invalid This should fail validation version: "1" replicas: min: 1 max: 5 EOF - Status updated on each reconciliation - Conditions follow Kubernetes pattern - Error states tracked - Retry logic with backoff implemented - kubectl can show readable status Testing patterns — Table-driven tests, subtests Goroutine testing — Using Eventually for async operations Mocking with interfaces — Using client.Client interface for testing BDD style testing — Ginkgo framework Describe , It , Eventually Test fixtures — Setting up test data - Write unit tests for your controller - Write integration tests with envtest - Test edge cases and error handling - Achieve good test coverage ✓ Creating InferenceService → Deployment created ✓ Updating spec → Deployment updated ✓ Deleting InferenceService → Deployment deleted via owner ref ✓ Invalid spec → Error state ✓ Missing resources → Retry ✓ Concurrent reconciliations → No race conditions Create controllers/inferenceservice controller test.go : package controllers import "context" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" inferencev1alpha1 "github.com/yourusername/inferno/api/v1alpha1" var = Describe "InferenceServiceReconciler", func { Context "Creating an InferenceService", func { It "should create a Deployment", func { ctx := context.Background isvc := &inferencev1alpha1.InferenceService{ ObjectMeta: metav1.ObjectMeta{ Name: "test-service", Namespace: "default", }, Spec: inferencev1alpha1.InferenceServiceSpec{ Model: inferencev1alpha1.ModelSpec{ URI: "s3://models/test", Framework: "fake", Version: "1", }, Replicas: inferencev1alpha1.ReplicaConfig{ Min: 1, Max: 3, }, Resources: inferencev1alpha1.ResourceRequirements{ CPU: "1", Memory: "1Gi", }, }, } // Create the InferenceService Expect k8sClient.Create ctx, isvc .Should Succeed // Eventually, a Deployment should exist deployment := &appsv1.Deployment{} deploymentName := types.NamespacedName{Name: "test-service", Namespace: "default"} Eventually func error { return k8sClient.Get ctx, deploymentName, deployment } .Should Succeed // Verify the Deployment spec Expect deployment.Spec.Replicas .To Equal int32 1 Expect deployment.Spec.Template.Spec.Containers 0 .Name .To Equal "inference-server" } It "should create a Service", func { ctx := context.Background isvc := &inferencev1alpha1.InferenceService{ ObjectMeta: metav1.ObjectMeta{ Name: "test-svc-2", Namespace: "default", }, Spec: inferencev1alpha1.InferenceServiceSpec{ Model: inferencev1alpha1.ModelSpec{ URI: "s3://models/test", Framework: "fake", Version: "1", }, Replicas: inferencev1alpha1.ReplicaConfig{Min: 1, Max: 3}, Resources: inferencev1alpha1.ResourceRequirements{ CPU: "1", Memory: "1Gi", }, }, } Expect k8sClient.Create ctx, isvc .Should Succeed // Eventually, a Service should exist service := &corev1.Service{} serviceName := types.NamespacedName{Name: "test-svc-2", Namespace: "default"} Eventually func error { return k8sClient.Get ctx, serviceName, service } .Should Succeed Expect service.Spec.Selector "app" .To Equal "test-svc-2" } } Context "Updating an InferenceService", func { It "should update the Deployment replicas", func { ctx := context.Background isvc := &inferencev1alpha1.InferenceService{ ObjectMeta: metav1.ObjectMeta{ Name: "update-test", Namespace: "default", }, Spec: inferencev1alpha1.InferenceServiceSpec{ Model: inferencev1alpha1.ModelSpec{URI: "s3://models/test", Framework: "fake", Version: "1"}, Replicas: inferencev1alpha1.ReplicaConfig{Min: 1, Max: 3}, Resources: inferencev1alpha1.ResourceRequirements{CPU: "1", Memory: "1Gi"}, }, } Expect k8sClient.Create ctx, isvc .Should Succeed // Update replica count isvc.Spec.Replicas.Min = 2 Expect k8sClient.Update ctx, isvc .Should Succeed // Verify Deployment was updated deployment := &appsv1.Deployment{} deploymentName := types.NamespacedName{Name: "update-test", Namespace: "default"} Eventually func int32 { k8sClient.Get ctx, deploymentName, deployment return deployment.Spec.Replicas } .Should Equal int32 2 } } Context "Deleting an InferenceService", func { It "should delete owned Deployment and Service", func { ctx := context.Background isvc := &inferencev1alpha1.InferenceService{ ObjectMeta: metav1.ObjectMeta{ Name: "delete-test", Namespace: "default", }, Spec: inferencev1alpha1.InferenceServiceSpec{ Model: inferencev1alpha1.ModelSpec{URI: "s3://models/test", Framework: "fake", Version: "1"}, Replicas: inferencev1alpha1.ReplicaConfig{Min: 1, Max: 3}, Resources: inferencev1alpha1.ResourceRequirements{CPU: "1", Memory: "1Gi"}, }, } Expect k8sClient.Create ctx, isvc .Should Succeed // Delete the InferenceService Expect k8sClient.Delete ctx, isvc .Should Succeed // Verify Deployment is deleted cascade delete via owner ref deployment := &appsv1.Deployment{} deploymentName := types.NamespacedName{Name: "delete-test", Namespace: "default"} Eventually func bool { err := k8sClient.Get ctx, deploymentName, deployment return err = nil // Should not exist } .Should BeTrue } } } Run all tests make test Run with coverage go test ./... -coverprofile=coverage.out go tool cover -html=coverage.out - Unit tests for controller logic - Integration tests with envtest - Tests for create, update, delete - Error case handling tested - 80% code coverage Doc comments — Go's documentation convention comments above functions/types Godoc — Auto-generating documentation from code comments YAML marshalling — How Go structs become YAML examples Code comments vs documentation — When to explain the why, not what - Document the API CRD - Create user-facing examples - Document installation - Prepare for deployment README.md — Project overview docs/API.md — API reference docs/DEPLOYMENT.md — How to deploy docs/EXAMPLES.md — Real-world examples Helm chart optional, for Day 10 docs/API.md: InferenceService API Reference Spec model required - uri string : S3 path to the model - framework string : fake , vllm , or triton - version string : Model version tag replicas required - min int : Minimum pods - max int : Maximum pods resources optional - cpu string : CPU request e.g., "2" - memory string : Memory request e.g., "8Gi" - gpu string : Number of GPUs e.g., "1" autoscaling optional - enabled bool : Enable HPA - targetConcurrency int : Target requests per pod - minReplicas int : HPA min overrides replicas.min - maxReplicas int : HPA max overrides replicas.max traffic optional - Array of {version, weight} for canary deployments Status - phase : Pending | Running | Failed - readyReplicas : How many pods are ready - conditions : Array of Kubernetes conditions docs/DEPLOYMENT.md: Deploying Inferno Prerequisites - Kubernetes 1.19+ - kind/minikube/real cluster Installation bash Clone and install git clone github.com/yourusername/inferno cd inferno Install CRDs make install Deploy operator make deploy Verify kubectl get deployment -n inferno-system kubectl apply -f config/samples/inference v1alpha1 inferenceservice.yaml kubectl get inferenceservices Deliverables - README.md with project description - docs/API.md with complete API reference - docs/DEPLOYMENT.md with setup instructions - docs/EXAMPLES.md with 3-5 real examples - All code has docstrings - CRD manifests are clean --- Day 10: Integration Testing & Final Polish Go Concepts You'll Learn Today - Bash scripting from Go — Shelling out to run commands - Error recovery — Handling failures gracefully - Cleanup patterns — Teardown after tests - Concurrency testing — Race conditions and data races - Real-world debugging — Reading logs, tracing issues Learning Objectives - End-to-end integration testing - Fix any remaining bugs - Prepare for production - Document next steps What You'll Do 1. E2E Test : Deploy operator → create InferenceService → verify all resources 2. Stress Test : Create many InferenceServices, verify controller handles it 3. Failure Test : Kill pods, verify operator recovers 4. Documentation : Finalize docs, create troubleshooting guide Hands-On: E2E Test Script Create e2e test.sh : bash /bin/bash set -e echo "=== Inferno E2E Test ===" Setup kind create cluster --name inferno-e2e || true kubectl config use-context kind-inferno-e2e Deploy make deploy echo "Waiting for operator to be ready..." kubectl wait --for=condition=available --timeout=300s \ deployment/inferno-controller-manager -n inferno-system Create InferenceService echo "Creating InferenceService..." kubectl apply -f config/samples/inference v1alpha1 inferenceservice.yaml Wait for Deployment echo "Waiting for Deployment..." kubectl rollout status deployment/test-llama --timeout=120s Verify Service echo "Verifying Service..." kubectl get svc test-llama kubectl get endpoints test-llama Check status echo "Checking status..." kubectl get inferenceservice test-llama -o yaml Test port-forward echo "Testing connectivity..." kubectl port-forward svc/test-llama 8000:80 & sleep 2 curl localhost:8000/health || true kill %1 echo "=== All tests passed ===" Cleanup kind delete cluster --name inferno-e2e Create many InferenceServices for i in {1..10}; do kubectl apply -f - <<EOF apiVersion: inference.inferno.io/v1alpha1 kind: InferenceService metadata: name: stress-test-$i spec: model: uri: s3://models/test framework: fake version: "1" replicas: min: 1 max: 3 resources: cpu: "1" memory: "1Gi" EOF done Verify all were created kubectl get inferenceservices kubectl get deployments Check controller logs for errors kubectl logs -n inferno-system deployment/inferno-controller-manager Create an InferenceService kubectl apply -f config/samples/inference v1alpha1 inferenceservice.yaml Kill the pod kubectl delete pod -l app=test-llama Verify operator recreates it kubectl get pods -l app=test-llama kubectl wait --for=condition=ready pod -l app=test-llama echo "Pod recovered successfully" - E2E test script passes - Stress test with 10+ InferenceServices works - Failure recovery tested and documented - All logs clean no errors - README complete with quick-start - API docs comprehensive - Code well-commented - RBAC rules correct - CRD validation rules correct Replace fake server with vLLM container image Add Prometheus metrics for monitoring Implement canary deployments with traffic splitting Add webhooks for advanced validation Create Helm chart for easy deployment GPU support with node affinity Multi-model serving with routing Model caching layer Integration with KEDA for custom metrics Security hardening and network policies Copy this into a todo tracker: Day 1: Kubernetes Controllers & Kubebuilder ☐ Understand reconciliation loop ☐ Initialize kubebuilder project ☐ Create InferenceService CRD scaffold ☐ Project compiles Day 2: API Schema ☐ Define ModelSpec, ReplicaConfig, AutoscalingSpec ☐ Add validation rules ☐ Implement status subresource ☐ Generate CRD manifests ☐ Validate YAML deploys Day 3: Reconciliation Logic ☐ Implement Reconcile function ☐ Create constructDeployment ☐ Add controller reference for cleanup ☐ Update status correctly ☐ Controller compiles and deploys Day 4: Local Development ☐ Create kind cluster ☐ Deploy operator locally ☐ Create test InferenceService ☐ Verify Deployment created ☐ Debug with logs Day 5: Service Creation ☐ Implement constructService ☐ Create Service alongside Deployment ☐ Update RBAC for Services ☐ Test service creation ☐ Verify DNS name works Day 6: Autoscaling ☐ Implement HPA creation logic ☐ Support min/max replicas ☐ Add autoscaling validation ☐ Test HPA creation ☐ Verify scaling behavior Day 7: Status & Conditions ☐ Implement updateStatus helper ☐ Add Kubernetes conditions ☐ Implement retry logic ☐ Handle error states ☐ Test status updates Day 8: Testing ☐ Write unit tests ☐ Write integration tests with envtest ☐ Test create/update/delete ☐ Test error cases ☐ Achieve 80% coverage Day 9: Documentation ☐ Write README.md ☐ Document API in docs/API.md ☐ Write deployment guide ☐ Create example YAML files ☐ Add troubleshooting guide Day 10: Integration & Polish ☐ Create E2E test script ☐ Stress test with 10+ services ☐ Test failure recovery ☐ Fix any remaining bugs ☐ Final code review and cleanup By Day 10, you should have: ✅ Working Kubernetes Operator that: - Watches InferenceService CRs - Creates Deployments, Services, HPAs automatically - Updates status correctly - Handles errors and retries - Passes integration tests ✅ Code Quality : - 80% test coverage - Clean logs no errors/warnings - Comprehensive error handling - Well-documented API ✅ Documentation : - User-friendly README - Complete API reference - Deployment guide - Real-world examples ✅ Ready for Next Phase : - Can replace fake server with real vLLM - Can add monitoring/observability - Can scale to production Go & Kubernetes: Operators & CRDs: Tools: Testing: You won't use all of Go, but you'll use these patterns constantly : // EVERY function call needs this: if err = nil { log.Error err, "what went wrong" return ctrl.Result{}, err } // For optional errors: if err = nil { if apierrors.IsNotFound err { // Handle not found } else { return ctrl.Result{}, err } } // Wrapping errors with context: if err = nil { return fmt.Errorf "failed to create deployment: %w", err } // Creating a pointer to an int count := int32 5 ptr := &count // Get address value := ptr // Dereference // In structs, K8s uses pointers for optional fields: spec.Replicas = &count // Optional field spec.Replicas // Read it // Nil-safe access: if spec.Replicas = nil { count = spec.Replicas } // Attach a method to a struct type MyReconciler struct { client.Client Scheme runtime.Scheme } // The r MyReconciler is the "receiver" — like "this" in Java func r MyReconciler Reconcile ctx context.Context, req ctrl.Request ctrl.Result, error { // Can access r.Client, r.Scheme err := r.Get ctx, req.NamespacedName, &obj } // An interface says "anything that has these methods" type Reader interface { Read p byte n int, err error } // k8s client.Client is an interface: type Client interface { Get ctx, key, obj Create ctx, obj Update ctx, obj Delete ctx, obj } // Your code takes interfaces, not concrete types: func MyFunction client client.Client { client.Get ... // Works with any client.Client } // Tags are metadata on struct fields type MyStruct struct { // JSON: how to encode as JSON/YAML Name string json:"name" // Multiple tags: Count int json:"count" kubebuilder:"validation:Minimum=1" // Omit empty values: Optional string json:"optional,omitempty" // In k8s, kubebuilder tags control CRD generation: // +kubebuilder:validation:Required // +kubebuilder:validation:Enum=value1;value2 // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list } js // Create a slice var items string // Append to slice items = append items, "new item" // Iterate for i, item := range items { fmt.Println i, item } // Find in slice for , item := range items { if item == "target" { // Found it } } // With maps finding, replacing : conditions := metav1.Condition{} for i, c := range conditions { if c.Type == "Ready" { conditions i = newCondition // Replace found = true } } if found { conditions = append conditions, newCondition // Add } // Context flows through your code func r Reconciler Reconcile ctx context.Context, req Request { // ctx can be cancelled at any time // Always check for cancellation: select { case <-ctx.Done : return ctx.Err // Was cancelled default: // Keep going } // Timeouts: ctx, cancel := context.WithTimeout context.Background , 10 time.Second defer cancel // Clean up err := r.Get ctx, name, obj // Will timeout after 10s } // Simple assertion if got = expected { t.Errorf "got %v, want %v", got, expected } // Eventually for async operations Eventually func bool { obj := &MyObject{} err := client.Get ctx, name, obj return err == nil // Keep checking until true } .Should BeTrue // Ginkgo-style BDD Describe "MyController", func { It "should create a Deployment", func { // Test code Expect actual .To Equal expected } } // Create a map labels := make map string string labels "app" = "myapp" labels "version" = "1.0" // Or with literal syntax: labels := map string string{ "app": "myapp", "version": "1.0", } // Access v := labels "app" // "myapp" v := labels "nonexistent" // "" zero value // Check if key exists: if val, ok := labels "app" ; ok { // Key exists, val has the value } // Iterate for key, value := range labels { fmt.Println key, value } import "time" now := time.Now // Create k8s time: timestamp := metav1.Time{Time: time.Now } // Calculate duration: future := now.Add 10 time.Second duration := future.Sub now // 10 seconds // Requeue delays: return ctrl.Result{RequeueAfter: 5 time.Second}, nil Common commands you'll use daily: Project setup do this once kubebuilder init --domain myorg.io --repo github.com/me/myop kubebuilder create api --group mygroup --version v1alpha1 --kind MyResource Generate manifests do this often make generate Generate code from struct tags make manifests Generate CRD/RBAC/manager YAML Local development make install Install CRDs into current cluster make uninstall Remove CRDs make run Run controller locally outside k8s make deploy Deploy controller to cluster make undeploy Remove controller Testing make envtest Set up test environment go test ./... Run all tests go test -v ./... Verbose test output go test -cover ./... Show coverage Cleanup make clean Remove generated files Every Kubernetes resource has this pattern: // TypeMeta + ObjectMeta required by all k8s resources type MyResource struct { metav1.TypeMeta json:",inline" // apiVersion, kind metav1.ObjectMeta json:"metadata,omitempty" // name, namespace, labels, etc Spec MyResourceSpec json:"spec,omitempty" // What user wants Status MyResourceStatus json:"status,omitempty" // What actually exists } // List version also required type MyResourceList struct { metav1.TypeMeta json:",inline" metav1.ListMeta json:"metadata,omitempty" Items MyResource json:"items" } // Status is separate from Spec important type MyResourceStatus struct { // Phase: the current state Phase string json:"phase,omitempty" // Pending, Running, Failed // Conditions: detailed state tracking Conditions metav1.Condition json:"conditions,omitempty" // ObservedGeneration: for detecting stale status ObservedGeneration int64 json:"observedGeneration" } // Update it separately: r.Status .Update ctx, resource // NOT r.Update func r MyReconciler Reconcile ctx context.Context, req ctrl.Request ctrl.Result, error { log := log.FromContext ctx // 1. Fetch the resource myresource := &myv1.MyResource{} if err := r.Get ctx, req.NamespacedName, myresource ; err = nil { if apierrors.IsNotFound err { return ctrl.Result{}, nil // Deleted, nothing to do } return ctrl.Result{}, err // Real error } // 2. Check desired state exists childResource := &SomeChildResource{} err := r.Get ctx, childName, childResource if err = nil { if apierrors.IsNotFound err { // 3a. Create it childResource = r.constructChild myresource controllerutil.SetControllerReference myresource, childResource, r.Scheme if err := r.Create ctx, childResource ; err = nil { log.Error err, "Failed to create child" return ctrl.Result{}, err } } else { return ctrl.Result{}, err } } else { // 3b. Update it childResource = r.constructChild myresource if err := r.Update ctx, childResource ; err = nil { log.Error err, "Failed to update child" return ctrl.Result{}, err } } // 4. Update status myresource.Status.ObservedGeneration = myresource.Generation myresource.Status.Phase = "Ready" if err := r.Status .Update ctx, myresource ; err = nil { log.Error err, "Failed to update status" return ctrl.Result{}, err } return ctrl.Result{}, nil } Good luck Build Inferno day by day, test often, and enjoy the journey. 🚀