INFERNO: Kubernetes-Native AI Inference Operator - 10-Day Expert Learning Plan

Project: Kubernetes-Native AI Inference Operator

Duration: 10 days

Level: Expert

Goal: Build a working Kubernetes controller that manages AI inference workloads

Go Knowledge Required:

• Basic syntax (variables, functions, structs)
• Interfaces
• Error handling with error

type - Goroutines basics

• Package management with go mod

If you're weak on these, take 1 day to review:

- https://gobyexample.com/ (30 min)
- Read: https://tour.golang.org/methods/1 (interfaces, 20 min)
- Practice: Write a simple HTTP server

Kubernetes Knowledge Required:

• What Pods, Deployments, Services are
• How to use kubectl (get, apply, describe, logs)
• YAML basics
• What a namespace is

If weak, do:

- kubectl cheatsheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
- Deploy a simple app: kubectl apply -f nginx.yaml
- Check it: kubectl get pods, kubectl logs
- Clean up: kubectl delete -f nginx.yaml

The Problem It Solves:

Building Kubernetes operators is hard. You need:

• API groups, versions, kinds
• CRD manifests
• Reconciliation loops
• RBAC rules
• Deployment manifests
• Testing setup

All of this is boilerplate. Kubebuilder is a scaffolding tool that generates all this for you.

What Kubebuilder Does:

kubebuilder init          # Create project skeleton
kubebuilder create api    # Generate API + controller template
make manifests            # Generate CRD YAML
make install              # Install CRD into cluster
make run                  # Run controller locally
make deploy               # Deploy controller to cluster

What Kubebuilder Generates:

my-operator/
├── api/v1alpha1/               # Your API definitions (the YAML structure)
│   ├── myresource_types.go      # Go structs that become YAML
│   └── myresource_webhook.go    # Validation webhooks (advanced)
├── controllers/                 # Your business logic
│   └── myresource_controller.go # The reconciliation loop
├── config/
│   ├── crd/                     # Kubernetes CRD manifests (generated)
│   ├── manager/                 # Operator deployment YAML (generated)
│   ├── rbac/                    # RBAC rules (generated)
│   └── samples/                 # Example CRs you create
├── main.go                      # Operator entry point
├── Dockerfile                   # Container image definition
├── Makefile                     # Build commands
└── go.mod                       # Go dependencies

The Workflow:

You Write:                          Kubebuilder Generates:
┌──────────────────────────┐       ┌──────────────────────────┐
│ 1. API types (Go structs)│ ────→ │ 1. CRD YAML manifests   │
│ 2. Controller logic      │       │ 2. RBAC rules           │
│ 3. Reconcile function    │       │ 3. OpenAPI docs         │
│ 4. Tests                 │       │ 4. Deployment manifest  │
└──────────────────────────┘       └──────────────────────────┘

Key Files You'll Edit:

— Define what users can write in YAMLapi/v1alpha1/myresource_types.go

— Write the reconciliation logiccontrollers/myresource_controller.go

— Example YAML for testingconfig/samples/*.yaml

Everything else is generated or boilerplate.

This is the heart of every Kubernetes operator:

┌─────────────────────────────────────────────┐
│  Kubernetes API Server                      │
│  (stores all resources)                     │
└────────────────┬────────────────────────────┘
                 │
                 │ Watch: "Tell me when anything changes"
                 ↓
┌─────────────────────────────────────────────┐
│  Controller (your code)                     │
│                                             │
│  func (r *MyReconciler) Reconcile(...) {   │
│    1. Read the current CR                  │
│    2. Check what's needed                  │
│    3. Create/update Deployments, Services  │
│    4. Update the CR's status               │
│    5. Return (no change) or Requeue        │
│  }                                         │
└────────────────┬────────────────────────────┘
                 │
                 │ Create/Update Resources
                 ↓
┌─────────────────────────────────────────────┐
│  Kubernetes Cluster                         │
│  (Deployments, Services, etc.)              │
└─────────────────────────────────────────────┘

When does Reconcile() get called?

• Someone creates/updates/deletes an InferenceService

• Kubernetes notifies the controller: "Something changed!"

• Controller calls Reconcile()

• Controller reads the CR and creates/updates resources

What should Reconcile() do?

func (r *MyReconciler) Reconcile(ctx, req) (Result, error) {
    // 1. Read the resource
    cr := &MyResource{}
    r.Get(ctx, req.NamespacedName, cr)
    
    // 2. Check: Does desired state == actual state?
    deployment := &appsv1.Deployment{}
    err := r.Get(ctx, deploymentName, deployment)
    
    if err != nil {
        // 3a. Desired state missing: CREATE it
        newDeployment := r.constructDeployment(cr)
        r.Create(ctx, newDeployment)
    } else {
        // 3b. Desired state exists: UPDATE it
        r.Update(ctx, deployment)
    }
    
    // 4. Update status
    cr.Status.Ready = true
    r.Status().Update(ctx, cr)
    
    // 5. Tell k8s "I'm done, check again later"
    return Result{RequeueAfter: 10*time.Second}, nil
}

The Goal: Make actual state match desired state. Forever.

This plan assumes you have basic Go and Kubernetes knowledge. If not, take 1-2 days to refresh using the links above. You'll build INFERNO in phases:

Days 1-3: Kubernetes controller fundamentals & Kubebuilder setup** Days 4-6**: Core operator logic & InferenceService CRD** Days 7-9**: Reconciliation, Deployment management, and testing** Day 10**: Polish, integration testing, and documentation

Each day has theory + hands-on code. By Day 10, you'll have a working operator that can deploy, scale, and manage fake inference workloads.

Struct tags— How Go metadata works (json:"field"

,kubebuilder:validation:

)Interfaces— Theclient.Client

interface (implements Get, Create, Update)Goroutines & channels— Controller-runtime uses these under the hood** Error handling**— Pattern ofif err != nil { return err }

Package management— Usingimport

for external libraries

• Understand how Kubernetes controllers work (watch → reconcile → act)
• Learn the controller-runtime library
• Set up your first Kubebuilder project
• Understand CRDs (Custom Resource Definitions)

Kubernetes Controller Pattern:

┌─────────────────────────────────┐
│  Watch: Is desired != actual?   │
│  If yes → Reconcile             │
│  Update cluster state           │
│  Requeue if needed              │
└─────────────────────────────────┘

controller-runtime: The library that manages this loop** Reconciliation**: The core function that makes changes** Leader election**: How operators stay safe in HA setups** Finalizers**: Cleanup when resources are deleted

go mod init github.com/yourusername/inferno
kubebuilder init --domain inferno.io --repo github.com/yourusername/inferno

kubebuilder create api --group inference --version v1alpha1 --kind InferenceService

api/v1alpha1/inferenceservice_types.go

— Your CRD schemacontrollers/inferenceservice_controller.go

— Your reconciliation logicconfig/crd/

— Kubernetes CRD manifestsconfig/manager/

— Operator deployment manifests

• Project initialized with kubebuilder
• InferenceService CRD created (empty, will fill tomorrow)
• Controller file exists and compiles
• Understand the reconciliation loop by reading generated code

Kubebuilder Book— Read chapters 1-3controller-runtime Overview- Watch: "How Kubernetes Controllers Work" (10 min explainer on YouTube)

Struct field tags— Howjson:"field"

controls YAML marshallingKubebuilder validation tags—+kubebuilder:validation:Required

,Enum=

,Min=

Pointer vs value types— When to use*int32

vsint

Slice operations— Working with arrays ([]TypeName

)Type embedding— Howmetav1.TypeMeta

andmetav1.ObjectMeta

inherit fields

• Design your CRD schema (the YAML users will write)
• Understand Go struct tags for validation & OpenAPI
• Implement status subresource
• Write validation rules

The user wants to write:

apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: llama
spec:
  model:
    uri: s3://models/llama-3
    framework: vllm
    version: "1"
  replicas:
    min: 1
    max: 5
  resources:
    requests:
      cpu: "4"
      memory: "16Gi"
      gpu: "1"
  autoscaling:
    enabled: true
    targetConcurrency: 10
  traffic:
    - version: "1"
      weight: 100

You need to translate this into Go structs.

Edit api/v1alpha1/inferenceservice_types.go

:

package v1alpha1

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// ModelSpec defines the model to deploy
type ModelSpec struct {
	// URI is the location of the model (s3://, http://, local path)
	// +kubebuilder:validation:Required
	URI string `json:"uri"`

	// Framework is the inference runtime (vllm, triton, etc)
	// +kubebuilder:validation:Enum=vllm;triton;fake
	Framework string `json:"framework"`

	// Version is the model version tag
	// +kubebuilder:validation:Required
	Version string `json:"version"`
}

// ResourceRequirements mirrors k8s resource requests
type ResourceRequirements struct {
	CPU    string `json:"cpu,omitempty"`
	Memory string `json:"memory,omitempty"`
	GPU    string `json:"gpu,omitempty"`
}

// AutoscalingSpec defines HPA behavior
type AutoscalingSpec struct {
	Enabled           bool `json:"enabled"`
	TargetConcurrency int  `json:"targetConcurrency,omitempty"`
	MinReplicas       int  `json:"minReplicas,omitempty"`
	MaxReplicas       int  `json:"maxReplicas,omitempty"`
}

// TrafficWeight defines canary traffic split
type TrafficWeight struct {
	Version string `json:"version"`
	Weight  int    `json:"weight"` // 0-100
}

// InferenceServiceSpec defines the desired state
type InferenceServiceSpec struct {
	Model       ModelSpec             `json:"model"`
	Replicas    ReplicaConfig         `json:"replicas"`
	Resources   ResourceRequirements  `json:"resources,omitempty"`
	Autoscaling AutoscalingSpec       `json:"autoscaling,omitempty"`
	Traffic     []TrafficWeight       `json:"traffic,omitempty"`
}

// ReplicaConfig defines scaling bounds
type ReplicaConfig struct {
	Min int `json:"min"`
	Max int `json:"max"`
}

// InferenceServiceStatus defines the observed state
type InferenceServiceStatus struct {
	// Phase: Pending, Running, Failed
	Phase string `json:"phase,omitempty"`

	// ReadyReplicas: how many pods are ready
	ReadyReplicas int `json:"readyReplicas"`

	// Message for debugging
	Message string `json:"message,omitempty"`

	// LastUpdateTime when the controller last reconciled
	LastUpdateTime *metav1.Time `json:"lastUpdateTime,omitempty"`

	// Conditions for events
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=isvc;scope=Namespaced
type InferenceService struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   InferenceServiceSpec   `json:"spec,omitempty"`
	Status InferenceServiceStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true
type InferenceServiceList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []InferenceService `json:"items"`
}

: Generate validation, OpenAPI docs, RBAC+kubebuilder:

annotationsStatus subresource: Separate spec (desired) from status (actual)** Validation tags**:Required

,Enum

,Min

,Max

— enforced at API serverConditions: Standard k8s pattern for tracking async operations

• Complete inferenceservice_types.go

with all structs - Run make generate

to create CRD manifests - CRD installs without errors: kubectl apply -f config/crd/

• Can create a test InferenceService YAML

Kubebuilder CRD TutorialGo API Conventions- Kubernetes API reference for metav1.Condition

Context— Howcontext.Context

controls cancellation and timeoutsThe—client

interfaceGet()

,Create()

,Update()

methodsError wrapping—if err != nil

patternsNamed return types—(ctrl.Result, error)

conventionDefer statements— Cleanup code (not used yet, but will on Day 7)** Type assertions**— Checking error types (apierrors.IsNotFound(err)

)

• Understand the reconciliation function signature
• Learn how to watch resources and trigger reconciliation
• Implement basic reconciliation logic
• Set up Deployment creation from InferenceService

The controller's main job:

Watch InferenceService CRsReconcile: For each CR, ensure a Deployment exists with the right spec** Update statuswith what it found Requeue**if needed (e.g., "check again in 10s")

Edit controllers/inferenceservice_controller.go

:

package controllers

import (
	"context"
	"fmt"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/predicate"

	inferencev1alpha1 "github.com/yourusername/inferno/api/v1alpha1"
)

// InferenceServiceReconciler reconciles an InferenceService object
type InferenceServiceReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=inference.inferno.io,resources=inferenceservices,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=inference.inferno.io,resources=inferenceservices/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete

func (r *InferenceServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := log.FromContext(ctx)

	// Step 1: Fetch the InferenceService
	var inferenceService inferencev1alpha1.InferenceService
	if err := r.Get(ctx, req.NamespacedName, &inferenceService); err != nil {
		if apierrors.IsNotFound(err) {
			log.Info("InferenceService not found, ignoring")
			return ctrl.Result{}, nil
		}
		log.Error(err, "Failed to fetch InferenceService")
		return ctrl.Result{}, err
	}

	log.Info("Reconciling InferenceService", "name", inferenceService.Name)

	// Step 2: Create or update Deployment
	deployment := &appsv1.Deployment{}
	deploymentName := types.NamespacedName{
		Name:      inferenceService.Name,
		Namespace: inferenceService.Namespace,
	}

	if err := r.Get(ctx, deploymentName, deployment); err != nil {
		if apierrors.IsNotFound(err) {
			// Create new Deployment
			deployment = r.constructDeployment(&inferenceService)
			if err := controllerutil.SetControllerReference(&inferenceService, deployment, r.Scheme); err != nil {
				log.Error(err, "Failed to set controller reference")
				return ctrl.Result{}, err
			}
			if err := r.Create(ctx, deployment); err != nil {
				log.Error(err, "Failed to create Deployment")
				return ctrl.Result{}, err
			}
			log.Info("Created Deployment", "deployment", deployment.Name)
		} else {
			log.Error(err, "Failed to fetch Deployment")
			return ctrl.Result{}, err
		}
	} else {
		// Update existing Deployment
		deployment = r.constructDeployment(&inferenceService)
		if err := r.Update(ctx, deployment); err != nil {
			log.Error(err, "Failed to update Deployment")
			return ctrl.Result{}, err
		}
		log.Info("Updated Deployment", "deployment", deployment.Name)
	}

	// Step 3: Update status
	inferenceService.Status.Phase = "Running"
	inferenceService.Status.ReadyReplicas = int(*deployment.Spec.Replicas)
	inferenceService.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}

	if err := r.Status().Update(ctx, &inferenceService); err != nil {
		log.Error(err, "Failed to update status")
		return ctrl.Result{}, err
	}

	log.Info("Reconciliation complete")
	return ctrl.Result{}, nil
}

// constructDeployment builds the Deployment spec from InferenceService
func (r *InferenceServiceReconciler) constructDeployment(isvc *inferencev1alpha1.InferenceService) *appsv1.Deployment {
	replicas := int32(isvc.Spec.Replicas.Min)

	labels := map[string]string{
		"app":      isvc.Name,
		"operator": "inferno",
	}

	deployment := &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name:      isvc.Name,
			Namespace: isvc.Namespace,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &replicas,
			Selector: &metav1.LabelSelector{
				MatchLabels: labels,
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: labels,
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name:  "inference-server",
							Image: "inference-server:fake", // TODO: vLLM later
							Ports: []corev1.ContainerPort{
								{
									ContainerPort: 8000,
									Name:          "http",
								},
							},
							Env: []corev1.EnvVar{
								{
									Name:  "MODEL_URI",
									Value: isvc.Spec.Model.URI,
								},
								{
									Name:  "MODEL_VERSION",
									Value: isvc.Spec.Model.Version,
								},
							},
							Resources: corev1.ResourceRequirements{
								Requests: corev1.ResourceList{
									corev1.ResourceCPU:    resource.MustParse(isvc.Spec.Resources.CPU),
									corev1.ResourceMemory: resource.MustParse(isvc.Spec.Resources.Memory),
								},
							},
						},
					},
				},
			},
		},
	}

	return deployment
}

func (r *InferenceServiceReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&inferencev1alpha1.InferenceService{}).
		Owns(&appsv1.Deployment{}). // Watch Deployments we create
		WithEventFilter(predicate.GenerationChangedPredicate{}). // Ignore status-only updates
		Complete(r)
}

: Called when an InferenceService changes or a watched resource changesReconcile()

: Links Deployment to InferenceService (for cleanup)SetControllerReference()

: Watch Deployments we create; if one changes, re-reconcile the InferenceServiceOwns()

Status subresource: Updated separately withr.Status().Update()

RBAC: The+kubebuilder:rbac:

comments generate Kubernetes RBAC rules

inferenceservice_controller.go

implementsReconcile()

constructDeployment()

creates valid k8s Deployment specs - Controller compiles: make build

• RBAC rules generate: make manifests

• Understand the reconciliation loop flow

Shell commands from Go— How to runkubectl

andkind

commandsLogging— Usinglog.FromContext(ctx)

andlog.Info()

Testing with real Kubernetes— Integration testing concepts** Environment setup**— HowKUBECONFIG

and kubectl contexts work

• Set up a local Kubernetes cluster with kind

• Deploy your operator locally

• Manually test reconciliation

• Debug with logs and kubectl

Create a local test environment where you can:

• Deploy your operator
• Create an InferenceService CR
• Watch the controller reconcile it into a Deployment
• Verify everything works

kind create cluster --name inferno-dev

kubectl get nodes
kubectl get pods -A

make install

make deploy

kubectl get deployment -n inferno-system
kubectl logs -n inferno-system deployment/inferno-controller-manager -f

Create config/samples/inference_v1alpha1_inferenceservice.yaml

:

apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: test-llama
spec:
  model:
    uri: s3://models/llama-3
    framework: fake
    version: "1"
  replicas:
    min: 1
    max: 3
  resources:
    cpu: "2"
    memory: "8Gi"
    gpu: "0"
  autoscaling:
    enabled: false
  traffic:
    - version: "1"
      weight: 100

Deploy and observe:

kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

kubectl logs -n inferno-system deployment/inferno-controller-manager -f

kubectl get deployments
kubectl get inferenceservices
kubectl describe inferenceservice test-llama

kubectl get inferenceservice test-llama -o yaml
kubectl logs -n inferno-system deployment/inferno-controller-manager --tail=100 -f

kubectl describe inferenceservice test-llama
kubectl describe deployment test-llama

kubectl get events --all-namespaces | grep inferno

kubectl exec -it deployment/inferno-controller-manager -n inferno-system -- bash

• kind cluster running locally

• Operator deployed: make deploy

• InferenceService CR created

• Deployment automatically created from the CR

• Status updated correctly

• Logs show clean reconciliation

Helper functions— Extracting common code intoconstructService()

Go interfaces as contracts— Services follow same pattern as Deployments** Type conversion**— Usingintstr.FromString()

for port typesMap creation— Building label maps for selectors

• Create and manage Kubernetes Services
• Expose inference endpoints
• Handle traffic routing
• Understand label selectors

When an InferenceService is created, the operator should also create a Kubernetes Service so clients can reach the inference server.

InferenceService (user writes this)
       ↓
   Controller
       ├→ Deployment (pods run the server)
       └→ Service (exposes the server)

Update constructDeployment()

and add a new constructService()

in your controller:

// In Reconcile(), after creating/updating Deployment:

// Step 2b: Create or update Service
service := &corev1.Service{}
serviceName := types.NamespacedName{
	Name:      inferenceService.Name,
	Namespace: inferenceService.Namespace,
}

if err := r.Get(ctx, serviceName, service); err != nil {
	if apierrors.IsNotFound(err) {
		service = r.constructService(&inferenceService)
		if err := controllerutil.SetControllerReference(&inferenceService, service, r.Scheme); err != nil {
			log.Error(err, "Failed to set service controller reference")
			return ctrl.Result{}, err
		}
		if err := r.Create(ctx, service); err != nil {
			log.Error(err, "Failed to create Service")
			return ctrl.Result{}, err
		}
		log.Info("Created Service", "service", service.Name)
	} else {
		log.Error(err, "Failed to fetch Service")
		return ctrl.Result{}, err
	}
}

// constructService builds a Service for the InferenceService
func (r *InferenceServiceReconciler) constructService(isvc *inferencev1alpha1.InferenceService) *corev1.Service {
	labels := map[string]string{
		"app":      isvc.Name,
		"operator": "inferno",
	}

	service := &corev1.Service{
		ObjectMeta: metav1.ObjectMeta{
			Name:      isvc.Name,
			Namespace: isvc.Namespace,
		},
		Spec: corev1.ServiceSpec{
			Selector: labels,
			Type:     corev1.ServiceTypeClusterIP,
			Ports: []corev1.ServicePort{
				{
					Name:       "http",
					Port:       80,
					TargetPort: intstr.FromString("http"),
				},
			},
		},
	}

	return service
}
make deploy

kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

kubectl get svc test-llama
kubectl describe svc test-llama

kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl http://test-llama/health

• Service created automatically with InferenceService

• Service selects the right Pods

• Labels and selectors match Deployment

• Service exposed at DNS name <name>.<namespace>.svc.cluster.local

• RBAC updated for Services: make manifests

Conditional creation— When to create vs skip resources** Pointer arithmetic**— Working with*int32

valuesDefault values— Setting sensible defaults in helpers** Complex struct composition**— HPA specs have deeply nested structs

• Create HorizontalPodAutoscaler (HPA) from the InferenceService
• Handle min/max replicas
• Understand metric-based scaling
• Integrate with KEDA (future enhancement)

When autoscaling.enabled: true

, the operator should create an HPA that scales the Deployment:

InferenceService spec:
  autoscaling:
    enabled: true
    targetConcurrency: 10
    minReplicas: 1
    maxReplicas: 5
       ↓
   HPA created
       ↓
   Metrics Server watches CPU/memory
       ↓
   Scales Deployment 1→5 based on load

In your controller, add:

// After Service creation, in Reconcile():

if inferenceService.Spec.Autoscaling.Enabled {
	hpa := &autoscalingv2.HorizontalPodAutoscaler{}
	hpaName := types.NamespacedName{
		Name:      inferenceService.Name,
		Namespace: inferenceService.Namespace,
	}

	if err := r.Get(ctx, hpaName, hpa); err != nil {
		if apierrors.IsNotFound(err) {
			hpa = r.constructHPA(&inferenceService)
			if err := controllerutil.SetControllerReference(&inferenceService, hpa, r.Scheme); err != nil {
				return ctrl.Result{}, err
			}
			if err := r.Create(ctx, hpa); err != nil {
				log.Error(err, "Failed to create HPA")
				return ctrl.Result{}, err
			}
			log.Info("Created HPA", "hpa", hpa.Name)
		} else {
			log.Error(err, "Failed to fetch HPA")
			return ctrl.Result{}, err
		}
	}
}

// constructHPA builds an HPA for the InferenceService
func (r *InferenceServiceReconciler) constructHPA(isvc *inferencev1alpha1.InferenceService) *autoscalingv2.HorizontalPodAutoscaler {
	minReplicas := int32(isvc.Spec.Replicas.Min)
	maxReplicas := int32(isvc.Spec.Replicas.Max)

	// Default to CPU-based scaling if no custom metric
	cpuUtilization := int32(80)

	hpa := &autoscalingv2.HorizontalPodAutoscaler{
		ObjectMeta: metav1.ObjectMeta{
			Name:      isvc.Name,
			Namespace: isvc.Namespace,
		},
		Spec: autoscalingv2.HorizontalPodAutoscalerSpec{
			ScaleTargetRef: autoscalingv2.CrossVersionObjectReference{
				APIVersion: "apps/v1",
				Kind:       "Deployment",
				Name:       isvc.Name,
			},
			MinReplicas: &minReplicas,
			MaxReplicas: maxReplicas,
			Metrics: []autoscalingv2.MetricSpec{
				{
					Type: autoscalingv2.ResourceMetricSourceType,
					Resource: &autoscalingv2.ResourceMetricSource{
						Name:                     corev1.ResourceCPU,
						TargetAverageUtilization: &cpuUtilization,
					},
				},
			},
		},
	}

	return hpa
}

Also update RBAC:

// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete
make deploy

cat <<EOF | kubectl apply -f -
apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: autoscale-test
spec:
  model:
    uri: s3://models/test
    framework: fake
    version: "1"
  replicas:
    min: 1
    max: 5
  resources:
    cpu: "1"
    memory: "1Gi"
  autoscaling:
    enabled: true
    targetConcurrency: 10
    minReplicas: 1
    maxReplicas: 5
EOF

kubectl get hpa
kubectl describe hpa autoscale-test

• HPA created when autoscaling.enabled: true

• Min/max replicas respected

• CPU-based scaling configured

• HPA updates when InferenceService spec changes

• RBAC includes HPA permissions

Receiver methods— The(r *Reconciler)

pattern (object-oriented Go)Slice manipulation— Appending, finding in arrays of conditions** Boolean logic**— Conditional field setting** Time handling**—time.Now()

andmetav1.Time

typesDefer for cleanup— Resource management patterns

• Implement proper status tracking
• Use Kubernetes Conditions pattern
• Handle error states gracefully
• Implement retry logic with exponential backoff

Track the lifecycle of an InferenceService:

Creating → Pending → Ready ✓
     ↓ (error)
     Failed → Retry

Update your InferenceServiceStatus

to track conditions:

// In controllers/inferenceservice_controller.go

func (r *InferenceServiceReconciler) updateStatus(ctx context.Context, isvc *inferencev1alpha1.InferenceService, phase string, message string) error {
	log := log.FromContext(ctx)

	isvc.Status.Phase = phase
	isvc.Status.Message = message
	isvc.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}

	// Add condition
	condition := metav1.Condition{
		Type:               "Ready",
		Status:             metav1.ConditionTrue,
		ObservedGeneration: isvc.Generation,
		Reason:             phase,
		Message:            message,
		LastTransitionTime: metav1.Time{Time: time.Now()},
	}

	if phase != "Running" {
		condition.Status = metav1.ConditionFalse
	}

	// Upsert condition (replace if exists, append if not)
	found := false
	for i, c := range isvc.Status.Conditions {
		if c.Type == condition.Type {
			isvc.Status.Conditions[i] = condition
			found = true
			break
		}
	}
	if !found {
		isvc.Status.Conditions = append(isvc.Status.Conditions, condition)
	}

	if err := r.Status().Update(ctx, isvc); err != nil {
		log.Error(err, "Failed to update status")
		return err
	}

	return nil
}

// In Reconcile(), use it like:
if err := r.updateStatus(ctx, &inferenceService, "Running", "Deployment is ready"); err != nil {
	return ctrl.Result{}, err
}

// For errors, retry with backoff:
if err != nil {
	r.updateStatus(ctx, &inferenceService, "Error", err.Error())
	return ctrl.Result{RequeueAfter: 10 * time.Second}, nil // Retry in 10s
}
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

kubectl get inferenceservice -o wide -w

kubectl get inferenceservice test-llama -o yaml | grep -A 10 conditions

kubectl apply -f - <<EOF
apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: bad-service
spec:
  model:
    uri: s3://models/test
    framework: invalid  # This should fail validation
    version: "1"
  replicas:
    min: 1
    max: 5
EOF

• Status updated on each reconciliation
• Conditions follow Kubernetes pattern
• Error states tracked
• Retry logic with backoff implemented
• kubectl can show readable status

Testing patterns— Table-driven tests, subtests** Goroutine testing**— UsingEventually()

for async operationsMocking with interfaces— Usingclient.Client

interface for testingBDD style testing— Ginkgo framework (Describe

,It

,Eventually

)Test fixtures— Setting up test data

• Write unit tests for your controller
• Write integration tests with envtest
• Test edge cases and error handling
• Achieve good test coverage

✓ Creating InferenceService → Deployment created
✓ Updating spec → Deployment updated
✓ Deleting InferenceService → Deployment deleted (via owner ref)
✓ Invalid spec → Error state
✓ Missing resources → Retry
✓ Concurrent reconciliations → No race conditions

Create controllers/inferenceservice_controller_test.go

:

package controllers

import (
	"context"

	. "github.com/onsi/ginkgo/v2"
	. "github.com/onsi/gomega"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/types"

	inferencev1alpha1 "github.com/yourusername/inferno/api/v1alpha1"
)

var _ = Describe("InferenceServiceReconciler", func() {
	Context("Creating an InferenceService", func() {
		It("should create a Deployment", func() {
			ctx := context.Background()

			isvc := &inferencev1alpha1.InferenceService{
				ObjectMeta: metav1.ObjectMeta{
					Name:      "test-service",
					Namespace: "default",
				},
				Spec: inferencev1alpha1.InferenceServiceSpec{
					Model: inferencev1alpha1.ModelSpec{
						URI:       "s3://models/test",
						Framework: "fake",
						Version:   "1",
					},
					Replicas: inferencev1alpha1.ReplicaConfig{
						Min: 1,
						Max: 3,
					},
					Resources: inferencev1alpha1.ResourceRequirements{
						CPU:    "1",
						Memory: "1Gi",
					},
				},
			}

			// Create the InferenceService
			Expect(k8sClient.Create(ctx, isvc)).Should(Succeed())

			// Eventually, a Deployment should exist
			deployment := &appsv1.Deployment{}
			deploymentName := types.NamespacedName{Name: "test-service", Namespace: "default"}
			Eventually(func() error {
				return k8sClient.Get(ctx, deploymentName, deployment)
			}).Should(Succeed())

			// Verify the Deployment spec
			Expect(*deployment.Spec.Replicas).To(Equal(int32(1)))
			Expect(deployment.Spec.Template.Spec.Containers[0].Name).To(Equal("inference-server"))
		})

		It("should create a Service", func() {
			ctx := context.Background()

			isvc := &inferencev1alpha1.InferenceService{
				ObjectMeta: metav1.ObjectMeta{
					Name:      "test-svc-2",
					Namespace: "default",
				},
				Spec: inferencev1alpha1.InferenceServiceSpec{
					Model: inferencev1alpha1.ModelSpec{
						URI:       "s3://models/test",
						Framework: "fake",
						Version:   "1",
					},
					Replicas: inferencev1alpha1.ReplicaConfig{Min: 1, Max: 3},
					Resources: inferencev1alpha1.ResourceRequirements{
						CPU:    "1",
						Memory: "1Gi",
					},
				},
			}

			Expect(k8sClient.Create(ctx, isvc)).Should(Succeed())

			// Eventually, a Service should exist
			service := &corev1.Service{}
			serviceName := types.NamespacedName{Name: "test-svc-2", Namespace: "default"}
			Eventually(func() error {
				return k8sClient.Get(ctx, serviceName, service)
			}).Should(Succeed())

			Expect(service.Spec.Selector["app"]).To(Equal("test-svc-2"))
		})
	})

	Context("Updating an InferenceService", func() {
		It("should update the Deployment replicas", func() {
			ctx := context.Background()

			isvc := &inferencev1alpha1.InferenceService{
				ObjectMeta: metav1.ObjectMeta{
					Name:      "update-test",
					Namespace: "default",
				},
				Spec: inferencev1alpha1.InferenceServiceSpec{
					Model:     inferencev1alpha1.ModelSpec{URI: "s3://models/test", Framework: "fake", Version: "1"},
					Replicas:  inferencev1alpha1.ReplicaConfig{Min: 1, Max: 3},
					Resources: inferencev1alpha1.ResourceRequirements{CPU: "1", Memory: "1Gi"},
				},
			}

			Expect(k8sClient.Create(ctx, isvc)).Should(Succeed())

			// Update replica count
			isvc.Spec.Replicas.Min = 2
			Expect(k8sClient.Update(ctx, isvc)).Should(Succeed())

			// Verify Deployment was updated
			deployment := &appsv1.Deployment{}
			deploymentName := types.NamespacedName{Name: "update-test", Namespace: "default"}
			Eventually(func(int32) {
				k8sClient.Get(ctx, deploymentName, deployment)
				return *deployment.Spec.Replicas
			}).Should(Equal(int32(2)))
		})
	})

	Context("Deleting an InferenceService", func() {
		It("should delete owned Deployment and Service", func() {
			ctx := context.Background()

			isvc := &inferencev1alpha1.InferenceService{
				ObjectMeta: metav1.ObjectMeta{
					Name:      "delete-test",
					Namespace: "default",
				},
				Spec: inferencev1alpha1.InferenceServiceSpec{
					Model:     inferencev1alpha1.ModelSpec{URI: "s3://models/test", Framework: "fake", Version: "1"},
					Replicas:  inferencev1alpha1.ReplicaConfig{Min: 1, Max: 3},
					Resources: inferencev1alpha1.ResourceRequirements{CPU: "1", Memory: "1Gi"},
				},
			}

			Expect(k8sClient.Create(ctx, isvc)).Should(Succeed())

			// Delete the InferenceService
			Expect(k8sClient.Delete(ctx, isvc)).Should(Succeed())

			// Verify Deployment is deleted (cascade delete via owner ref)
			deployment := &appsv1.Deployment{}
			deploymentName := types.NamespacedName{Name: "delete-test", Namespace: "default"}
			Eventually(func() bool {
				err := k8sClient.Get(ctx, deploymentName, deployment)
				return err != nil // Should not exist
			}).Should(BeTrue())
		})
	})
})
make test

go test ./... -coverprofile=coverage.out
go tool cover -html=coverage.out

• Unit tests for controller logic
• Integration tests with envtest
• Tests for create, update, delete
• Error case handling tested

• 80% code coverage

Doc comments— Go's documentation convention (comments above functions/types)** Godoc**— Auto-generating documentation from code comments** YAML marshalling**— How Go structs become YAML examples** Code comments vs documentation**— When to explain the why, not what

• Document the API (CRD)
• Create user-facing examples
• Document installation
• Prepare for deployment

README.md— Project overview** docs/API.md**— API reference** docs/DEPLOYMENT.md**— How to deploy** docs/EXAMPLES.md**— Real-world examples** Helm chart**(optional, for Day 10)

docs/API.md:

## Spec

### model (required)
- `uri` (string): S3 path to the model
- `framework` (string): `fake`, `vllm`, or `triton`
- `version` (string): Model version tag

### replicas (required)
- `min` (int): Minimum pods
- `max` (int): Maximum pods

### resources (optional)
- `cpu` (string): CPU request (e.g., "2")
- `memory` (string): Memory request (e.g., "8Gi")
- `gpu` (string): Number of GPUs (e.g., "1")

### autoscaling (optional)
- `enabled` (bool): Enable HPA
- `targetConcurrency` (int): Target requests per pod
- `minReplicas` (int): HPA min (overrides replicas.min)
- `maxReplicas` (int): HPA max (overrides replicas.max)

### traffic (optional)
- Array of `{version, weight}` for canary deployments

## Status

- `phase`: Pending | Running | Failed
- `readyReplicas`: How many pods are ready
- `conditions`: Array of Kubernetes conditions

docs/DEPLOYMENT.md:

## Prerequisites
- Kubernetes 1.19+
- kind/minikube/real cluster

## Installation

``` bash
git clone github.com/yourusername/inferno
cd inferno

make install

make deploy

kubectl get deployment -n inferno-system
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml
kubectl get inferenceservices
### Deliverables
- [ ] README.md with project description
- [ ] docs/API.md with complete API reference
- [ ] docs/DEPLOYMENT.md with setup instructions
- [ ] docs/EXAMPLES.md with 3-5 real examples
- [ ] All code has docstrings
- [ ] CRD manifests are clean

---

## Day 10: Integration Testing & Final Polish

### Go Concepts You'll Learn Today
- **Bash scripting from Go** — Shelling out to run commands
- **Error recovery** — Handling failures gracefully
- **Cleanup patterns** — Teardown after tests
- **Concurrency testing** — Race conditions and data races
- **Real-world debugging** — Reading logs, tracing issues

### Learning Objectives
- End-to-end integration testing
- Fix any remaining bugs
- Prepare for production
- Document next steps

### What You'll Do

1. **E2E Test**: Deploy operator → create InferenceService → verify all resources
2. **Stress Test**: Create many InferenceServices, verify controller handles it
3. **Failure Test**: Kill pods, verify operator recovers
4. **Documentation**: Finalize docs, create troubleshooting guide

### Hands-On: E2E Test Script

Create `e2e_test.sh`:

``` bash
#!/bin/bash
set -e

echo "=== Inferno E2E Test ==="

kind create cluster --name inferno-e2e || true
kubectl config use-context kind-inferno-e2e

make deploy

echo "Waiting for operator to be ready..."
kubectl wait --for=condition=available --timeout=300s \
  deployment/inferno-controller-manager -n inferno-system

echo "Creating InferenceService..."
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

echo "Waiting for Deployment..."
kubectl rollout status deployment/test-llama --timeout=120s

echo "Verifying Service..."
kubectl get svc test-llama
kubectl get endpoints test-llama

echo "Checking status..."
kubectl get inferenceservice test-llama -o yaml

echo "Testing connectivity..."
kubectl port-forward svc/test-llama 8000:80 &
sleep 2
curl localhost:8000/health || true
kill %1

echo "=== All tests passed ==="

kind delete cluster --name inferno-e2e
for i in {1..10}; do
  kubectl apply -f - <<EOF
apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: stress-test-$i
spec:
  model:
    uri: s3://models/test
    framework: fake
    version: "1"
  replicas:
    min: 1
    max: 3
  resources:
    cpu: "1"
    memory: "1Gi"
EOF
done

kubectl get inferenceservices
kubectl get deployments

kubectl logs -n inferno-system deployment/inferno-controller-manager
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

kubectl delete pod -l app=test-llama

kubectl get pods -l app=test-llama
kubectl wait --for=condition=ready pod -l app=test-llama

echo "Pod recovered successfully"

• E2E test script passes
• Stress test with 10+ InferenceServices works
• Failure recovery tested and documented
• All logs clean (no errors)
• README complete with quick-start
• API docs comprehensive
• Code well-commented
• RBAC rules correct
• CRD validation rules correct

Replace fake server with vLLM container imageAdd Prometheus metrics for monitoringImplement canary deployments with traffic splittingAdd webhooks for advanced validationCreate Helm chart for easy deploymentGPU support with node affinityMulti-model serving with routingModel caching layerIntegration with KEDA for custom metricsSecurity hardening and network policies

Copy this into a todo tracker:

Day 1: Kubernetes Controllers & Kubebuilder
  ☐ Understand reconciliation loop
  ☐ Initialize kubebuilder project
  ☐ Create InferenceService CRD scaffold
  ☐ Project compiles

Day 2: API Schema
  ☐ Define ModelSpec, ReplicaConfig, AutoscalingSpec
  ☐ Add validation rules
  ☐ Implement status subresource
  ☐ Generate CRD manifests
  ☐ Validate YAML deploys

Day 3: Reconciliation Logic
  ☐ Implement Reconcile() function
  ☐ Create constructDeployment()
  ☐ Add controller reference for cleanup
  ☐ Update status correctly
  ☐ Controller compiles and deploys

Day 4: Local Development
  ☐ Create kind cluster
  ☐ Deploy operator locally
  ☐ Create test InferenceService
  ☐ Verify Deployment created
  ☐ Debug with logs

Day 5: Service Creation
  ☐ Implement constructService()
  ☐ Create Service alongside Deployment
  ☐ Update RBAC for Services
  ☐ Test service creation
  ☐ Verify DNS name works

Day 6: Autoscaling
  ☐ Implement HPA creation logic
  ☐ Support min/max replicas
  ☐ Add autoscaling validation
  ☐ Test HPA creation
  ☐ Verify scaling behavior

Day 7: Status & Conditions
  ☐ Implement updateStatus() helper
  ☐ Add Kubernetes conditions
  ☐ Implement retry logic
  ☐ Handle error states
  ☐ Test status updates

Day 8: Testing
  ☐ Write unit tests
  ☐ Write integration tests with envtest
  ☐ Test create/update/delete
  ☐ Test error cases
  ☐ Achieve >80% coverage

Day 9: Documentation
  ☐ Write README.md
  ☐ Document API in docs/API.md
  ☐ Write deployment guide
  ☐ Create example YAML files
  ☐ Add troubleshooting guide

Day 10: Integration & Polish
  ☐ Create E2E test script
  ☐ Stress test with 10+ services
  ☐ Test failure recovery
  ☐ Fix any remaining bugs
  ☐ Final code review and cleanup

By Day 10, you should have:

✅ Working Kubernetes Operator that:

• Watches InferenceService CRs
• Creates Deployments, Services, HPAs automatically
• Updates status correctly
• Handles errors and retries
• Passes integration tests

✅ Code Quality:

80% test coverage

• Clean logs (no errors/warnings)
• Comprehensive error handling
• Well-documented API

✅ Documentation:

• User-friendly README
• Complete API reference
• Deployment guide
• Real-world examples

✅ Ready for Next Phase:

• Can replace fake server with real vLLM
• Can add monitoring/observability
• Can scale to production

Go & Kubernetes:

Operators & CRDs:

Tools:

Testing:

You won't use all of Go, but you'll use these patterns constantly:

// EVERY function call needs this:
if err != nil {
    log.Error(err, "what went wrong")
    return ctrl.Result{}, err
}

// For optional errors:
if err != nil {
    if apierrors.IsNotFound(err) {
        // Handle not found
    } else {
        return ctrl.Result{}, err
    }
}

// Wrapping errors with context:
if err != nil {
    return fmt.Errorf("failed to create deployment: %w", err)
}
// Creating a pointer to an int
count := int32(5)
ptr := &count          // Get address
value := *ptr          // Dereference

// In structs, K8s uses pointers for optional fields:
spec.Replicas = &count // Optional field
*spec.Replicas         // Read it

// Nil-safe access:
if spec.Replicas != nil {
    count = *spec.Replicas
}
// Attach a method to a struct
type MyReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// The (r *MyReconciler) is the "receiver" — like "this" in Java
func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Can access r.Client, r.Scheme
    err := r.Get(ctx, req.NamespacedName, &obj)
}
// An interface says "anything that has these methods"
type Reader interface {
    Read(p []byte) (n int, err error)
}

// k8s client.Client is an interface:
type Client interface {
    Get(ctx, key, obj)
    Create(ctx, obj)
    Update(ctx, obj)
    Delete(ctx, obj)
}

// Your code takes interfaces, not concrete types:
func MyFunction(client client.Client) {
    client.Get(...)  // Works with any client.Client
}
// Tags are metadata on struct fields
type MyStruct struct {
    // JSON: how to encode as JSON/YAML
    Name string `json:"name"`
    
    // Multiple tags:
    Count int `json:"count" kubebuilder:"validation:Minimum=1"`
    
    // Omit empty values:
    Optional string `json:"optional,omitempty"`
    
    // In k8s, kubebuilder tags control CRD generation:
    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Enum=value1;value2
    // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list
}
js
// Create a slice
var items []string

// Append to slice
items = append(items, "new item")

// Iterate
for i, item := range items {
    fmt.Println(i, item)
}

// Find in slice
for _, item := range items {
    if item == "target" {
        // Found it
    }
}

// With maps (finding, replacing):
conditions := []metav1.Condition{}
for i, c := range conditions {
    if c.Type == "Ready" {
        conditions[i] = newCondition  // Replace
        found = true
    }
}
if !found {
    conditions = append(conditions, newCondition)  // Add
}
// Context flows through your code
func (r *Reconciler) Reconcile(ctx context.Context, req Request) {
    // ctx can be cancelled at any time
    // Always check for cancellation:
    
    select {
    case <-ctx.Done():
        return ctx.Err()  // Was cancelled
    default:
        // Keep going
    }
    
    // Timeouts:
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()  // Clean up
    
    err := r.Get(ctx, name, obj)  // Will timeout after 10s
}
// Simple assertion
if got != expected {
    t.Errorf("got %v, want %v", got, expected)
}

// Eventually (for async operations)
Eventually(func() bool {
    obj := &MyObject{}
    err := client.Get(ctx, name, obj)
    return err == nil  // Keep checking until true
}).Should(BeTrue())

// Ginkgo-style (BDD)
Describe("MyController", func() {
    It("should create a Deployment", func() {
        // Test code
        Expect(actual).To(Equal(expected))
    })
})
// Create a map
labels := make(map[string]string)
labels["app"] = "myapp"
labels["version"] = "1.0"

// Or with literal syntax:
labels := map[string]string{
    "app":     "myapp",
    "version": "1.0",
}

// Access
v := labels["app"]  // "myapp"
v := labels["nonexistent"]  // "" (zero value)

// Check if key exists:
if val, ok := labels["app"]; ok {
    // Key exists, val has the value
}

// Iterate
for key, value := range labels {
    fmt.Println(key, value)
}
import "time"

now := time.Now()

// Create k8s time:
timestamp := metav1.Time{Time: time.Now()}

// Calculate duration:
future := now.Add(10 * time.Second)
duration := future.Sub(now)  // 10 seconds

// Requeue delays:
return ctrl.Result{RequeueAfter: 5 * time.Second}, nil

Common commands you'll use daily:

kubebuilder init --domain myorg.io --repo github.com/me/myop
kubebuilder create api --group mygroup --version v1alpha1 --kind MyResource

make generate          # Generate code from struct tags
make manifests         # Generate CRD/RBAC/manager YAML

make install           # Install CRDs into current cluster
make uninstall         # Remove CRDs
make run              # Run controller locally (outside k8s)
make deploy           # Deploy controller to cluster
make undeploy         # Remove controller

make envtest          # Set up test environment
go test ./...         # Run all tests
go test -v ./...      # Verbose test output
go test -cover ./...  # Show coverage

make clean            # Remove generated files

Every Kubernetes resource has this pattern:

// TypeMeta + ObjectMeta (required by all k8s resources)
type MyResource struct {
    metav1.TypeMeta   `json:",inline"`          // apiVersion, kind
    metav1.ObjectMeta `json:"metadata,omitempty"` // name, namespace, labels, etc
    
    Spec   MyResourceSpec   `json:"spec,omitempty"`     // What user wants
    Status MyResourceStatus `json:"status,omitempty"`   // What actually exists
}

// List version (also required)
type MyResourceList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []MyResource `json:"items"`
}
// Status is separate from Spec (important!)
type MyResourceStatus struct {
    // Phase: the current state
    Phase string `json:"phase,omitempty"`  // Pending, Running, Failed
    
    // Conditions: detailed state tracking
    Conditions []metav1.Condition `json:"conditions,omitempty"`
    
    // ObservedGeneration: for detecting stale status
    ObservedGeneration int64 `json:"observedGeneration"`
}

// Update it separately:
r.Status().Update(ctx, resource)  // NOT r.Update()
func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)
    
    // 1. Fetch the resource
    myresource := &myv1.MyResource{}
    if err := r.Get(ctx, req.NamespacedName, myresource); err != nil {
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil  // Deleted, nothing to do
        }
        return ctrl.Result{}, err  // Real error
    }
    
    // 2. Check desired state exists
    childResource := &SomeChildResource{}
    err := r.Get(ctx, childName, childResource)
    if err != nil {
        if apierrors.IsNotFound(err) {
            // 3a. Create it
            childResource = r.constructChild(myresource)
            controllerutil.SetControllerReference(myresource, childResource, r.Scheme)
            if err := r.Create(ctx, childResource); err != nil {
                log.Error(err, "Failed to create child")
                return ctrl.Result{}, err
            }
        } else {
            return ctrl.Result{}, err
        }
    } else {
        // 3b. Update it
        childResource = r.constructChild(myresource)
        if err := r.Update(ctx, childResource); err != nil {
            log.Error(err, "Failed to update child")
            return ctrl.Result{}, err
        }
    }
    
    // 4. Update status
    myresource.Status.ObservedGeneration = myresource.Generation
    myresource.Status.Phase = "Ready"
    if err := r.Status().Update(ctx, myresource); err != nil {
        log.Error(err, "Failed to update status")
        return ctrl.Result{}, err
    }
    
    return ctrl.Result{}, nil
}

Good luck! Build Inferno day by day, test often, and enjoy the journey. 🚀

source & further reading

gist.github.com — original article AI Grocery Assistant with Raspberry Pi Minimal LLM Watermarking from scratch A system prompt to stop AI from writing like a LinkedIn/X thought leader.