# INFERNO: Kubernetes-Native AI Inference Operator - 10-Day Expert Learning Plan

> Source: <https://gist.github.com/mwaykole/c78942324691d872f318806e65658fc5>
> Published: 2026-08-24 06:59:43+00:00

**Project:** Kubernetes-Native AI Inference Operator

**Duration:** 10 days

**Level:** Expert

**Goal:** Build a working Kubernetes controller that manages AI inference workloads

**Go Knowledge Required:**

- Basic syntax (variables, functions, structs)
- Interfaces
- Error handling with
`error`

type - Goroutines basics
- Package management with
`go mod`

**If you're weak on these**, take 1 day to review:

```
# Quick Go refresher resources
- https://gobyexample.com/ (30 min)
- Read: https://tour.golang.org/methods/1 (interfaces, 20 min)
- Practice: Write a simple HTTP server
```

**Kubernetes Knowledge Required:**

- What Pods, Deployments, Services are
- How to use kubectl (get, apply, describe, logs)
- YAML basics
- What a namespace is

**If weak, do:**

```
# Quick k8s refresher
- kubectl cheatsheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
- Deploy a simple app: kubectl apply -f nginx.yaml
- Check it: kubectl get pods, kubectl logs
- Clean up: kubectl delete -f nginx.yaml
```

**The Problem It Solves:**

Building Kubernetes operators is hard. You need:

- API groups, versions, kinds
- CRD manifests
- Reconciliation loops
- RBAC rules
- Deployment manifests
- Testing setup

All of this is **boilerplate**. Kubebuilder is a **scaffolding tool** that generates all this for you.

**What Kubebuilder Does:**

```
kubebuilder init          # Create project skeleton
kubebuilder create api    # Generate API + controller template
make manifests            # Generate CRD YAML
make install              # Install CRD into cluster
make run                  # Run controller locally
make deploy               # Deploy controller to cluster
```

**What Kubebuilder Generates:**

```
my-operator/
├── api/v1alpha1/               # Your API definitions (the YAML structure)
│   ├── myresource_types.go      # Go structs that become YAML
│   └── myresource_webhook.go    # Validation webhooks (advanced)
├── controllers/                 # Your business logic
│   └── myresource_controller.go # The reconciliation loop
├── config/
│   ├── crd/                     # Kubernetes CRD manifests (generated)
│   ├── manager/                 # Operator deployment YAML (generated)
│   ├── rbac/                    # RBAC rules (generated)
│   └── samples/                 # Example CRs you create
├── main.go                      # Operator entry point
├── Dockerfile                   # Container image definition
├── Makefile                     # Build commands
└── go.mod                       # Go dependencies
```

**The Workflow:**

```
You Write:                          Kubebuilder Generates:
┌──────────────────────────┐       ┌──────────────────────────┐
│ 1. API types (Go structs)│ ────→ │ 1. CRD YAML manifests   │
│ 2. Controller logic      │       │ 2. RBAC rules           │
│ 3. Reconcile function    │       │ 3. OpenAPI docs         │
│ 4. Tests                 │       │ 4. Deployment manifest  │
└──────────────────────────┘       └──────────────────────────┘
```

**Key Files You'll Edit:**

— Define what users can write in YAML`api/v1alpha1/myresource_types.go`

— Write the reconciliation logic`controllers/myresource_controller.go`

— Example YAML for testing`config/samples/*.yaml`

**Everything else is generated or boilerplate.**

This is the **heart** of every Kubernetes operator:

```
┌─────────────────────────────────────────────┐
│  Kubernetes API Server                      │
│  (stores all resources)                     │
└────────────────┬────────────────────────────┘
                 │
                 │ Watch: "Tell me when anything changes"
                 ↓
┌─────────────────────────────────────────────┐
│  Controller (your code)                     │
│                                             │
│  func (r *MyReconciler) Reconcile(...) {   │
│    1. Read the current CR                  │
│    2. Check what's needed                  │
│    3. Create/update Deployments, Services  │
│    4. Update the CR's status               │
│    5. Return (no change) or Requeue        │
│  }                                         │
└────────────────┬────────────────────────────┘
                 │
                 │ Create/Update Resources
                 ↓
┌─────────────────────────────────────────────┐
│  Kubernetes Cluster                         │
│  (Deployments, Services, etc.)              │
└─────────────────────────────────────────────┘
```

**When does Reconcile() get called?**

- Someone creates/updates/deletes an InferenceService
- Kubernetes notifies the controller: "Something changed!"
- Controller calls
`Reconcile()`

- Controller reads the CR and creates/updates resources

**What should Reconcile() do?**

```
func (r *MyReconciler) Reconcile(ctx, req) (Result, error) {
    // 1. Read the resource
    cr := &MyResource{}
    r.Get(ctx, req.NamespacedName, cr)
    
    // 2. Check: Does desired state == actual state?
    deployment := &appsv1.Deployment{}
    err := r.Get(ctx, deploymentName, deployment)
    
    if err != nil {
        // 3a. Desired state missing: CREATE it
        newDeployment := r.constructDeployment(cr)
        r.Create(ctx, newDeployment)
    } else {
        // 3b. Desired state exists: UPDATE it
        r.Update(ctx, deployment)
    }
    
    // 4. Update status
    cr.Status.Ready = true
    r.Status().Update(ctx, cr)
    
    // 5. Tell k8s "I'm done, check again later"
    return Result{RequeueAfter: 10*time.Second}, nil
}
```

**The Goal:** Make actual state match desired state. Forever.

This plan assumes you have **basic** Go and Kubernetes knowledge. If not, take 1-2 days to refresh using the links above. You'll build INFERNO in phases:

**Days 1-3**: Kubernetes controller fundamentals & Kubebuilder setup** Days 4-6**: Core operator logic & InferenceService CRD** Days 7-9**: Reconciliation, Deployment management, and testing** Day 10**: Polish, integration testing, and documentation

Each day has theory + hands-on code. By Day 10, you'll have a working operator that can deploy, scale, and manage fake inference workloads.

**Struct tags**— How Go metadata works (`json:"field"`

,`kubebuilder:validation:`

)**Interfaces**— The`client.Client`

interface (implements Get, Create, Update)**Goroutines & channels**— Controller-runtime uses these under the hood** Error handling**— Pattern of`if err != nil { return err }`

**Package management**— Using`import`

for external libraries

- Understand how Kubernetes controllers work (watch → reconcile → act)
- Learn the controller-runtime library
- Set up your first Kubebuilder project
- Understand CRDs (Custom Resource Definitions)

**Kubernetes Controller Pattern:**

```
┌─────────────────────────────────┐
│  Watch: Is desired != actual?   │
│  If yes → Reconcile             │
│  Update cluster state           │
│  Requeue if needed              │
└─────────────────────────────────┘
```

**controller-runtime**: The library that manages this loop** Reconciliation**: The core function that makes changes** Leader election**: How operators stay safe in HA setups** Finalizers**: Cleanup when resources are deleted

```
# Create project structure
go mod init github.com/yourusername/inferno
kubebuilder init --domain inferno.io --repo github.com/yourusername/inferno

# Create the InferenceService API
kubebuilder create api --group inference --version v1alpha1 --kind InferenceService
```

`api/v1alpha1/inferenceservice_types.go`

— Your CRD schema`controllers/inferenceservice_controller.go`

— Your reconciliation logic`config/crd/`

— Kubernetes CRD manifests`config/manager/`

— Operator deployment manifests

- Project initialized with kubebuilder
- InferenceService CRD created (empty, will fill tomorrow)
- Controller file exists and compiles
- Understand the reconciliation loop by reading generated code

[Kubebuilder Book](https://book.kubebuilder.io/)— Read chapters 1-3[controller-runtime Overview](https://pkg.go.dev/sigs.k8s.io/controller-runtime)- Watch: "How Kubernetes Controllers Work" (10 min explainer on YouTube)

**Struct field tags**— How`json:"field"`

controls YAML marshalling**Kubebuilder validation tags**—`+kubebuilder:validation:Required`

,`Enum=`

,`Min=`

**Pointer vs value types**— When to use`*int32`

vs`int`

**Slice operations**— Working with arrays (`[]TypeName`

)**Type embedding**— How`metav1.TypeMeta`

and`metav1.ObjectMeta`

inherit fields

- Design your CRD schema (the YAML users will write)
- Understand Go struct tags for validation & OpenAPI
- Implement status subresource
- Write validation rules

The user wants to write:

```
apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: llama
spec:
  model:
    uri: s3://models/llama-3
    framework: vllm
    version: "1"
  replicas:
    min: 1
    max: 5
  resources:
    requests:
      cpu: "4"
      memory: "16Gi"
      gpu: "1"
  autoscaling:
    enabled: true
    targetConcurrency: 10
  traffic:
    - version: "1"
      weight: 100
```

You need to translate this into Go structs.

Edit `api/v1alpha1/inferenceservice_types.go`

:

```
package v1alpha1

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// ModelSpec defines the model to deploy
type ModelSpec struct {
	// URI is the location of the model (s3://, http://, local path)
	// +kubebuilder:validation:Required
	URI string `json:"uri"`

	// Framework is the inference runtime (vllm, triton, etc)
	// +kubebuilder:validation:Enum=vllm;triton;fake
	Framework string `json:"framework"`

	// Version is the model version tag
	// +kubebuilder:validation:Required
	Version string `json:"version"`
}

// ResourceRequirements mirrors k8s resource requests
type ResourceRequirements struct {
	CPU    string `json:"cpu,omitempty"`
	Memory string `json:"memory,omitempty"`
	GPU    string `json:"gpu,omitempty"`
}

// AutoscalingSpec defines HPA behavior
type AutoscalingSpec struct {
	Enabled           bool `json:"enabled"`
	TargetConcurrency int  `json:"targetConcurrency,omitempty"`
	MinReplicas       int  `json:"minReplicas,omitempty"`
	MaxReplicas       int  `json:"maxReplicas,omitempty"`
}

// TrafficWeight defines canary traffic split
type TrafficWeight struct {
	Version string `json:"version"`
	Weight  int    `json:"weight"` // 0-100
}

// InferenceServiceSpec defines the desired state
type InferenceServiceSpec struct {
	Model       ModelSpec             `json:"model"`
	Replicas    ReplicaConfig         `json:"replicas"`
	Resources   ResourceRequirements  `json:"resources,omitempty"`
	Autoscaling AutoscalingSpec       `json:"autoscaling,omitempty"`
	Traffic     []TrafficWeight       `json:"traffic,omitempty"`
}

// ReplicaConfig defines scaling bounds
type ReplicaConfig struct {
	Min int `json:"min"`
	Max int `json:"max"`
}

// InferenceServiceStatus defines the observed state
type InferenceServiceStatus struct {
	// Phase: Pending, Running, Failed
	Phase string `json:"phase,omitempty"`

	// ReadyReplicas: how many pods are ready
	ReadyReplicas int `json:"readyReplicas"`

	// Message for debugging
	Message string `json:"message,omitempty"`

	// LastUpdateTime when the controller last reconciled
	LastUpdateTime *metav1.Time `json:"lastUpdateTime,omitempty"`

	// Conditions for events
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=isvc;scope=Namespaced
type InferenceService struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   InferenceServiceSpec   `json:"spec,omitempty"`
	Status InferenceServiceStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true
type InferenceServiceList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []InferenceService `json:"items"`
}
```

: Generate validation, OpenAPI docs, RBAC`+kubebuilder:`

annotations**Status subresource**: Separate spec (desired) from status (actual)** Validation tags**:`Required`

,`Enum`

,`Min`

,`Max`

— enforced at API server**Conditions**: Standard k8s pattern for tracking async operations

- Complete
`inferenceservice_types.go`

with all structs - Run
`make generate`

to create CRD manifests - CRD installs without errors:
`kubectl apply -f config/crd/`

- Can create a test InferenceService YAML

[Kubebuilder CRD Tutorial](https://book.kubebuilder.io/cronjob-tutorial/api-design.html)[Go API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md)- Kubernetes API reference for
`metav1.Condition`

**Context**— How`context.Context`

controls cancellation and timeouts**The**—`client`

interface`Get()`

,`Create()`

,`Update()`

methods**Error wrapping**—`if err != nil`

patterns**Named return types**—`(ctrl.Result, error)`

convention**Defer statements**— Cleanup code (not used yet, but will on Day 7)** Type assertions**— Checking error types (`apierrors.IsNotFound(err)`

)

- Understand the reconciliation function signature
- Learn how to watch resources and trigger reconciliation
- Implement basic reconciliation logic
- Set up Deployment creation from InferenceService

The controller's main job:

**Watch** InferenceService CRs**Reconcile**: For each CR, ensure a Deployment exists with the right spec** Update status**with what it found** Requeue**if needed (e.g., "check again in 10s")

Edit `controllers/inferenceservice_controller.go`

:

```
package controllers

import (
	"context"
	"fmt"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/predicate"

	inferencev1alpha1 "github.com/yourusername/inferno/api/v1alpha1"
)

// InferenceServiceReconciler reconciles an InferenceService object
type InferenceServiceReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=inference.inferno.io,resources=inferenceservices,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=inference.inferno.io,resources=inferenceservices/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete

func (r *InferenceServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := log.FromContext(ctx)

	// Step 1: Fetch the InferenceService
	var inferenceService inferencev1alpha1.InferenceService
	if err := r.Get(ctx, req.NamespacedName, &inferenceService); err != nil {
		if apierrors.IsNotFound(err) {
			log.Info("InferenceService not found, ignoring")
			return ctrl.Result{}, nil
		}
		log.Error(err, "Failed to fetch InferenceService")
		return ctrl.Result{}, err
	}

	log.Info("Reconciling InferenceService", "name", inferenceService.Name)

	// Step 2: Create or update Deployment
	deployment := &appsv1.Deployment{}
	deploymentName := types.NamespacedName{
		Name:      inferenceService.Name,
		Namespace: inferenceService.Namespace,
	}

	if err := r.Get(ctx, deploymentName, deployment); err != nil {
		if apierrors.IsNotFound(err) {
			// Create new Deployment
			deployment = r.constructDeployment(&inferenceService)
			if err := controllerutil.SetControllerReference(&inferenceService, deployment, r.Scheme); err != nil {
				log.Error(err, "Failed to set controller reference")
				return ctrl.Result{}, err
			}
			if err := r.Create(ctx, deployment); err != nil {
				log.Error(err, "Failed to create Deployment")
				return ctrl.Result{}, err
			}
			log.Info("Created Deployment", "deployment", deployment.Name)
		} else {
			log.Error(err, "Failed to fetch Deployment")
			return ctrl.Result{}, err
		}
	} else {
		// Update existing Deployment
		deployment = r.constructDeployment(&inferenceService)
		if err := r.Update(ctx, deployment); err != nil {
			log.Error(err, "Failed to update Deployment")
			return ctrl.Result{}, err
		}
		log.Info("Updated Deployment", "deployment", deployment.Name)
	}

	// Step 3: Update status
	inferenceService.Status.Phase = "Running"
	inferenceService.Status.ReadyReplicas = int(*deployment.Spec.Replicas)
	inferenceService.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}

	if err := r.Status().Update(ctx, &inferenceService); err != nil {
		log.Error(err, "Failed to update status")
		return ctrl.Result{}, err
	}

	log.Info("Reconciliation complete")
	return ctrl.Result{}, nil
}

// constructDeployment builds the Deployment spec from InferenceService
func (r *InferenceServiceReconciler) constructDeployment(isvc *inferencev1alpha1.InferenceService) *appsv1.Deployment {
	replicas := int32(isvc.Spec.Replicas.Min)

	labels := map[string]string{
		"app":      isvc.Name,
		"operator": "inferno",
	}

	deployment := &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name:      isvc.Name,
			Namespace: isvc.Namespace,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &replicas,
			Selector: &metav1.LabelSelector{
				MatchLabels: labels,
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: labels,
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name:  "inference-server",
							Image: "inference-server:fake", // TODO: vLLM later
							Ports: []corev1.ContainerPort{
								{
									ContainerPort: 8000,
									Name:          "http",
								},
							},
							Env: []corev1.EnvVar{
								{
									Name:  "MODEL_URI",
									Value: isvc.Spec.Model.URI,
								},
								{
									Name:  "MODEL_VERSION",
									Value: isvc.Spec.Model.Version,
								},
							},
							Resources: corev1.ResourceRequirements{
								Requests: corev1.ResourceList{
									corev1.ResourceCPU:    resource.MustParse(isvc.Spec.Resources.CPU),
									corev1.ResourceMemory: resource.MustParse(isvc.Spec.Resources.Memory),
								},
							},
						},
					},
				},
			},
		},
	}

	return deployment
}

func (r *InferenceServiceReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&inferencev1alpha1.InferenceService{}).
		Owns(&appsv1.Deployment{}). // Watch Deployments we create
		WithEventFilter(predicate.GenerationChangedPredicate{}). // Ignore status-only updates
		Complete(r)
}
```

: Called when an InferenceService changes or a watched resource changes`Reconcile()`

: Links Deployment to InferenceService (for cleanup)`SetControllerReference()`

: Watch Deployments we create; if one changes, re-reconcile the InferenceService`Owns()`

**Status subresource**: Updated separately with`r.Status().Update()`

**RBAC**: The`+kubebuilder:rbac:`

comments generate Kubernetes RBAC rules

-
`inferenceservice_controller.go`

implements`Reconcile()`

-
`constructDeployment()`

creates valid k8s Deployment specs - Controller compiles:
`make build`

- RBAC rules generate:
`make manifests`

- Understand the reconciliation loop flow

**Shell commands from Go**— How to run`kubectl`

and`kind`

commands**Logging**— Using`log.FromContext(ctx)`

and`log.Info()`

**Testing with real Kubernetes**— Integration testing concepts** Environment setup**— How`KUBECONFIG`

and kubectl contexts work

- Set up a local Kubernetes cluster with
`kind`

- Deploy your operator locally
- Manually test reconciliation
- Debug with logs and kubectl

Create a local test environment where you can:

- Deploy your operator
- Create an InferenceService CR
- Watch the controller reconcile it into a Deployment
- Verify everything works

```
# Create a kind cluster
kind create cluster --name inferno-dev

# Verify it works
kubectl get nodes
kubectl get pods -A

# Install your operator CRDs
make install

# Deploy the operator
make deploy

# Check it's running
kubectl get deployment -n inferno-system
kubectl logs -n inferno-system deployment/inferno-controller-manager -f
```

Create `config/samples/inference_v1alpha1_inferenceservice.yaml`

:

```
apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: test-llama
spec:
  model:
    uri: s3://models/llama-3
    framework: fake
    version: "1"
  replicas:
    min: 1
    max: 3
  resources:
    cpu: "2"
    memory: "8Gi"
    gpu: "0"
  autoscaling:
    enabled: false
  traffic:
    - version: "1"
      weight: 100
```

Deploy and observe:

```
# Apply the CR
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

# Watch the controller work
kubectl logs -n inferno-system deployment/inferno-controller-manager -f

# Check if Deployment was created
kubectl get deployments
kubectl get inferenceservices
kubectl describe inferenceservice test-llama

# Check the status
kubectl get inferenceservice test-llama -o yaml
# View controller logs
kubectl logs -n inferno-system deployment/inferno-controller-manager --tail=100 -f

# Describe what happened
kubectl describe inferenceservice test-llama
kubectl describe deployment test-llama

# Check events
kubectl get events --all-namespaces | grep inferno

# Interactive debugging (if needed)
kubectl exec -it deployment/inferno-controller-manager -n inferno-system -- bash
```

- kind cluster running locally
- Operator deployed:
`make deploy`

- InferenceService CR created
- Deployment automatically created from the CR
- Status updated correctly
- Logs show clean reconciliation

**Helper functions**— Extracting common code into`constructService()`

**Go interfaces as contracts**— Services follow same pattern as Deployments** Type conversion**— Using`intstr.FromString()`

for port types**Map creation**— Building label maps for selectors

- Create and manage Kubernetes Services
- Expose inference endpoints
- Handle traffic routing
- Understand label selectors

When an InferenceService is created, the operator should also create a Kubernetes Service so clients can reach the inference server.

```
InferenceService (user writes this)
       ↓
   Controller
       ├→ Deployment (pods run the server)
       └→ Service (exposes the server)
```

Update `constructDeployment()`

and add a new `constructService()`

in your controller:

```
// In Reconcile(), after creating/updating Deployment:

// Step 2b: Create or update Service
service := &corev1.Service{}
serviceName := types.NamespacedName{
	Name:      inferenceService.Name,
	Namespace: inferenceService.Namespace,
}

if err := r.Get(ctx, serviceName, service); err != nil {
	if apierrors.IsNotFound(err) {
		service = r.constructService(&inferenceService)
		if err := controllerutil.SetControllerReference(&inferenceService, service, r.Scheme); err != nil {
			log.Error(err, "Failed to set service controller reference")
			return ctrl.Result{}, err
		}
		if err := r.Create(ctx, service); err != nil {
			log.Error(err, "Failed to create Service")
			return ctrl.Result{}, err
		}
		log.Info("Created Service", "service", service.Name)
	} else {
		log.Error(err, "Failed to fetch Service")
		return ctrl.Result{}, err
	}
}

// constructService builds a Service for the InferenceService
func (r *InferenceServiceReconciler) constructService(isvc *inferencev1alpha1.InferenceService) *corev1.Service {
	labels := map[string]string{
		"app":      isvc.Name,
		"operator": "inferno",
	}

	service := &corev1.Service{
		ObjectMeta: metav1.ObjectMeta{
			Name:      isvc.Name,
			Namespace: isvc.Namespace,
		},
		Spec: corev1.ServiceSpec{
			Selector: labels,
			Type:     corev1.ServiceTypeClusterIP,
			Ports: []corev1.ServicePort{
				{
					Name:       "http",
					Port:       80,
					TargetPort: intstr.FromString("http"),
				},
			},
		},
	}

	return service
}
# Deploy updated operator
make deploy

# Create an InferenceService
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

# Verify Service was created
kubectl get svc test-llama
kubectl describe svc test-llama

# Test from inside the cluster
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl http://test-llama/health
```

- Service created automatically with InferenceService
- Service selects the right Pods
- Labels and selectors match Deployment
- Service exposed at DNS name
`<name>.<namespace>.svc.cluster.local`

- RBAC updated for Services:
`make manifests`

**Conditional creation**— When to create vs skip resources** Pointer arithmetic**— Working with`*int32`

values**Default values**— Setting sensible defaults in helpers** Complex struct composition**— HPA specs have deeply nested structs

- Create HorizontalPodAutoscaler (HPA) from the InferenceService
- Handle min/max replicas
- Understand metric-based scaling
- Integrate with KEDA (future enhancement)

When `autoscaling.enabled: true`

, the operator should create an HPA that scales the Deployment:

```
InferenceService spec:
  autoscaling:
    enabled: true
    targetConcurrency: 10
    minReplicas: 1
    maxReplicas: 5
       ↓
   HPA created
       ↓
   Metrics Server watches CPU/memory
       ↓
   Scales Deployment 1→5 based on load
```

In your controller, add:

```
// After Service creation, in Reconcile():

if inferenceService.Spec.Autoscaling.Enabled {
	hpa := &autoscalingv2.HorizontalPodAutoscaler{}
	hpaName := types.NamespacedName{
		Name:      inferenceService.Name,
		Namespace: inferenceService.Namespace,
	}

	if err := r.Get(ctx, hpaName, hpa); err != nil {
		if apierrors.IsNotFound(err) {
			hpa = r.constructHPA(&inferenceService)
			if err := controllerutil.SetControllerReference(&inferenceService, hpa, r.Scheme); err != nil {
				return ctrl.Result{}, err
			}
			if err := r.Create(ctx, hpa); err != nil {
				log.Error(err, "Failed to create HPA")
				return ctrl.Result{}, err
			}
			log.Info("Created HPA", "hpa", hpa.Name)
		} else {
			log.Error(err, "Failed to fetch HPA")
			return ctrl.Result{}, err
		}
	}
}

// constructHPA builds an HPA for the InferenceService
func (r *InferenceServiceReconciler) constructHPA(isvc *inferencev1alpha1.InferenceService) *autoscalingv2.HorizontalPodAutoscaler {
	minReplicas := int32(isvc.Spec.Replicas.Min)
	maxReplicas := int32(isvc.Spec.Replicas.Max)

	// Default to CPU-based scaling if no custom metric
	cpuUtilization := int32(80)

	hpa := &autoscalingv2.HorizontalPodAutoscaler{
		ObjectMeta: metav1.ObjectMeta{
			Name:      isvc.Name,
			Namespace: isvc.Namespace,
		},
		Spec: autoscalingv2.HorizontalPodAutoscalerSpec{
			ScaleTargetRef: autoscalingv2.CrossVersionObjectReference{
				APIVersion: "apps/v1",
				Kind:       "Deployment",
				Name:       isvc.Name,
			},
			MinReplicas: &minReplicas,
			MaxReplicas: maxReplicas,
			Metrics: []autoscalingv2.MetricSpec{
				{
					Type: autoscalingv2.ResourceMetricSourceType,
					Resource: &autoscalingv2.ResourceMetricSource{
						Name:                     corev1.ResourceCPU,
						TargetAverageUtilization: &cpuUtilization,
					},
				},
			},
		},
	}

	return hpa
}
```

Also update RBAC:

```
// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete
# Deploy
make deploy

# Create an InferenceService with autoscaling
cat <<EOF | kubectl apply -f -
apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: autoscale-test
spec:
  model:
    uri: s3://models/test
    framework: fake
    version: "1"
  replicas:
    min: 1
    max: 5
  resources:
    cpu: "1"
    memory: "1Gi"
  autoscaling:
    enabled: true
    targetConcurrency: 10
    minReplicas: 1
    maxReplicas: 5
EOF

# Check HPA was created
kubectl get hpa
kubectl describe hpa autoscale-test

# (HPA needs metrics-server; kind doesn't have it by default, so skip load testing for now)
```

- HPA created when
`autoscaling.enabled: true`

- Min/max replicas respected
- CPU-based scaling configured
- HPA updates when InferenceService spec changes
- RBAC includes HPA permissions

**Receiver methods**— The`(r *Reconciler)`

pattern (object-oriented Go)**Slice manipulation**— Appending, finding in arrays of conditions** Boolean logic**— Conditional field setting** Time handling**—`time.Now()`

and`metav1.Time`

types**Defer for cleanup**— Resource management patterns

- Implement proper status tracking
- Use Kubernetes Conditions pattern
- Handle error states gracefully
- Implement retry logic with exponential backoff

Track the lifecycle of an InferenceService:

```
Creating → Pending → Ready ✓
     ↓ (error)
     Failed → Retry
```

Update your `InferenceServiceStatus`

to track conditions:

```
// In controllers/inferenceservice_controller.go

func (r *InferenceServiceReconciler) updateStatus(ctx context.Context, isvc *inferencev1alpha1.InferenceService, phase string, message string) error {
	log := log.FromContext(ctx)

	isvc.Status.Phase = phase
	isvc.Status.Message = message
	isvc.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}

	// Add condition
	condition := metav1.Condition{
		Type:               "Ready",
		Status:             metav1.ConditionTrue,
		ObservedGeneration: isvc.Generation,
		Reason:             phase,
		Message:            message,
		LastTransitionTime: metav1.Time{Time: time.Now()},
	}

	if phase != "Running" {
		condition.Status = metav1.ConditionFalse
	}

	// Upsert condition (replace if exists, append if not)
	found := false
	for i, c := range isvc.Status.Conditions {
		if c.Type == condition.Type {
			isvc.Status.Conditions[i] = condition
			found = true
			break
		}
	}
	if !found {
		isvc.Status.Conditions = append(isvc.Status.Conditions, condition)
	}

	if err := r.Status().Update(ctx, isvc); err != nil {
		log.Error(err, "Failed to update status")
		return err
	}

	return nil
}

// In Reconcile(), use it like:
if err := r.updateStatus(ctx, &inferenceService, "Running", "Deployment is ready"); err != nil {
	return ctrl.Result{}, err
}

// For errors, retry with backoff:
if err != nil {
	r.updateStatus(ctx, &inferenceService, "Error", err.Error())
	return ctrl.Result{RequeueAfter: 10 * time.Second}, nil // Retry in 10s
}
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

# Watch status in real-time
kubectl get inferenceservice -o wide -w

# View conditions
kubectl get inferenceservice test-llama -o yaml | grep -A 10 conditions

# Create a bad spec to trigger error
kubectl apply -f - <<EOF
apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: bad-service
spec:
  model:
    uri: s3://models/test
    framework: invalid  # This should fail validation
    version: "1"
  replicas:
    min: 1
    max: 5
EOF
```

- Status updated on each reconciliation
- Conditions follow Kubernetes pattern
- Error states tracked
- Retry logic with backoff implemented
- kubectl can show readable status

**Testing patterns**— Table-driven tests, subtests** Goroutine testing**— Using`Eventually()`

for async operations**Mocking with interfaces**— Using`client.Client`

interface for testing**BDD style testing**— Ginkgo framework (`Describe`

,`It`

,`Eventually`

)**Test fixtures**— Setting up test data

- Write unit tests for your controller
- Write integration tests with envtest
- Test edge cases and error handling
- Achieve good test coverage

```
✓ Creating InferenceService → Deployment created
✓ Updating spec → Deployment updated
✓ Deleting InferenceService → Deployment deleted (via owner ref)
✓ Invalid spec → Error state
✓ Missing resources → Retry
✓ Concurrent reconciliations → No race conditions
```

Create `controllers/inferenceservice_controller_test.go`

:

```
package controllers

import (
	"context"

	. "github.com/onsi/ginkgo/v2"
	. "github.com/onsi/gomega"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/types"

	inferencev1alpha1 "github.com/yourusername/inferno/api/v1alpha1"
)

var _ = Describe("InferenceServiceReconciler", func() {
	Context("Creating an InferenceService", func() {
		It("should create a Deployment", func() {
			ctx := context.Background()

			isvc := &inferencev1alpha1.InferenceService{
				ObjectMeta: metav1.ObjectMeta{
					Name:      "test-service",
					Namespace: "default",
				},
				Spec: inferencev1alpha1.InferenceServiceSpec{
					Model: inferencev1alpha1.ModelSpec{
						URI:       "s3://models/test",
						Framework: "fake",
						Version:   "1",
					},
					Replicas: inferencev1alpha1.ReplicaConfig{
						Min: 1,
						Max: 3,
					},
					Resources: inferencev1alpha1.ResourceRequirements{
						CPU:    "1",
						Memory: "1Gi",
					},
				},
			}

			// Create the InferenceService
			Expect(k8sClient.Create(ctx, isvc)).Should(Succeed())

			// Eventually, a Deployment should exist
			deployment := &appsv1.Deployment{}
			deploymentName := types.NamespacedName{Name: "test-service", Namespace: "default"}
			Eventually(func() error {
				return k8sClient.Get(ctx, deploymentName, deployment)
			}).Should(Succeed())

			// Verify the Deployment spec
			Expect(*deployment.Spec.Replicas).To(Equal(int32(1)))
			Expect(deployment.Spec.Template.Spec.Containers[0].Name).To(Equal("inference-server"))
		})

		It("should create a Service", func() {
			ctx := context.Background()

			isvc := &inferencev1alpha1.InferenceService{
				ObjectMeta: metav1.ObjectMeta{
					Name:      "test-svc-2",
					Namespace: "default",
				},
				Spec: inferencev1alpha1.InferenceServiceSpec{
					Model: inferencev1alpha1.ModelSpec{
						URI:       "s3://models/test",
						Framework: "fake",
						Version:   "1",
					},
					Replicas: inferencev1alpha1.ReplicaConfig{Min: 1, Max: 3},
					Resources: inferencev1alpha1.ResourceRequirements{
						CPU:    "1",
						Memory: "1Gi",
					},
				},
			}

			Expect(k8sClient.Create(ctx, isvc)).Should(Succeed())

			// Eventually, a Service should exist
			service := &corev1.Service{}
			serviceName := types.NamespacedName{Name: "test-svc-2", Namespace: "default"}
			Eventually(func() error {
				return k8sClient.Get(ctx, serviceName, service)
			}).Should(Succeed())

			Expect(service.Spec.Selector["app"]).To(Equal("test-svc-2"))
		})
	})

	Context("Updating an InferenceService", func() {
		It("should update the Deployment replicas", func() {
			ctx := context.Background()

			isvc := &inferencev1alpha1.InferenceService{
				ObjectMeta: metav1.ObjectMeta{
					Name:      "update-test",
					Namespace: "default",
				},
				Spec: inferencev1alpha1.InferenceServiceSpec{
					Model:     inferencev1alpha1.ModelSpec{URI: "s3://models/test", Framework: "fake", Version: "1"},
					Replicas:  inferencev1alpha1.ReplicaConfig{Min: 1, Max: 3},
					Resources: inferencev1alpha1.ResourceRequirements{CPU: "1", Memory: "1Gi"},
				},
			}

			Expect(k8sClient.Create(ctx, isvc)).Should(Succeed())

			// Update replica count
			isvc.Spec.Replicas.Min = 2
			Expect(k8sClient.Update(ctx, isvc)).Should(Succeed())

			// Verify Deployment was updated
			deployment := &appsv1.Deployment{}
			deploymentName := types.NamespacedName{Name: "update-test", Namespace: "default"}
			Eventually(func(int32) {
				k8sClient.Get(ctx, deploymentName, deployment)
				return *deployment.Spec.Replicas
			}).Should(Equal(int32(2)))
		})
	})

	Context("Deleting an InferenceService", func() {
		It("should delete owned Deployment and Service", func() {
			ctx := context.Background()

			isvc := &inferencev1alpha1.InferenceService{
				ObjectMeta: metav1.ObjectMeta{
					Name:      "delete-test",
					Namespace: "default",
				},
				Spec: inferencev1alpha1.InferenceServiceSpec{
					Model:     inferencev1alpha1.ModelSpec{URI: "s3://models/test", Framework: "fake", Version: "1"},
					Replicas:  inferencev1alpha1.ReplicaConfig{Min: 1, Max: 3},
					Resources: inferencev1alpha1.ResourceRequirements{CPU: "1", Memory: "1Gi"},
				},
			}

			Expect(k8sClient.Create(ctx, isvc)).Should(Succeed())

			// Delete the InferenceService
			Expect(k8sClient.Delete(ctx, isvc)).Should(Succeed())

			// Verify Deployment is deleted (cascade delete via owner ref)
			deployment := &appsv1.Deployment{}
			deploymentName := types.NamespacedName{Name: "delete-test", Namespace: "default"}
			Eventually(func() bool {
				err := k8sClient.Get(ctx, deploymentName, deployment)
				return err != nil // Should not exist
			}).Should(BeTrue())
		})
	})
})
# Run all tests
make test

# Run with coverage
go test ./... -coverprofile=coverage.out
go tool cover -html=coverage.out
```

- Unit tests for controller logic
- Integration tests with envtest
- Tests for create, update, delete
- Error case handling tested
- >80% code coverage

**Doc comments**— Go's documentation convention (comments above functions/types)** Godoc**— Auto-generating documentation from code comments** YAML marshalling**— How Go structs become YAML examples** Code comments vs documentation**— When to explain the why, not what

- Document the API (CRD)
- Create user-facing examples
- Document installation
- Prepare for deployment

**README.md**— Project overview** docs/API.md**— API reference** docs/DEPLOYMENT.md**— How to deploy** docs/EXAMPLES.md**— Real-world examples** Helm chart**(optional, for Day 10)

**docs/API.md:**

```
# InferenceService API Reference

## Spec

### model (required)
- `uri` (string): S3 path to the model
- `framework` (string): `fake`, `vllm`, or `triton`
- `version` (string): Model version tag

### replicas (required)
- `min` (int): Minimum pods
- `max` (int): Maximum pods

### resources (optional)
- `cpu` (string): CPU request (e.g., "2")
- `memory` (string): Memory request (e.g., "8Gi")
- `gpu` (string): Number of GPUs (e.g., "1")

### autoscaling (optional)
- `enabled` (bool): Enable HPA
- `targetConcurrency` (int): Target requests per pod
- `minReplicas` (int): HPA min (overrides replicas.min)
- `maxReplicas` (int): HPA max (overrides replicas.max)

### traffic (optional)
- Array of `{version, weight}` for canary deployments

## Status

- `phase`: Pending | Running | Failed
- `readyReplicas`: How many pods are ready
- `conditions`: Array of Kubernetes conditions
```

**docs/DEPLOYMENT.md:**

```
# Deploying Inferno

## Prerequisites
- Kubernetes 1.19+
- kind/minikube/real cluster

## Installation

``` bash
# Clone and install
git clone github.com/yourusername/inferno
cd inferno

# Install CRDs
make install

# Deploy operator
make deploy

# Verify
kubectl get deployment -n inferno-system
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml
kubectl get inferenceservices
### Deliverables
- [ ] README.md with project description
- [ ] docs/API.md with complete API reference
- [ ] docs/DEPLOYMENT.md with setup instructions
- [ ] docs/EXAMPLES.md with 3-5 real examples
- [ ] All code has docstrings
- [ ] CRD manifests are clean

---

## Day 10: Integration Testing & Final Polish

### Go Concepts You'll Learn Today
- **Bash scripting from Go** — Shelling out to run commands
- **Error recovery** — Handling failures gracefully
- **Cleanup patterns** — Teardown after tests
- **Concurrency testing** — Race conditions and data races
- **Real-world debugging** — Reading logs, tracing issues

### Learning Objectives
- End-to-end integration testing
- Fix any remaining bugs
- Prepare for production
- Document next steps

### What You'll Do

1. **E2E Test**: Deploy operator → create InferenceService → verify all resources
2. **Stress Test**: Create many InferenceServices, verify controller handles it
3. **Failure Test**: Kill pods, verify operator recovers
4. **Documentation**: Finalize docs, create troubleshooting guide

### Hands-On: E2E Test Script

Create `e2e_test.sh`:

``` bash
#!/bin/bash
set -e

echo "=== Inferno E2E Test ==="

# Setup
kind create cluster --name inferno-e2e || true
kubectl config use-context kind-inferno-e2e

# Deploy
make deploy

echo "Waiting for operator to be ready..."
kubectl wait --for=condition=available --timeout=300s \
  deployment/inferno-controller-manager -n inferno-system

# Create InferenceService
echo "Creating InferenceService..."
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

# Wait for Deployment
echo "Waiting for Deployment..."
kubectl rollout status deployment/test-llama --timeout=120s

# Verify Service
echo "Verifying Service..."
kubectl get svc test-llama
kubectl get endpoints test-llama

# Check status
echo "Checking status..."
kubectl get inferenceservice test-llama -o yaml

# Test port-forward
echo "Testing connectivity..."
kubectl port-forward svc/test-llama 8000:80 &
sleep 2
curl localhost:8000/health || true
kill %1

echo "=== All tests passed ==="

# Cleanup
kind delete cluster --name inferno-e2e
# Create many InferenceServices
for i in {1..10}; do
  kubectl apply -f - <<EOF
apiVersion: inference.inferno.io/v1alpha1
kind: InferenceService
metadata:
  name: stress-test-$i
spec:
  model:
    uri: s3://models/test
    framework: fake
    version: "1"
  replicas:
    min: 1
    max: 3
  resources:
    cpu: "1"
    memory: "1Gi"
EOF
done

# Verify all were created
kubectl get inferenceservices
kubectl get deployments

# Check controller logs for errors
kubectl logs -n inferno-system deployment/inferno-controller-manager
# Create an InferenceService
kubectl apply -f config/samples/inference_v1alpha1_inferenceservice.yaml

# Kill the pod
kubectl delete pod -l app=test-llama

# Verify operator recreates it
kubectl get pods -l app=test-llama
kubectl wait --for=condition=ready pod -l app=test-llama

echo "Pod recovered successfully"
```

- E2E test script passes
- Stress test with 10+ InferenceServices works
- Failure recovery tested and documented
- All logs clean (no errors)
- README complete with quick-start
- API docs comprehensive
- Code well-commented
- RBAC rules correct
- CRD validation rules correct

**Replace fake server** with vLLM container image**Add Prometheus metrics** for monitoring**Implement canary deployments** with traffic splitting**Add webhooks** for advanced validation**Create Helm chart** for easy deployment**GPU support** with node affinity**Multi-model serving** with routing**Model caching** layer**Integration with KEDA** for custom metrics**Security hardening** and network policies

Copy this into a todo tracker:

```
Day 1: Kubernetes Controllers & Kubebuilder
  ☐ Understand reconciliation loop
  ☐ Initialize kubebuilder project
  ☐ Create InferenceService CRD scaffold
  ☐ Project compiles

Day 2: API Schema
  ☐ Define ModelSpec, ReplicaConfig, AutoscalingSpec
  ☐ Add validation rules
  ☐ Implement status subresource
  ☐ Generate CRD manifests
  ☐ Validate YAML deploys

Day 3: Reconciliation Logic
  ☐ Implement Reconcile() function
  ☐ Create constructDeployment()
  ☐ Add controller reference for cleanup
  ☐ Update status correctly
  ☐ Controller compiles and deploys

Day 4: Local Development
  ☐ Create kind cluster
  ☐ Deploy operator locally
  ☐ Create test InferenceService
  ☐ Verify Deployment created
  ☐ Debug with logs

Day 5: Service Creation
  ☐ Implement constructService()
  ☐ Create Service alongside Deployment
  ☐ Update RBAC for Services
  ☐ Test service creation
  ☐ Verify DNS name works

Day 6: Autoscaling
  ☐ Implement HPA creation logic
  ☐ Support min/max replicas
  ☐ Add autoscaling validation
  ☐ Test HPA creation
  ☐ Verify scaling behavior

Day 7: Status & Conditions
  ☐ Implement updateStatus() helper
  ☐ Add Kubernetes conditions
  ☐ Implement retry logic
  ☐ Handle error states
  ☐ Test status updates

Day 8: Testing
  ☐ Write unit tests
  ☐ Write integration tests with envtest
  ☐ Test create/update/delete
  ☐ Test error cases
  ☐ Achieve >80% coverage

Day 9: Documentation
  ☐ Write README.md
  ☐ Document API in docs/API.md
  ☐ Write deployment guide
  ☐ Create example YAML files
  ☐ Add troubleshooting guide

Day 10: Integration & Polish
  ☐ Create E2E test script
  ☐ Stress test with 10+ services
  ☐ Test failure recovery
  ☐ Fix any remaining bugs
  ☐ Final code review and cleanup
```

By Day 10, you should have:

✅ **Working Kubernetes Operator** that:

- Watches InferenceService CRs
- Creates Deployments, Services, HPAs automatically
- Updates status correctly
- Handles errors and retries
- Passes integration tests

✅ **Code Quality**:

-
80% test coverage

- Clean logs (no errors/warnings)
- Comprehensive error handling
- Well-documented API

✅ **Documentation**:

- User-friendly README
- Complete API reference
- Deployment guide
- Real-world examples

✅ **Ready for Next Phase**:

- Can replace fake server with real vLLM
- Can add monitoring/observability
- Can scale to production

**Go & Kubernetes:**

**Operators & CRDs:**

**Tools:**

**Testing:**

You won't use all of Go, but you'll use these patterns *constantly*:

```
// EVERY function call needs this:
if err != nil {
    log.Error(err, "what went wrong")
    return ctrl.Result{}, err
}

// For optional errors:
if err != nil {
    if apierrors.IsNotFound(err) {
        // Handle not found
    } else {
        return ctrl.Result{}, err
    }
}

// Wrapping errors with context:
if err != nil {
    return fmt.Errorf("failed to create deployment: %w", err)
}
// Creating a pointer to an int
count := int32(5)
ptr := &count          // Get address
value := *ptr          // Dereference

// In structs, K8s uses pointers for optional fields:
spec.Replicas = &count // Optional field
*spec.Replicas         // Read it

// Nil-safe access:
if spec.Replicas != nil {
    count = *spec.Replicas
}
// Attach a method to a struct
type MyReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// The (r *MyReconciler) is the "receiver" — like "this" in Java
func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Can access r.Client, r.Scheme
    err := r.Get(ctx, req.NamespacedName, &obj)
}
// An interface says "anything that has these methods"
type Reader interface {
    Read(p []byte) (n int, err error)
}

// k8s client.Client is an interface:
type Client interface {
    Get(ctx, key, obj)
    Create(ctx, obj)
    Update(ctx, obj)
    Delete(ctx, obj)
}

// Your code takes interfaces, not concrete types:
func MyFunction(client client.Client) {
    client.Get(...)  // Works with any client.Client
}
// Tags are metadata on struct fields
type MyStruct struct {
    // JSON: how to encode as JSON/YAML
    Name string `json:"name"`
    
    // Multiple tags:
    Count int `json:"count" kubebuilder:"validation:Minimum=1"`
    
    // Omit empty values:
    Optional string `json:"optional,omitempty"`
    
    // In k8s, kubebuilder tags control CRD generation:
    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Enum=value1;value2
    // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list
}
js
// Create a slice
var items []string

// Append to slice
items = append(items, "new item")

// Iterate
for i, item := range items {
    fmt.Println(i, item)
}

// Find in slice
for _, item := range items {
    if item == "target" {
        // Found it
    }
}

// With maps (finding, replacing):
conditions := []metav1.Condition{}
for i, c := range conditions {
    if c.Type == "Ready" {
        conditions[i] = newCondition  // Replace
        found = true
    }
}
if !found {
    conditions = append(conditions, newCondition)  // Add
}
// Context flows through your code
func (r *Reconciler) Reconcile(ctx context.Context, req Request) {
    // ctx can be cancelled at any time
    // Always check for cancellation:
    
    select {
    case <-ctx.Done():
        return ctx.Err()  // Was cancelled
    default:
        // Keep going
    }
    
    // Timeouts:
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()  // Clean up
    
    err := r.Get(ctx, name, obj)  // Will timeout after 10s
}
// Simple assertion
if got != expected {
    t.Errorf("got %v, want %v", got, expected)
}

// Eventually (for async operations)
Eventually(func() bool {
    obj := &MyObject{}
    err := client.Get(ctx, name, obj)
    return err == nil  // Keep checking until true
}).Should(BeTrue())

// Ginkgo-style (BDD)
Describe("MyController", func() {
    It("should create a Deployment", func() {
        // Test code
        Expect(actual).To(Equal(expected))
    })
})
// Create a map
labels := make(map[string]string)
labels["app"] = "myapp"
labels["version"] = "1.0"

// Or with literal syntax:
labels := map[string]string{
    "app":     "myapp",
    "version": "1.0",
}

// Access
v := labels["app"]  // "myapp"
v := labels["nonexistent"]  // "" (zero value)

// Check if key exists:
if val, ok := labels["app"]; ok {
    // Key exists, val has the value
}

// Iterate
for key, value := range labels {
    fmt.Println(key, value)
}
import "time"

now := time.Now()

// Create k8s time:
timestamp := metav1.Time{Time: time.Now()}

// Calculate duration:
future := now.Add(10 * time.Second)
duration := future.Sub(now)  // 10 seconds

// Requeue delays:
return ctrl.Result{RequeueAfter: 5 * time.Second}, nil
```

Common commands you'll use daily:

```
# Project setup (do this once)
kubebuilder init --domain myorg.io --repo github.com/me/myop
kubebuilder create api --group mygroup --version v1alpha1 --kind MyResource

# Generate manifests (do this often)
make generate          # Generate code from struct tags
make manifests         # Generate CRD/RBAC/manager YAML

# Local development
make install           # Install CRDs into current cluster
make uninstall         # Remove CRDs
make run              # Run controller locally (outside k8s)
make deploy           # Deploy controller to cluster
make undeploy         # Remove controller

# Testing
make envtest          # Set up test environment
go test ./...         # Run all tests
go test -v ./...      # Verbose test output
go test -cover ./...  # Show coverage

# Cleanup
make clean            # Remove generated files
```

Every Kubernetes resource has this pattern:

```
// TypeMeta + ObjectMeta (required by all k8s resources)
type MyResource struct {
    metav1.TypeMeta   `json:",inline"`          // apiVersion, kind
    metav1.ObjectMeta `json:"metadata,omitempty"` // name, namespace, labels, etc
    
    Spec   MyResourceSpec   `json:"spec,omitempty"`     // What user wants
    Status MyResourceStatus `json:"status,omitempty"`   // What actually exists
}

// List version (also required)
type MyResourceList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []MyResource `json:"items"`
}
// Status is separate from Spec (important!)
type MyResourceStatus struct {
    // Phase: the current state
    Phase string `json:"phase,omitempty"`  // Pending, Running, Failed
    
    // Conditions: detailed state tracking
    Conditions []metav1.Condition `json:"conditions,omitempty"`
    
    // ObservedGeneration: for detecting stale status
    ObservedGeneration int64 `json:"observedGeneration"`
}

// Update it separately:
r.Status().Update(ctx, resource)  // NOT r.Update()
func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)
    
    // 1. Fetch the resource
    myresource := &myv1.MyResource{}
    if err := r.Get(ctx, req.NamespacedName, myresource); err != nil {
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil  // Deleted, nothing to do
        }
        return ctrl.Result{}, err  // Real error
    }
    
    // 2. Check desired state exists
    childResource := &SomeChildResource{}
    err := r.Get(ctx, childName, childResource)
    if err != nil {
        if apierrors.IsNotFound(err) {
            // 3a. Create it
            childResource = r.constructChild(myresource)
            controllerutil.SetControllerReference(myresource, childResource, r.Scheme)
            if err := r.Create(ctx, childResource); err != nil {
                log.Error(err, "Failed to create child")
                return ctrl.Result{}, err
            }
        } else {
            return ctrl.Result{}, err
        }
    } else {
        // 3b. Update it
        childResource = r.constructChild(myresource)
        if err := r.Update(ctx, childResource); err != nil {
            log.Error(err, "Failed to update child")
            return ctrl.Result{}, err
        }
    }
    
    // 4. Update status
    myresource.Status.ObservedGeneration = myresource.Generation
    myresource.Status.Phase = "Ready"
    if err := r.Status().Update(ctx, myresource); err != nil {
        log.Error(err, "Failed to update status")
        return ctrl.Result{}, err
    }
    
    return ctrl.Result{}, nil
}
```

**Good luck! Build Inferno day by day, test often, and enjoy the journey.** 🚀