Thanks for taking the time to read. If you’ve worked on AI platforms, cloud infrastructure, or platform engineering, I’d love to hear how your architecture differs in the comments.
In Part 4 of AI Infrastructure for Cloud Engineers, we looked at FinOps for AI and how GPU utilization, token consumption, model choice, and inference volume affect cost.
Read Part 4:
[FinOps for AI: Understanding GPU, Token, and Inference Costs]
So far, we have looked at individual parts of AI infrastructure:
Kubernetes
GPUs
Scheduling
Model Serving
Observability
FinOps
But production systems rarely operate as separate pieces.
The real challenge is bringing them together into a platform that developers can deploy to, operators can understand, security teams can govern, and businesses can afford to run.
That is what this final article is about.
A simplified architecture might look like this:
Developer
↓
Git Repository
↓
CI Pipeline
↓
Container Registry
↓
GitOps Repository
↓
Kubernetes
│
├── AI Applications
├── Model Servers
├── GPU Workloads
├── Vector Services
└── AI Agents
│
↓
Observability + Security + FinOps
The model is only one component.
A production AI platform also needs:
The goal is not simply to make an AI application run.
The goal is to make it repeatable, secure, observable, scalable, and recoverable.
Creating infrastructure manually might work for an experiment.
Production needs something reproducible.
Instead of engineers manually creating:
Kubernetes Cluster
GPU Node Pool
Network
Storage
Identity
Secrets Integration
Monitoring
the infrastructure should be defined as code.
Conceptually:
Infrastructure Code
↓
Review
↓
Plan
↓
Apply
↓
Cloud Infrastructure
This makes infrastructure:
A typical repository might look like:
infrastructure/
├── network/
├── kubernetes/
├── gpu-nodes/
├── identity/
├── monitoring/
└── environments/
├── dev/
├── staging/
└── production/
The important principle is not the specific IaC tool.
It is that infrastructure changes follow the same engineering discipline as application changes.
Kubernetes becomes the runtime where the AI platform operates.
It might host:
AI API
Model Server
Embedding Service
Vector Search
AI Agents
Background Workers
GPU Workloads
For example:
Kubernetes
│
┌───────────────┼───────────────┐
↓ ↓ ↓
AI Services Model Servers AI Agents
│ │ │
└───────────────┼───────────────┘
↓
GPU Pool
This gives the platform a common way to manage:
Scheduling
Scaling
Networking
Health checks
Rollouts
Configuration
Resource allocation
Kubernetes adoption for AI is already moving in this direction. CNCF's 2025 survey found that 66% of organizations hosting generative AI models use Kubernetes for at least some inference workloads.
The interesting part is that AI infrastructure begins to look less like a collection of individual servers and more like a shared platform.
A useful pattern is to separate building software from deploying software.
The CI pipeline can handle:
Code
↓
Tests
↓
Security Scan
↓
Container Build
↓
Container Registry
Deployment can then be handled separately through GitOps.
Container Registry
↓
Deployment Configuration
↓
Git Repository
↓
GitOps Controller
↓
Kubernetes
Why separate them?
Because the CI system does not need broad credentials to modify production clusters directly.
Instead, production configuration lives in Git.
A change becomes something like:
Pull Request
↓
Review
↓
Merge
↓
GitOps Reconciliation
↓
Deployment
Kubernetes itself recommends declarative, version-controlled configuration for production workloads, which also fits naturally with GitOps workflows.
Imagine production currently runs:
model-version: v12
replicas: 4
A new release requires:
model-version: v13
replicas: 6
Instead of manually changing the cluster, the team updates the configuration in Git.
model:
version: v13
replicas: 6
The GitOps controller compares:
Desired State in Git
↓
Actual State in Cluster
and reconciles the difference.
That gives teams:
Change history
Code review
Rollback
Auditability
Environment consistency
If something goes wrong, reverting the Git change can restore the previous desired configuration.
This becomes particularly useful for AI systems where changes may involve:
Model version
Prompt configuration
Resource limits
GPU requirements
Inference replicas
Routing policies
Security should not be added after deployment.
It should exist throughout the platform.
A request path might look like:
User
↓
Authentication
↓
API Gateway
↓
AI Application
↓
Authorized Tool / Model
↓
Protected Resource
Important controls include:
The model should never become the security boundary.
If an AI agent requests access to a database, API, or production tool, the infrastructure still needs to verify whether that operation is allowed.
A useful principle is:
AI decides what it wants to do. The platform decides what it is allowed to do.
This becomes especially important as AI agents begin interacting directly with operational infrastructure.
AI applications may need credentials for:
Model providers
Databases
Vector stores
External APIs
Cloud services
MCP servers
These should not appear inside:
Source code
Container images
Git repositories
Application logs
Instead:
Secrets Manager
↓
Workload Identity
↓
Application
Where possible, workload identity is preferable to long-lived static credentials.
If credentials are required, they should have:
The same principle applies to development, staging, and production.
Each environment should have its own trust boundary.
In Part 3, we looked at AI observability in detail.
At platform level, we want a common telemetry path.
Applications
GPU Nodes
Model Servers
AI Agents
│
├── Metrics
├── Logs
└── Traces
│
↓
OpenTelemetry / Exporters
↓
Observability Platform
OpenTelemetry provides Kubernetes tooling for collectors, operators, and workload instrumentation, making it useful as a common telemetry layer.
A production AI platform should let engineers move from:
User says AI is slow
to:
Request ID
↓
API trace
↓
Model inference latency
↓
GPU saturation
↓
Growing queue depth
without searching through five unrelated systems.
The platform should make diagnosis easier by default.
Traditional SRE signals still matter:
Availability
Latency
Errors
Traffic
AI adds another layer:
Time to first token
Tokens per second
Queue depth
GPU utilization
Model errors
Tool-call failures
Provider latency
A service might look healthy at Kubernetes level:
Pods: Healthy
CPU: Normal
Memory: Normal
while users experience:
Queue: Growing
TTFT: Increasing
GPU: Saturated
Production readiness means connecting both views.
Cost should not live in a completely separate dashboard owned only by finance.
The platform already knows:
GPU utilization
GPU hours
Requests
Tokens
Models
Tenants
Workloads
Those signals can be connected to cost.
AI Workload
↓
Resource Usage
↓
Cost Allocation
↓
Team / Tenant / Product
For example:
Workload: document-summary
Model: model-a
GPU Hours: 420
Requests: 180,000
Cost / Request: $0.018
Now engineering teams can make better decisions about:
Scaling
Model choice
Prompt size
Caching
GPU capacity
FinOps becomes part of platform engineering rather than something reviewed only when the cloud bill arrives.
A production AI release should move through controlled stages.
Developer
↓
Pull Request
↓
Tests
↓
Security Checks
↓
Build Image
↓
Deploy to Staging
↓
Validation
↓
Production Approval
↓
GitOps Deployment
Validation may include:
Unit tests
Integration tests
Model evaluations
Security tests
Smoke tests
Performance tests
AI adds an important distinction.
The service may deploy successfully while the model behaves worse.
So release validation should consider both:
Infrastructure Health
+
AI Quality
A technically healthy deployment is not necessarily a successful AI release.
Imagine model version v13
increases latency or produces worse results.
Production should not depend on someone remembering a long sequence of commands.
If configuration is version controlled:
v12
↓
v13
↓
Problem detected
↓
Revert
↓
v12
Rollback becomes part of the deployment design.
The same applies to:
Container versions
Prompt configurations
Routing policies
Resource limits
Model versions
Recovery should be tested before an incident occurs.
Now the complete architecture starts to look like this:
Developer
↓
Git Repository
↓
CI / Validation
↓
Container Registry
↓
GitOps Configuration
↓
GitOps Controller
↓
Kubernetes
┌─────────────┼─────────────┐
↓ ↓ ↓
AI Services Model Servers AI Agents
│ │ │
└─────────────┼─────────────┘
↓
GPU Infrastructure
↓
┌───────────────┼───────────────┐
↓ ↓ ↓
Observability Security FinOps
Supporting everything:
Infrastructure as Code
Identity
Secrets
Policies
Networking
Storage
Testing
This is less about choosing one perfect tool.
It is about creating clear operational boundaries.
A developer building an AI feature should not need to understand every detail of:
GPU scheduling
Network policy
Secret rotation
Prometheus configuration
GitOps controllers
Cloud billing
Ideally, the platform provides a supported path.
For example:
Developer defines:
Model
GPU requirement
Scaling policy
Environment
The platform handles:
Infrastructure
Deployment
Security
Observability
Cost allocation
This is where AI infrastructure starts overlapping with platform engineering.
Modern internal developer platforms increasingly combine Kubernetes, GitOps, observability, governance, security, and self-service into standardized workflows. AI agents are now beginning to become consumers of those same platforms alongside human developers.
Before calling an AI platform production-ready, I would want clear answers to these questions:
If several of these depend on manual knowledge, the platform still has operational risk.
This article completes the AI Infrastructure for Cloud Engineers series.
We started with:
Why Kubernetes?
Then moved through:
GPU Scheduling
↓
Model Serving
↓
Observability
↓
FinOps
↓
Production Platform
The biggest lesson for me is that production AI is not only a machine-learning problem.
It is also a:
Cloud problem
Distributed systems problem
Platform engineering problem
Security problem
SRE problem
FinOps problem
And that is exactly why cloud engineers have an important role in the AI ecosystem.
A model can be impressive in a notebook.
A production AI system needs much more.
It needs:
Repeatable infrastructure
Controlled delivery
Secure access
Reliable compute
Observability
Cost visibility
Recovery
Kubernetes provides the runtime foundation.
Infrastructure as Code makes the environment reproducible.
GitOps makes deployment controlled and auditable.
Security defines what workloads are allowed to access.
Observability tells us what the system is doing.
FinOps tells us whether we are using those resources efficiently.
Together, those pieces turn an AI application into an operable production platform.
And for cloud, DevOps, SRE, and platform engineers, that may be one of the most interesting parts of the current AI shift.
This article completes my AI Infrastructure for Cloud Engineers series:
Thanks to everyone who has read, commented, or shared their experience throughout the series.
I regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, platform engineering, and the infrastructure behind production AI systems.
LinkedIn: Connect with me on LinkedIn
If you were designing an AI platform from scratch today, which part would you standardize first: infrastructure, deployment, security, observability, or cost management?