The engineering journey behind OdinRun: building a self-hosted, observable, self-healing, and AI-assisted CI/CD runtime.
A few months ago, this project was called PipelineOS.
It started as an experiment in building a CI/CD execution engine from scratch: a control plane, a Docker-based runner, a simple dashboard, and a way to execute pipeline stages reliably.
Since then, the project grew far beyond that original scope. It gained durable persistence, event-driven execution, real-time observability, runtime telemetry, automated remediation, AI-assisted diagnosis, recovery planning, and an interactive execution debugger.
Along the way, the project also outgrew its original name.
PipelineOS is now OdinRun.
This is the story of that evolution—and what I learned building a complete CI/CD platform layer by layer.
The first major lesson was simple: A CI/CD platform is only as reliable as its execution engine.
When the project began under the name PipelineOS, the original runner was an ~810-line monolith. It was responsible for everything: API communication, Docker container lifecycle management, log streaming, metric collection, retries, auto-remediation, and task orchestration.
It was a classic dumping ground. As a result, subtle bugs plagued the runtime:
PIP-33 became the first major reliability milestone. I decomposed the monolithic runner into clean, single-responsibility components:
Some of the most valuable engineering work wasn't a shiny new feature—it was fixing fundamental runtime behaviors that should never have been broken in the first place.
Docker stdout / stderr Demultiplexing
Docker's raw multiplexed stream (/containers/{id}/attach) prefixes stdout and stderr frames with an 8-byte header ([stream_type, 0, 0, 0, size1, size2, size3, size4]). The original runner treated this as raw text, scattering binary headers into output logs and corrupting log streams. The engine now explicitly demultiplexes the Docker stream into separate, clean channels.
Enforcing Stage Timeouts
While pipeline YAML configurations allowed developers to specify timeouts, the old runner ignored them. If a subprocess hung indefinitely, the runner hung indefinitely. Under PIP-33, stage execution was wrapped in a hard timeout context—forcefully terminating runaway steps before they drain compute fleet resources.
Graceful Container Cleanup
When a runner receives a termination signal (SIGTERM/SIGINT), it shouldn't abandon active Docker containers. The engine maintains an in-memory active container registry and executes an emergency teardown sequence during shutdown to guarantee zero zombie containers.
Resource Boundaries
Stages can now be configured with explicit CPU cores and memory limits (memory_limit: 512m, cpu_quota: 1.5), preventing a single rogue step from causing Out-Of-Memory (OOM) crashes across shared runner hosts.
These aren't glamorous features, but they form the mandatory foundation of an execution engine you can trust.
Once stage execution became reliable, a new architectural bottleneck surfaced: Tight coupling between execution and transport.
Whenever a stage changed state, created logs, or recorded metrics, the runner made synchronous HTTP REST calls back to the API:
Stage State Change ──> HTTP POST /api/runs/:id/status
Log Output ──> HTTP POST /api/runs/:id/logs
Metric Snapshot ──> HTTP POST /api/runs/:id/metrics
This HTTP chatter choked the runner, introduced network latency spikes, and made it impossible to attach new capabilities (like live streaming or remediation) without modifying the core execution loop.
The solution was transitioning to an In-Process Domain Event
Architecture:
Instead of execution logic invoking HTTP endpoints directly, the runner emits strongly-typed domain events to an in-process bus. Independent consumers subscribe to these events asynchronously.
The execution engine no longer cares who consumes the events—whether it's an S3 up, a WebSocket streamer, or an auto-remediation rule engine.
Real-time execution is useless if your build history vanishes when a node restarts.
Through PIP-31, PIP-32, and PIP-34, the platform established clean persistence abstractions:
By placing storage operations behind abstract interfaces (ILogStorage, IArtifactStorage), the runtime remains completely agnostic to the underlying infrastructure. Whether artifacts are stored on a local NVMe drive during local testing or uploaded to AWS S3 / MinIO in production, zero changes are required in the core engine.
Then came PIP-35 one of the most satisfying milestones in the project.
A modern CI/CD engine shouldn't just print Build Failed after 5 minutes. It should expose what is happening inside the execution container in real time.
OdinRun began capturing rich telemetry metrics directly from the Docker daemon during stage runs:
By decoupling telemetry capture from presentation via Server-Sent Events (SSE) and WebSockets, developers can watch CPU graphs spike, memory usage climb, and logs stream instantly in the UI.
The platform could finally see itself running.
Once real-time observability was in place, the next logical question emerged: What should the system do when a failure occurs?
This led to PIP-36: Rule-Based Auto-Remediation.
Most pipeline retries are triggered by human developers clicking "Re-run" on transient infrastructure glitches (NPM timeouts, Docker registry rate limits, flaky DNS queries). OdinRun automates this through a deterministic Rule Engine:
Key Principle: Deterministic Rules First, AI Second
If a failure matches a known regex pattern with a proven recovery policy (e.g. npm ERR! network timeout -> exponential backoff sleep & retry), the engine handles it deterministically. No heavy LLMs needed.
What happens when a build fails for a reason the rule engine has never seen before?
That brought the system to PIP-37: AI Failure Diagnosis.
Instead of dumping 10,000 raw lines of unformatted terminal output into a Large Language Model, OdinRun leverages the structured context built during PIP-35:
AI Diagnosis Context Packet
Pipeline: kpm-clinic
Stage: deploy
Exit code: 1
Duration: 3m 46s
CPU: 42.3% avg
Memory: 968.9 MiB peak
Captured Error Logs:
AccessDenied: User: arn:aws:iam::123:user/deployer is not authorized
to perform: s3:PutObject on resource arn:aws:s3:::prod-bucket/app
Because the LLM receives structured execution context (CPU/Memory profile, exit code, stage duration, and scrubbed log snippets), it can deliver concise, actionable root-cause analysis directly in the UI:
AI Diagnosis: The deploy stage failed due to missing IAM permissions. The deployer IAM user lacks the s3:PutObject policy on arn:aws:s3:::prod-bucket/app.
Suggested Fix: Attach s3:PutObject to the deployment IAM role before retrying.
Safety in automated execution engines is non-negotiable. An AI engine inside a CI/CD platform should never be allowed to execute arbitrary shell commands on production infrastructure.
OdinRun strictly enforces a Decision Boundary:
PIP-37B introduced historical outcome tracking to create a feedback loop for recovery actions:
Failure ──> Diagnosis ──> Recovery ──> Outcome Record ──> Historical Confidence ──> Better Future Decisions
The goal isn't to let an AI model blindly rewrite the CI runtime. Instead, every recovery outcome (whether successful or failed) is recorded as structured evidence.
Over time, this historical evidence ranks effective remediation rules higher, isolates failing rules, and surfaces candidates for developers to turn into permanent, deterministic rules.
At this point, OdinRun could reliably execute, observe, diagnose, and auto-remediate. But scrolling through text logs to understand complex, multi-stage pipelines was still painful.
PIP-38 introduced the Visual Execution Timeline:
0s 30s 60s 90s 120s
│──────────│──────────│──────────│──────────│
lint ██████████ (Passed)
build ███████████████ (Passed)
test ██████ (Failed) ──> [Retry #1] ──> ██████ (Passed)
By mapping stage lifecycle domain events onto a visual timeline, developers can immediately spot execution bottlenecks, stage overlaps, retries, and failure points at a glance.
A powerful backend architecture is useless if the developer interface is clunky. The final UX pass turned OdinRun into a cohesive developer dashboard:
What started as a simple execution script evolved into a decoupled, layered CI/CD platform:
Looking back, the evolution of the project can be summarized as a sequence of core engineering questions:
There is one more important milestone that isn't represented by a PIP number.
The project was renamed.
The project originally began under the name PipelineOS. At that stage, the name accurately described what I was building: an operating layer for executing CI/CD pipelines.
But as the architecture evolved, the scope of the project changed.
It was no longer just a pipeline executor. It had become a complete execution and recovery system combining:
The name OdinRun represents that broader direction.
Why OdinRun?
The rename wasn't intended to erase the history of the project.
PipelineOS is the name under which the architecture was designed and developed. OdinRun is the identity the project is moving forward with.
The existing architecture, milestones, and engineering decisions remain part of the same project.
So the evolution is:
In other words, OdinRun isn't a completely new project. It is the next identity of the project that began as PipelineOS.
The earlier technical milestones—from persistence and the runner engine through observability and intelligence—form the foundation of OdinRun. The original project documentation describes PipelineOS as a self-hosted CI/CD runtime with Docker-backed execution and a React-based dashboard, while the later milestones expanded that foundation substantially.
Building PipelineOS—and eventually evolving it into OdinRun—started as an exploration of CI/CD internals.
It became an exercise in:
The most important lesson was that these aren't isolated problems. They form a dependency chain:
You can't skip directly to the end.
A system can't intelligently diagnose execution it can't observe. It can't safely remediate failures it can't understand. And it can't build useful historical intelligence without durable execution data.
That's what made this project interesting to build.
It wasn't about simply adding AI to CI/CD. It was about gradually building the infrastructure that makes intelligent CI/CD possible.
PipelineOS was where the journey started. OdinRun is where it continues.
OdinRun is an open-source, self-hosted CI/CD runtime built around execution reliability, observability, automated remediation, and developer control.