When architecting a modern container orchestrator like ** Gubernator (gbnt)** — designed to strike the
In this article, we break down how we designed and implemented these two major subsystems in Gubernator v2.24.0 & v2.25.0, and how we leveraged Google Antigravity (AGY) as an autonomous AI engineering partner to architect, implement, test, and live-deploy Full-Stack features (Go + SQLite + Flutter Web + CLI) across a live 3-node multi-host cluster.
Stateful container workloads present a fundamental orchestration challenge: how can a container move between different physical hosts while maintaining access to its persistent disk storage?
┌─────────────────────────────────────────────────────────────────────────┐
│ GUBERNATOR STORAGE & BACKUP ENGINE │
├─────────────────────────────────────────────────────────────────────────┤
│ /var/contenedores (Shared Mobility Pool: NFS, GlusterFS, CephFS) │
│ Point-in-Time Compressed Tarballs (.tar.gz) + SHA-256 Checksums │
│ Background Cron Scheduler & Automated Retention Pruning │
│ Zero-Downtime Consistent Freeze (docker -> tar -> un) │
└─────────────────┬───────────────────────────────────┬───────────────────┘
│ │
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ Centurion 1 (Manager) │ │ Centurion 2 (Worker 1) │
│ IP: 192.168.252.27 │ │ IP: 192.168.252.25 │
│ Mount: /var/contened.. │ │ Mount: /var/contened.. │
└─────────────────────────┘ └─────────────────────────┘
/var/contenedores
)
Gubernator standardizes volume mobility by designating /var/contenedores
across all cluster nodes. When backed by a distributed file system (NFS, GlusterFS, CephFS, CIFS) or localized volumes:
used / total
, percentage, and node read/write mount health).Backing up a running relational database (PostgreSQL, MariaDB, SQLite) while active transactions are in flight risks data corruption.
We implemented an optional Atomic Freeze Strategy:
// internal/storage/backup.go
func CreateBackup(name, targetPath, stackName, serviceName string, Container bool) (*db.Backup, error) {
if Container && containerID != "" {
slog.Info("backup: pausing container for consistent snapshot", "container", containerID)
_ = dockerClient.Container(ctx, containerID)
defer dockerClient.ContainerUn(ctx, containerID)
}
// Stream directory to tar.gz with SHA-256 calculation
archiveFile, sha256Checksum, sizeBytes, err := archiveDirectory(targetPath, destFile)
if err != nil {
return nil, err
}
// ... Save record to SQLite ...
}
Gubernator's background backup daemon evaluates standard cron expressions (e.g. 0 2 * * *
for nightly 2:00 AM backups) and automatically prunes older snapshots according to a configured retention count (e.g. keep last 7 copies).
Deploying third-party container images blindly introduces severe supply-chain risks. In v2.25.0, we introduced a complete Pre-Deployment Admission Gatekeeper:
[ Stack Deploy / Container Run Request ]
│
▼
┌──────────────────────────────────────────────────┐
│ GUBERNATOR ADMISSION GATEKEEPER (Port 4000) │
│ - Evaluates Cluster & Stack Security Policy │
└────────────────────────┬─────────────────────────┘
│
┌───────────────────────────────┴───────────────────────────────┐
▼ ▼
┌───────────────────────────┐ ┌───────────────────────────┐
│ 1. Cryptographic Sign │ │ 🔍 2. CVE Vulnerability │
│ (Cosign / Sigstore) │ │ Scanning & CVSS Scores │
├───────────────────────────┤ ├───────────────────────────┤
│ Is the image signed with │ │ Does image exceed Max │
│ a trusted cluster key? │ │ Severity (Critical/High)? │
└─────────────┬─────────────┘ └─────────────┬─────────────┘
│ │
├───────── ❌ Unsigned / Invalid ├───────── ❌ Exceeds Threshold
│ (If policy = 'ENFORCE') │ (If policy = 'BLOCK')
▼ ▼
╔═══════════════════════════╗ ╔═══════════════════════════╗
║ ⛔ DEPLOYMENT REJECTED ║ ║ ⛔ DEPLOYMENT BLOCKED ║
║ "Signature check failed" ║ ║ "Found 2 Critical CVEs" ║
╚═══════════════════════════╝ ╚═══════════════════════════╝
│ │
└──────────────────────────┬──────────────────────────────────┘
│ ✅ Passes All Admission Checks
▼
╔═════════════════════════════════╗
║ 🚀 Container Scheduled on Hosts ║
╚═════════════════════════════════╝
To eliminate heavy external binary dependencies like cosign
or CGO toolchains, we implemented the cryptographic signing engine using Go's standard library (crypto/ecdsa
, crypto/elliptic
, crypto/x509
, crypto/sha256
):
// internal/security/signing.go
func GenerateCosignKeypair(name string) (pubPEM string, privPEM string, err error) {
privKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return "", "", err
}
privBytes, _ := x509.MarshalECPrivateKey(privKey)
privPEMBlock := &pem.Block{Type: "EC PRIVATE KEY", Bytes: privBytes}
privPEM = string(pem.EncodeToMemory(privPEMBlock))
pubBytes, _ := x509.MarshalPKIXPublicKey(&privKey.PublicKey)
pubPEMBlock := &pem.Block{Type: "PUBLIC KEY", Bytes: pubBytes}
pubPEM = string(pem.EncodeToMemory(pubPEMBlock))
return pubPEM, privPEM, nil
}
For software inventory audits and compliance, Gubernator automatically analyzes container image layers, extracts packages and OS libraries (musl, glibc, OpenSSL, busybox), and exports standardized Software Bill of Materials in:
Rather than requiring users to register images manually, Gubernator continuously discovers all container images running across every node in the cluster (Manager
, Worker 1
, Worker 2
). The UI dynamically renders host badges and service tags indicating where every container instance is hosted.
Gubernator's Web Dashboard (Port 4001) provides two rich Material Design 3 interfaces:
.tar.gz
browser downloads, and backup restore modals.Used in: caddy, promtail on Manager, Worker 1, Worker 2
).Audit / Warn Only
vs Strict Enforcement
, CVE severity threshold blocking).Every capability is accessible directly through the gbnt
CLI:
gbnt volume ls
gbnt backup ls
gbnt backup create --name "postgres-nightly" -- /var/contenedores/postgres
gbnt backup restore <backup-id> --target /var/contenedores/postgres
gbnt scan
gbnt scan postgres:16-alpine
gbnt sbom postgres:16-alpine --format cyclonedx-json > sbom.json
gbnt security key generate --name "prod-release-key"
gbnt image sign company/payments:2.1.0 --key /path/to/private.key
gbnt image verify company/payments:2.1.0
gbnt security policy
Building a distributed orchestrator with state synchronization, cryptographic operations, cross-compilation, and Full-Stack Web UIs is an intricate endeavor. Here is how Google Antigravity accelerated development:
Before writing code, we used Antigravity to formalize comprehensive architectural blueprints:
Having structured specifications allowed the AI to implement the entire pipeline (GORM database schemas, pure Go cryptography, REST API routes, Flutter Dart models, and CLI flags) with complete architectural alignment.
During initial testing of the backup scheduler, we encountered a recursive mutex deadlock: StartBackupScheduler()
was holding cronMutex.Lock()
while calling SyncSchedules()
, which also attempted to acquire cronMutex.Lock()
. Antigravity inspected the call graph, refactored syncSchedulesLocked()
, and verified thread-safety without human intervention.
Antigravity seamlessly built Linux ARM64 binaries (CGO_ENABLED=0 GOOS=linux GOARCH=arm64
), transferred them to a live 3-node Multipass virtualized cluster (gbnt-manager
, gbnt-worker1
, gbnt-worker2
), and executed live HTTP and CLI verification checks against Port 4000, 4001, and 4002.
With Storage & Backups (v2.24.0) and Image Security & Cosign (v2.25.0), Gubernator bridges the gap between lightweight simplicity and enterprise-grade resilience.
Whether you are running a single-node homelab or an edge-distributed cluster, you can now:
Explore the project on GitHub:
GitHub: mario-ezquerro/gubernator
Official Documentation & Guides
Have you implemented image signing or shared volume mobility in your container setups? Share your thoughts in the comments below!