{"slug": "show-hn-rca-lab-test-observability-tools-on-real-failures", "title": "Show HN: RCA-lab – test observability tools on real failures", "summary": "Coroot released rca-lab, an open-source failure lab that runs real, reproducible incidents on a live Kubernetes cluster to test root-cause-analysis tooling, including AI-based systems. The lab deploys a polyglot microservice stack with real databases and uses genuine failure mechanisms such as allocation regressions and heavy analytics queries, with durable revert via custom resources. It is available on GitHub and requires kubectl and helm.", "body_md": "A realistic, reproducible failure lab for evaluating root-cause-analysis (RCA) tooling — human or AI — on a live Kubernetes cluster.\n\nMost RCA benchmarks replay canned telemetry from toy environments with synthetic faults toggled by feature flags. rca-lab takes the opposite approach:\n\n**A real polyglot microservice stack**(Python, Go, Java, Node.js, Rust, PHP) behind an API gateway, with continuous generated load.** Real databases under production-grade operators**: PostgreSQL, MySQL and MongoDB via Percona operators, a Valkey Cluster via the valkey-operator, Kafka via Strimzi — with seeded data volumes.**Real failure mechanisms only.** No chaos flags inside the apps. A GC pressure incident is a genuine allocation regression shipped as a new image version and rolled back later; a database incident is an analytics workload running heavy queries against the production database; a traffic spike is actually more traffic.**Durable revert.** Every failure scenario is a`FailureScenario`\n\ncustom resource driven by an operator that restores the normal state when the scenario ends, is disabled, or is deleted — even across operator restarts.**Rich telemetry, bring your own backend.** Every service is instrumented with OpenTelemetry SDKs: traces, SDK-emitted metrics (JVM/runtime/HTTP), and logs (to stdout*and*OTLP, trace-correlated). Everything flows to a bundled otel-collector that**discards data by default**— point it at any OTLP backend with one variable.\n\nRequirements: `kubectl`\n\n+ `helm`\n\npointed at a cluster (any distribution;\na default StorageClass, ~8 CPU / 16 GiB across nodes for the full-size lab).\n\n```\ngit clone https://github.com/coroot/rca-lab && cd rca-lab\nmake deploy                 # everything: operators → databases → Kafka → apps → seed\n```\n\nSingle-node cluster (kind/k3d/minikube):\n\n```\nmake deploy SINGLE_NODE=1\n```\n\nSend telemetry somewhere (e.g. Coroot, or any OTLP endpoint):\n\n```\nmake deploy OTLP_ENDPOINT=my-backend:4317\n```\n\nOther variables: `STORAGE_CLASS=<name>`\n\n, `SEED_SIZE_GB=<n>`\n\n(0 skips seeding),\n`OTLP_HEADERS=k=v`\n\n, `YES=1`\n\n(no confirmation prompt). Re-running `make deploy`\n\nconverges idempotently — it is also how you change any of these settings.\n\nTeardown:\n\n```\nmake clean                  # KEEP_DATA=1 keeps the database volumes\n```\n\nScenarios are Kubernetes custom resources:\n\n```\nkubectl get failurescenarios\nkubectl patch failurescenario pg-analytics-queries --type=merge -p '{\"spec\":{\"enabled\":true}}'\n```\n\nor use the web UI:\n\n```\nkubectl port-forward svc/rca-lab-operator 8080\n```\n\nThe UI lists every scenario grouped by category, with severity and live state, and starts or stops each one with a click.\n\nEach scenario documents its mechanism and the telemetry symptoms an RCA tool\nshould be able to observe. Scenarios can also run on a cron schedule with a\nfixed duration — see `scenarios/`\n\n.\n\nNever install rca-lab on a shared or production cluster.The scenario operator deliberately has the power to degrade workloads in its namespace.\n\nEvery scenario uses a genuine real-world mechanism — never a synthetic fault\nflag inside the app — and reverts durably. Each carries an `expectedSymptoms`\n\nlist that doubles as documentation and a grading rubric for RCA tools.\n\nThe `reliability`\n\ncategory is a different *kind* of test. The other scenarios\nare acute incidents (latency, errors, saturation) that exercise **RCA** — given\na symptom, find the cause. Reliability scenarios are latent, slow-burn risks\n(bloat, stale stats, blocked vacuum, replication lag, checkpoint pressure) that\noften produce **no user-facing symptom at onset**; they exercise proactive\n**detection** — whether a tool flags a developing risk before it becomes an\noutage. Their `expectedSymptoms`\n\nare early-warning indicators, not incident\nsymptoms.\n\n| Scenario | Mechanism | What an RCA tool should find |\n|---|---|---|\n`pg-analytics-queries` |\nAn `analytics-reporting` workload runs heavy multi-join/aggregation queries (full scans of the ~10 GB products table) against the production PostgreSQL, through the same pgBouncer pool as the apps. |\nElevated `product-catalog` /`inventory-service` latency; PostgreSQL CPU/IO saturation; new full-scan query fingerprints in `pg_stat_statements` attributable to the `analytics-reporting` workload. |\n`pg-exclusive-lock` |\nA stalled `schema-migration` transaction takes a real `ACCESS EXCLUSIVE` lock on the `products` table (`LOCK TABLE` ) and then hangs holding it — the \"a migration grabbed the lock and never let go\" incident. |\nproduct-catalog queries on `products` block on the lock; its connection pool fills and the service goes unavailable, so `api-gateway` product endpoints error — yet PostgreSQL CPU/IO stay flat because nothing is executing. The tell is lock waits (`pg_locks` / `pg_blocking_pids` ), not resource saturation. |\n`mysql-lock-contention` |\nA stalled transaction holds InnoDB row locks on the hot end of the `orders` table (`SELECT … FOR UPDATE` , including the gap above the max id) and then hangs — a transaction that grabbed locks and stalled. |\nReads keep working (InnoDB MVCC snapshots), but `order-service` writes (new orders, status updates) block and fail with `Lock wait timeout exceeded (50s)` while PXC CPU/IO stay flat. The tell is row-lock waits (`information_schema.innodb_trx` / `performance_schema.data_lock_waits` ), not resource saturation — and reads-fine/writes-blocked distinguishes it from Postgres's table-level lock. |\n`mysql-analytics-queries` |\nThe same `analytics-reporting` actor runs large join/aggregation queries (filesort, temp tables) against the production MySQL `orders` database via HAProxy. |\nElevated `order-service` /checkout latency and errors; PXC CPU/IO saturation; heavy statements in the slow query log attributable to the workload. |\n\n| Scenario | Mechanism | What an RCA tool should find |\n|---|---|---|\n`order-service-gc-regression` |\nA genuine bad deploy: `order-service` rolls out `1.1.0` , a real code regression that deep-copies every order read into an ineffective cache. GC pressure builds; revert rolls back to the known-good image. |\np99 rises after the rollout while p50 stays flat; JVM allocation rate and GC time climb; heap sawtooth trends toward the limit; onset correlates exactly with the deployment event. |\n`order-service-memory-leak` |\nA genuine bad deploy: `order-service` rolls out `1.4.0` , a real regression that appends a batch of small \"audit trail\" objects per read into a registry that is never pruned. Slow leak of millions of tiny objects; revert rolls back to the known-good image. |\np95/p99 creep up gradually (no crash, no step change); old-gen/live-set trends up; GC time and mixed-collection frequency rise as the live set grows; onset matches the rollout. Distinct from the fast OOM-crash leaks. |\n`product-catalog-gc-pressure` |\nA genuine bad deploy: `product-catalog` rolls out `1.1.0` , whose server-side \"product cards\" re-encode every returned product into large short-lived buffers on each read. Nothing retained (no leak) — pure allocation churn; revert rolls back. |\nGo GC CPU fraction and cycle frequency spike; allocation rate jumps while heap in-use stays bounded (no OOM); `product-catalog` CPU saturates/throttles and latency rises, propagating to `api-gateway` ; Postgres stays healthy. |\n`review-service-event-loop` |\nA genuine bad deploy: `review-service` rolls out `1.1.0` , adding a synchronous \"content safety\" CPU loop on the request path that blocks the single-threaded Node.js event loop for tens of ms per read. Revert rolls back. |\np95/p99 balloon at flat RPS; event-loop lag spikes and one CPU core pegs; latency grows with concurrency (requests serialize), not with DB time; MongoDB stays healthy — the bottleneck is in-process CPU, not the database. |\n`recommendation-memory-leak` |\nA genuine bad deploy: `recommendation-service` rolls out `1.1.0` , a real Go regression that retains a ~256 KB profile per gRPC call in an unbounded map. Revert rolls back to the known-good image. |\nRSS/Go heap climb steadily to the memory limit → OOMKill (exit 137) → restart sawtooth; `product-catalog` /`api-gateway` see recommendation gRPC errors during restarts; onset matches the rollout. |\n\n| Scenario | Mechanism | What an RCA tool should find |\n|---|---|---|\n`traffic-spike` |\nThe `load-generator` Deployment is scaled to 5 replicas — real extra traffic across the whole stack. |\nUniform RPS increase everywhere; saturation (latency/errors) appears only at the weakest component, testing cause-vs-consequence reasoning. |\n`cpu-noisy-neighbor` |\nA batch `video-transcoder` workload is co-located (pod affinity) onto the nodes running `order-service` and burns all their cores. |\nNode CPU saturates (~100%); the Burstable `order-service` is starved far below its normal CPU; its dependencies (MySQL, Kafka) stay healthy — the cause is node-local CPU contention from a co-tenant, not the victim. |\n\n| Scenario | Mechanism | What an RCA tool should find |\n|---|---|---|\n`dns-slow-resolution` |\nChaos Mesh delays the app tier's packets to the cluster DNS service (~500 ms) — a real network condition on the DNS path, not fabricated answers — so every name lookup is slow. | Services show intermittent p95/p99 spikes on all outbound calls (each new connection front-loads a slow lookup), while every dependency and CoreDNS itself stay healthy (flat CPU). The tell is DNS query latency, not any one hop — the classic \"it's always DNS.\" |\n`network-delay-product-catalog` |\nChaos Mesh injects ~200 ms of egress latency on `product-catalog` (a `NetworkChaos` fault with a dead-man `spec.duration` ). |\n`api-gateway` latency for catalog-backed endpoints jumps to ~1 s while `product-catalog` 's own CPU/DB stay healthy; the delay is on the network path, not in the service or PostgreSQL. |\n\nLatent, slow-burn risks — **detection, not RCA** (see the note above). Each often has no acute symptom at onset; the \"should find\" column is the early-warning signal a tool should surface.\n\n| Scenario | Mechanism | What a tool should detect |\n|---|---|---|\n`pg-table-bloat` |\nAutovacuum is disabled on the (a per-table `products` table only`ALTER TABLE … SET (autovacuum_enabled=false)` , the daemon stays on) and a background job rewrites a hot row window, so dead tuples accumulate with nothing to reclaim them. |\nNo acute symptom at onset — `n_dead_tup` /dead-tuple ratio climbs on that one table with `last_autovacuum` old, the heap and GIN index grow on disk, cache-hit ratio drifts down, while the rest of the cluster vacuums normally. A tool should flag the developing per-table bloat before it turns into an outage. |\n`pg-stale-statistics` |\nAutoanalyze is off on `products` , stats are frozen at a good point, then ~10 % of rows are re-labelled into category values the histogram has never seen. |\nPlanner row estimates for the changed values are off by orders of magnitude (est. ~1, actual large) → poor plans; `n_mod_since_analyze` large, `last_analyze` old. The tell is stale statistics + a large unanalyzed change, not bloat. |\n`pg-vacuum-blocked` |\nA `REPEATABLE READ` \"reporting\" transaction takes a snapshot and stalls, pinning the xmin horizon, while a job churns rows. Revert terminates the stalled session by `application_name` so the horizon releases deterministically. |\nAutovacuum runs successfully (`last_autovacuum` recent) yet `n_dead_tup` still climbs — it can't remove tuples newer than the held snapshot; a very old transaction / `backend_xmin` age holds the horizon. Not lock contention — no query is blocked. |\n`pg-replication-lag` |\nChaos Mesh adds ~300 ms of egress latency to the current standby (selected by `role=replica` , so it follows failovers), throttling the WAL stream via flow control while a write job generates WAL. |\nThe standby stays streaming but its replication lag (seconds behind primary, and bytes) grows while the primary stays healthy; replica reads go stale and the failover safety margin shrinks. The tell is on the network path to the replica, not the engine — the replica's CPU/disk are fine. |\n`pg-checkpointer` |\nA write-heavy batch rewrites a large row window continuously, generating WAL far faster than baseline, so checkpoints fire on `max_wal_size` instead of the 5-min timer. |\nCheckpoints shift timed→requested (`num_requested` in `pg_stat_checkpointer` rises), checkpoint write/sync time and WAL rate climb, full-page writes amplify WAL; foreground write latency gets choppy while query rate is constant. The cost is checkpoint/WAL IO, not the queries. |\n\nMore scenarios (bad migrations, connection-pool leaks, Kafka consumer lag, cache eviction pressure, and others) are on the roadmap; each will follow the same real-mechanism, durable-revert rule.\n\nEdges: **solid** = HTTP, **dotted** = gRPC, **thick** = Kafka event.\n\n``` php\nflowchart LR\n    LG([load-generator]):::gen --> GW[api-gateway]:::gw\n\n    GW --> PC[product-catalog]\n    GW --> CART[cart-service]\n    GW --> ORD[order-service]\n    GW --> REV[review-service]\n    GW --> INV[inventory-service]\n    GW -. gRPC .-> REC[recommendation-service]\n    PC -. gRPC .-> REC\n    CART -- checkout --> ORD\n    ORD -- sync --> PAY[payment-service]\n\n    PC --> PGP[(products)]:::db\n    INV --> PGI[(inventory)]:::db\n    CART --> VK[(Valkey Cluster)]:::db\n    ORD --> MYO[(orders)]:::db\n    PAY --> MYP[(payments)]:::db\n    REV --> MG[(reviews)]:::db\n\n    ORD == order-events ==> KAFKA{{Kafka}}:::kafka\n    KAFKA ==> FUL[fulfillment-service]\n    FUL -- reserve --> INV\n    FUL --> MYO\n    FUL == shipment-events ==> KAFKA\n    KAFKA ==> ORD\n\n    subgraph PGsub [Percona PostgreSQL]\n        PGP\n        PGI\n    end\n    subgraph PXCsub [Percona XtraDB Cluster]\n        MYO\n        MYP\n    end\n    subgraph PSMDBsub [Percona Server for MongoDB]\n        MG\n    end\n\n    classDef gen fill:#dbeafe,stroke:#2563eb,color:#0b213f\n    classDef gw fill:#ede9fe,stroke:#7c3aed,color:#241046\n    classDef db fill:#dcfce7,stroke:#16a34a,color:#052e16\n    classDef kafka fill:#fef3c7,stroke:#d97706,color:#3a2606\n```\n\nEvery service exports OTLP — traces, SDK metrics, and logs — to a bundled\n`otel-collector`\n\nthat **discards data by default**; set `OTLP_ENDPOINT`\n\nto\nforward it to any backend (Coroot, Grafana, etc.). Logs also go to stdout, so\n`kubectl logs`\n\nstill works.\n\n``` php\nflowchart LR\n    SVCS[all services<br/>traces · metrics · logs] -- OTLP --> COL[otel-collector]\n    COL -- default --> NULL[discard]\n    COL -. OTLP_ENDPOINT .-> BACKEND[(your OTLP backend)]\n```\n\nEverything lab-related runs in the `default`\n\nnamespace; the database and Kafka\noperators live in their own (`pg-operator`\n\n, `pxc-operator`\n\n, `psmdb-operator`\n\n,\n`strimzi`\n\n, `valkey-operator`\n\n, `chaos-mesh`\n\n).\n\nEach is a separate deployable in `services/`\n\n, instrumented with OpenTelemetry.\n\n| Service | Language / framework | Role | Backing store |\n|---|---|---|---|\n`api-gateway` |\nPython · FastAPI | Public entry point; reverse-proxies to the services | — |\n`product-catalog` |\nGo · net/http + pgx | Product listing & search; calls recommendation over gRPC | PostgreSQL `products` |\n`recommendation-service` |\nGo · gRPC | Product recommendations | in-memory |\n`cart-service` |\nPython · Flask | Shopping cart | Valkey (cluster) |\n`order-service` |\nJava · Spring Boot | Orders; publishes `order-events` , consumes `shipment-events` |\nMySQL `orders` |\n`payment-service` |\nRust · Actix-web + sqlx | Payment processing | MySQL `payments` |\n`inventory-service` |\nPHP · FPM + nginx | Stock levels & reservations | PostgreSQL `inventory` |\n`review-service` |\nNode.js · Express + Mongoose | Product reviews | MongoDB `reviews` |\n`fulfillment-service` |\nGo · franz-go | Consumes `order-events` → reserves stock, writes shipments, emits `shipment-events` |\nMySQL `orders` , Kafka |\n`load-generator` |\nGo | Continuously drives realistic traffic through the gateway | — |\n`data-seeder` |\nPython | One-off Job that seeds the databases | all databases |\n\n`services/`\n\n— application sources, one directory per service;`variants/`\n\nsubdirectories hold**bad-deploy variants**: real code regressions built into plausibly-versioned images for deploy/rollback scenarios.`deploy/`\n\n— Kubernetes manifests (databases, Kafka, otel, apps) and helm values for the operators.`scenarios/`\n\n— the failure scenario library.`operator/`\n\n— the`FailureScenario`\n\noperator, its embedded web UI, and the`dbtool`\n\nused by database scenario workloads.`scripts/`\n\n—`deploy.sh`\n\n/`clean.sh`\n\n/`status.sh`\n\ndriven by the Makefile.", "url": "https://wpnews.pro/news/show-hn-rca-lab-test-observability-tools-on-real-failures", "canonical_source": "https://github.com/coroot/rca-lab", "published_at": "2026-08-14 20:31:13+00:00", "updated_at": "2026-08-14 20:42:59.035541+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops"], "entities": ["Coroot", "rca-lab", "Kubernetes", "OpenTelemetry", "PostgreSQL", "MySQL", "MongoDB", "Kafka"], "alternates": {"html": "https://wpnews.pro/news/show-hn-rca-lab-test-observability-tools-on-real-failures", "markdown": "https://wpnews.pro/news/show-hn-rca-lab-test-observability-tools-on-real-failures.md", "text": "https://wpnews.pro/news/show-hn-rca-lab-test-observability-tools-on-real-failures.txt", "jsonld": "https://wpnews.pro/news/show-hn-rca-lab-test-observability-tools-on-real-failures.jsonld"}}