Show HN: RCA-lab – test observability tools on real failures

Coroot released rca-lab, an open-source failure lab that runs real, reproducible incidents on a live Kubernetes cluster to test root-cause-analysis tooling, including AI-based systems. The lab deploys a polyglot microservice stack with real databases and uses genuine failure mechanisms such as allocation regressions and heavy analytics queries, with durable revert via custom resources. It is available on GitHub and requires kubectl and helm.

A realistic, reproducible failure lab for evaluating root-cause-analysis RCA tooling — human or AI — on a live Kubernetes cluster. Most RCA benchmarks replay canned telemetry from toy environments with synthetic faults toggled by feature flags. rca-lab takes the opposite approach: A real polyglot microservice stack Python, Go, Java, Node.js, Rust, PHP behind an API gateway, with continuous generated load. Real databases under production-grade operators : PostgreSQL, MySQL and MongoDB via Percona operators, a Valkey Cluster via the valkey-operator, Kafka via Strimzi — with seeded data volumes. Real failure mechanisms only. No chaos flags inside the apps. A GC pressure incident is a genuine allocation regression shipped as a new image version and rolled back later; a database incident is an analytics workload running heavy queries against the production database; a traffic spike is actually more traffic. Durable revert. Every failure scenario is a FailureScenario custom resource driven by an operator that restores the normal state when the scenario ends, is disabled, or is deleted — even across operator restarts. Rich telemetry, bring your own backend. Every service is instrumented with OpenTelemetry SDKs: traces, SDK-emitted metrics JVM/runtime/HTTP , and logs to stdout and OTLP, trace-correlated . Everything flows to a bundled otel-collector that discards data by default — point it at any OTLP backend with one variable. Requirements: kubectl + helm pointed at a cluster any distribution; a default StorageClass, ~8 CPU / 16 GiB across nodes for the full-size lab . git clone https://github.com/coroot/rca-lab && cd rca-lab make deploy everything: operators → databases → Kafka → apps → seed Single-node cluster kind/k3d/minikube : make deploy SINGLE NODE=1 Send telemetry somewhere e.g. Coroot, or any OTLP endpoint : make deploy OTLP ENDPOINT=my-backend:4317 Other variables: STORAGE CLASS=<name , SEED SIZE GB=<n 0 skips seeding , OTLP HEADERS=k=v , YES=1 no confirmation prompt . Re-running make deploy converges idempotently — it is also how you change any of these settings. Teardown: make clean KEEP DATA=1 keeps the database volumes Scenarios are Kubernetes custom resources: kubectl get failurescenarios kubectl patch failurescenario pg-analytics-queries --type=merge -p '{"spec":{"enabled":true}}' or use the web UI: kubectl port-forward svc/rca-lab-operator 8080 The UI lists every scenario grouped by category, with severity and live state, and starts or stops each one with a click. Each scenario documents its mechanism and the telemetry symptoms an RCA tool should be able to observe. Scenarios can also run on a cron schedule with a fixed duration — see scenarios/ . Never install rca-lab on a shared or production cluster.The scenario operator deliberately has the power to degrade workloads in its namespace. Every scenario uses a genuine real-world mechanism — never a synthetic fault flag inside the app — and reverts durably. Each carries an expectedSymptoms list that doubles as documentation and a grading rubric for RCA tools. The reliability category is a different kind of test. The other scenarios are acute incidents latency, errors, saturation that exercise RCA — given a symptom, find the cause. Reliability scenarios are latent, slow-burn risks bloat, stale stats, blocked vacuum, replication lag, checkpoint pressure that often produce no user-facing symptom at onset ; they exercise proactive detection — whether a tool flags a developing risk before it becomes an outage. Their expectedSymptoms are early-warning indicators, not incident symptoms. | Scenario | Mechanism | What an RCA tool should find | |---|---|---| pg-analytics-queries | An analytics-reporting workload runs heavy multi-join/aggregation queries full scans of the ~10 GB products table against the production PostgreSQL, through the same pgBouncer pool as the apps. | Elevated product-catalog / inventory-service latency; PostgreSQL CPU/IO saturation; new full-scan query fingerprints in pg stat statements attributable to the analytics-reporting workload. | pg-exclusive-lock | A stalled schema-migration transaction takes a real ACCESS EXCLUSIVE lock on the products table LOCK TABLE and then hangs holding it — the "a migration grabbed the lock and never let go" incident. | product-catalog queries on products block on the lock; its connection pool fills and the service goes unavailable, so api-gateway product endpoints error — yet PostgreSQL CPU/IO stay flat because nothing is executing. The tell is lock waits pg locks / pg blocking pids , not resource saturation. | mysql-lock-contention | A stalled transaction holds InnoDB row locks on the hot end of the orders table SELECT … FOR UPDATE , including the gap above the max id and then hangs — a transaction that grabbed locks and stalled. | Reads keep working InnoDB MVCC snapshots , but order-service writes new orders, status updates block and fail with Lock wait timeout exceeded 50s while PXC CPU/IO stay flat. The tell is row-lock waits information schema.innodb trx / performance schema.data lock waits , not resource saturation — and reads-fine/writes-blocked distinguishes it from Postgres's table-level lock. | mysql-analytics-queries | The same analytics-reporting actor runs large join/aggregation queries filesort, temp tables against the production MySQL orders database via HAProxy. | Elevated order-service /checkout latency and errors; PXC CPU/IO saturation; heavy statements in the slow query log attributable to the workload. | | Scenario | Mechanism | What an RCA tool should find | |---|---|---| order-service-gc-regression | A genuine bad deploy: order-service rolls out 1.1.0 , a real code regression that deep-copies every order read into an ineffective cache. GC pressure builds; revert rolls back to the known-good image. | p99 rises after the rollout while p50 stays flat; JVM allocation rate and GC time climb; heap sawtooth trends toward the limit; onset correlates exactly with the deployment event. | order-service-memory-leak | A genuine bad deploy: order-service rolls out 1.4.0 , a real regression that appends a batch of small "audit trail" objects per read into a registry that is never pruned. Slow leak of millions of tiny objects; revert rolls back to the known-good image. | p95/p99 creep up gradually no crash, no step change ; old-gen/live-set trends up; GC time and mixed-collection frequency rise as the live set grows; onset matches the rollout. Distinct from the fast OOM-crash leaks. | product-catalog-gc-pressure | A genuine bad deploy: product-catalog rolls out 1.1.0 , whose server-side "product cards" re-encode every returned product into large short-lived buffers on each read. Nothing retained no leak — pure allocation churn; revert rolls back. | Go GC CPU fraction and cycle frequency spike; allocation rate jumps while heap in-use stays bounded no OOM ; product-catalog CPU saturates/throttles and latency rises, propagating to api-gateway ; Postgres stays healthy. | review-service-event-loop | A genuine bad deploy: review-service rolls out 1.1.0 , adding a synchronous "content safety" CPU loop on the request path that blocks the single-threaded Node.js event loop for tens of ms per read. Revert rolls back. | p95/p99 balloon at flat RPS; event-loop lag spikes and one CPU core pegs; latency grows with concurrency requests serialize , not with DB time; MongoDB stays healthy — the bottleneck is in-process CPU, not the database. | recommendation-memory-leak | A genuine bad deploy: recommendation-service rolls out 1.1.0 , a real Go regression that retains a ~256 KB profile per gRPC call in an unbounded map. Revert rolls back to the known-good image. | RSS/Go heap climb steadily to the memory limit → OOMKill exit 137 → restart sawtooth; product-catalog / api-gateway see recommendation gRPC errors during restarts; onset matches the rollout. | | Scenario | Mechanism | What an RCA tool should find | |---|---|---| traffic-spike | The load-generator Deployment is scaled to 5 replicas — real extra traffic across the whole stack. | Uniform RPS increase everywhere; saturation latency/errors appears only at the weakest component, testing cause-vs-consequence reasoning. | cpu-noisy-neighbor | A batch video-transcoder workload is co-located pod affinity onto the nodes running order-service and burns all their cores. | Node CPU saturates ~100% ; the Burstable order-service is starved far below its normal CPU; its dependencies MySQL, Kafka stay healthy — the cause is node-local CPU contention from a co-tenant, not the victim. | | Scenario | Mechanism | What an RCA tool should find | |---|---|---| dns-slow-resolution | Chaos Mesh delays the app tier's packets to the cluster DNS service ~500 ms — a real network condition on the DNS path, not fabricated answers — so every name lookup is slow. | Services show intermittent p95/p99 spikes on all outbound calls each new connection front-loads a slow lookup , while every dependency and CoreDNS itself stay healthy flat CPU . The tell is DNS query latency, not any one hop — the classic "it's always DNS." | network-delay-product-catalog | Chaos Mesh injects ~200 ms of egress latency on product-catalog a NetworkChaos fault with a dead-man spec.duration . | api-gateway latency for catalog-backed endpoints jumps to ~1 s while product-catalog 's own CPU/DB stay healthy; the delay is on the network path, not in the service or PostgreSQL. | Latent, slow-burn risks — detection, not RCA see the note above . Each often has no acute symptom at onset; the "should find" column is the early-warning signal a tool should surface. | Scenario | Mechanism | What a tool should detect | |---|---|---| pg-table-bloat | Autovacuum is disabled on the a per-table products table only ALTER TABLE … SET autovacuum enabled=false , the daemon stays on and a background job rewrites a hot row window, so dead tuples accumulate with nothing to reclaim them. | No acute symptom at onset — n dead tup /dead-tuple ratio climbs on that one table with last autovacuum old, the heap and GIN index grow on disk, cache-hit ratio drifts down, while the rest of the cluster vacuums normally. A tool should flag the developing per-table bloat before it turns into an outage. | pg-stale-statistics | Autoanalyze is off on products , stats are frozen at a good point, then ~10 % of rows are re-labelled into category values the histogram has never seen. | Planner row estimates for the changed values are off by orders of magnitude est. ~1, actual large → poor plans; n mod since analyze large, last analyze old. The tell is stale statistics + a large unanalyzed change, not bloat. | pg-vacuum-blocked | A REPEATABLE READ "reporting" transaction takes a snapshot and stalls, pinning the xmin horizon, while a job churns rows. Revert terminates the stalled session by application name so the horizon releases deterministically. | Autovacuum runs successfully last autovacuum recent yet n dead tup still climbs — it can't remove tuples newer than the held snapshot; a very old transaction / backend xmin age holds the horizon. Not lock contention — no query is blocked. | pg-replication-lag | Chaos Mesh adds ~300 ms of egress latency to the current standby selected by role=replica , so it follows failovers , throttling the WAL stream via flow control while a write job generates WAL. | The standby stays streaming but its replication lag seconds behind primary, and bytes grows while the primary stays healthy; replica reads go stale and the failover safety margin shrinks. The tell is on the network path to the replica, not the engine — the replica's CPU/disk are fine. | pg-checkpointer | A write-heavy batch rewrites a large row window continuously, generating WAL far faster than baseline, so checkpoints fire on max wal size instead of the 5-min timer. | Checkpoints shift timed→requested num requested in pg stat checkpointer rises , checkpoint write/sync time and WAL rate climb, full-page writes amplify WAL; foreground write latency gets choppy while query rate is constant. The cost is checkpoint/WAL IO, not the queries. | More scenarios bad migrations, connection-pool leaks, Kafka consumer lag, cache eviction pressure, and others are on the roadmap; each will follow the same real-mechanism, durable-revert rule. Edges: solid = HTTP, dotted = gRPC, thick = Kafka event. php flowchart LR LG load-generator :::gen -- GW api-gateway :::gw GW -- PC product-catalog GW -- CART cart-service GW -- ORD order-service GW -- REV review-service GW -- INV inventory-service GW -. gRPC .- REC recommendation-service PC -. gRPC .- REC CART -- checkout -- ORD ORD -- sync -- PAY payment-service PC -- PGP products :::db INV -- PGI inventory :::db CART -- VK Valkey Cluster :::db ORD -- MYO orders :::db PAY -- MYP payments :::db REV -- MG reviews :::db ORD == order-events == KAFKA{{Kafka}}:::kafka KAFKA == FUL fulfillment-service FUL -- reserve -- INV FUL -- MYO FUL == shipment-events == KAFKA KAFKA == ORD subgraph PGsub Percona PostgreSQL PGP PGI end subgraph PXCsub Percona XtraDB Cluster MYO MYP end subgraph PSMDBsub Percona Server for MongoDB MG end classDef gen fill: dbeafe,stroke: 2563eb,color: 0b213f classDef gw fill: ede9fe,stroke: 7c3aed,color: 241046 classDef db fill: dcfce7,stroke: 16a34a,color: 052e16 classDef kafka fill: fef3c7,stroke: d97706,color: 3a2606 Every service exports OTLP — traces, SDK metrics, and logs — to a bundled otel-collector that discards data by default ; set OTLP ENDPOINT to forward it to any backend Coroot, Grafana, etc. . Logs also go to stdout, so kubectl logs still works. php flowchart LR SVCS all services<br/ traces · metrics · logs -- OTLP -- COL otel-collector COL -- default -- NULL discard COL -. OTLP ENDPOINT .- BACKEND your OTLP backend Everything lab-related runs in the default namespace; the database and Kafka operators live in their own pg-operator , pxc-operator , psmdb-operator , strimzi , valkey-operator , chaos-mesh . Each is a separate deployable in services/ , instrumented with OpenTelemetry. | Service | Language / framework | Role | Backing store | |---|---|---|---| api-gateway | Python · FastAPI | Public entry point; reverse-proxies to the services | — | product-catalog | Go · net/http + pgx | Product listing & search; calls recommendation over gRPC | PostgreSQL products | recommendation-service | Go · gRPC | Product recommendations | in-memory | cart-service | Python · Flask | Shopping cart | Valkey cluster | order-service | Java · Spring Boot | Orders; publishes order-events , consumes shipment-events | MySQL orders | payment-service | Rust · Actix-web + sqlx | Payment processing | MySQL payments | inventory-service | PHP · FPM + nginx | Stock levels & reservations | PostgreSQL inventory | review-service | Node.js · Express + Mongoose | Product reviews | MongoDB reviews | fulfillment-service | Go · franz-go | Consumes order-events → reserves stock, writes shipments, emits shipment-events | MySQL orders , Kafka | load-generator | Go | Continuously drives realistic traffic through the gateway | — | data-seeder | Python | One-off Job that seeds the databases | all databases | services/ — application sources, one directory per service; variants/ subdirectories hold bad-deploy variants : real code regressions built into plausibly-versioned images for deploy/rollback scenarios. deploy/ — Kubernetes manifests databases, Kafka, otel, apps and helm values for the operators. scenarios/ — the failure scenario library. operator/ — the FailureScenario operator, its embedded web UI, and the dbtool used by database scenario workloads. scripts/ — deploy.sh / clean.sh / status.sh driven by the Makefile.