NeatContext infra-team profile · knowledge/infra-team attached
Root cause found: the 08:58Z pool-size cut. Revert it with a RELOAD
.
Evidence, checked with read-only tools
- The deploy log shows
default_pool_size 100 → 40
on pgbouncer at 08:58Z
— minutes before the first checkout-api 5xx. - Logs confirm the pool saturated at
40/40
with clients queuing — the exact signature of postmortem INC-0977. - The Postgres primary is healthy and far below
max_connections=200
— the ceiling is the pool setting, not the database.
Safe actions
- Revert
default_pool_size
to 100
and RELOAD
pgbouncer — zero dropped connections. - Do not fail over the primary — the TSG forbids failover for saturation; it would drop every in-flight connection and widen the outage.
- Watch the pool drain and confirm checkout-api 5xx returns to baseline.
Cites the exact runbook and TSG linespostgres-connection-pool.md:38-42db-failover-tsg.md:6-9inc-2026-0511-pool-sizing-regression.md:40-44 ✳General SRE agent No profile · no runbooks · no team context
Elevated 5xx can have many causes — here is a general checklist.
- Review recent application deployments and roll back if anything looks suspicious.⚠The app deploy is innocent — the change that matters is the 08:58Z pgbouncer pool-size cut, which only the team's deploy log and runbook point to.
- Check database health; if connections are stuck, restart the database or fail over to a replica.⚠Failing over a saturated-but-healthy primary drops every in-flight connection across every service — db-failover-tsg.md forbids exactly this.
- Increase max_connections on Postgres to relieve the pressure.⚠Wrong knob: the pgbouncer pool ceiling is the constraint. Raising max_connections just moves the failure into the database.
- Check the logs for errors and escalate to the appropriate team if the issue persists.
∅ No sources cited — generic best practice, nothing to verify.