Saturation at GitHub: the saga continues

GitHub suffered a site-wide outage on September 13 after an internal data-cleanup job saturated the primary node of a shared database cluster that stores permission data read on nearly every authenticated request, according to GitHub's incident report. The job's pacing safeguard monitored only replica lag, which stayed low, so the job kept writing until the primary exhausted its available connections at 07:33 UTC; a retry loop around token creation then re-sent failing writes and held the database saturated. Monitoring declared the incident at 08:50 UTC, first responders shed internal load and paused the job, and all services recovered by 10:44 UTC.

When it comes to public incident write-ups, GitHub continues to be the gift that keeps on giving. They keep suffering collateral damage from the AI boom, as more developers using AI means more load on their system. Their developer focus also means that they provide some technical details about these incidents. Today I’m writing about the incident that happened to them on September 13. The report has the vague title Incident with several GitHub Services https://www.githubstatus.com/incidents/0rn90wk115q9 , which was the title when the incident was declared. It’s a shame they don’t re-title their incidents later on. The report is short enough that I’ll simply excerpt the three paragraphs that discuss the failure mode. The cause was an internal data-cleanup job that began writing to a shared database cluster at 07:33 UTC. That cluster stores permission data read on nearly every authenticated request. The safeguard that was pacing the background job watched only one health signal — how far the database replicas were lagging — and that signal stayed low the whole time. It did not account for the load building on the primary itself, so the job kept writing while the primary quietly ran toward its limit. When the primary ran out of available connections, requests that needed it could not complete. First, there was no quick timeout on these database calls, so request handlers waited on the stalled database instead of failing fast, and the shared request-handling capacity degraded into site-wide errors. Second, a retry loop around token creation kept re-sending the writes that were already failing, which held the database saturated rather than letting it recover. Monitoring declared the incident at 08:50 UTC, but due to the broad impact and amplification from token creation, it took time to identify the source of the load. First responders mitigated by shedding internal load and pausing the job, and all services recovered by 10:44 UTC. Here’s my attempt to depict this text with a diagram, and to rephrase the above in my own words. Within GitHub, there is a database cluster in the typical SQL database style cluster configuration, with a primary and read replicas, where the primary is responsible for handling all of the writes. This cluster takes online traffic: meaning that activity from external users will generate queries against this cluster. GitHub ran an offline job i.e., not driven by user traffic that made queries against the database to clean up unneeded data. The rate at which this cleanup job queried the database was governed by a safeguard mechanism that monitored the health of the database: the safeguard system would reduce the rate of queries if the database appeared to be going unhealthy. The safeguard monitored the health of the database using the replica lag if they were using MySQL, presumably they were tracking Seconds Behind Source https://dev.mysql.com/doc/refman/26.7/en/show-replica-status.html . An increase in this replica lag can be a symptom of high load on the replicas. In this scenario, the replicas themselves were healthy, and the replica lag stayed below the threshold that the safeguard was monitoring. However, it was the primary node of the database cluster that was at risk of saturation. Specifically, the primary reached its maximum number of client connections on MySQL, this is controlled by the max connections https://dev.mysql.com/doc/refman/26.7/en/server-system-variables.html sysvar max connections server system variable . This means that any attempt to query the database over a new connection would fail with an error on MySQL, this would be ER CON COUNT ERROR https://dev.mysql.com/doc/mysql-errors/26.7/en/server-error-reference.html error er con count error , with an error message of Too many connections . When some web servers attempted to make calls to this database, the requests did not succeed immediately, but neither did they fail immediately. Instead, these requests were stuck waiting. Unfortunately, the timeouts configured on these request handlers were long. This meant that these blocked requests accumulated in the impacted web servers. Each in-flight request consumes some amount of resources, and enough of these blocked requests accumulated in the web servers that they themselves became saturated, which led to sitewide impact. To make matters work, there was a token creation process that kept trying to query the primary, failing, and then retrying. This placed a continual load on the primary that made it more difficult for the system to recover. Aside: some unanswered technical questions Here some questions I had that I couldn’t figure out from the text. How did exhausted connections lead to timeouts on the web servers? Was it the case that the web servers were using connection pooling, not all of the connections in the pool were active, and the clients were stuck waiting for a new active connection that never came? How did that background job lead to the primary exceeding the maximum number of connections? Did this job consume a surprisingly number of connections? How close was the database to the limit before the job started? — You can never truly know where the safety boundary is When you have an offline job running against an online database, there’s always a risk that it can negatively impact the performance of the database, and thereby affect customer traffic. The nice thing about offline jobs is that they are, in principle, fully controllable by the organization. With user traffic, on the other hand, you are at the mercy of your users. And, so, GitHub had an automated system the safeguard monitor the health of the database while the cleanup job was running, to ensure that the job was not applying too much load to the database. If the load was getting high, then the safeguard would slow down the rate at which the cleanup job was querying the database. The problem was that the metric being monitored by the safeguard did not give a complete picture of the health of the cluster. As far, as the safeguard was concerned, the database was still healthy, even though it was running out of connections. This is a great example of a gray failure : when your internal health monitoring systems register, incorrectly, that the system is healthy. In other word, the metrics are good, but the users are unhappy. Putting things in resilience engineering terms, the system misjudged the location of the safety boundary https://surfingcomplexity.blog/2020/05/25/sre-cse-and-the-safety-boundary/ . There was no signal that the system was getting close to being in an unsafe state until the boundary was crossed. Note that we never actually know where the safety boundary is until we actually cross it. But crossing the safety boundary is very, very bad. And so we always have to make an estimate about where the safety boundary is, and then we avoid getting too close. But there are so many limits in the system, that the chances of us not monitoring all of them is, tragically, quite high. Offline jobs and online databases: sometimes you gotta run ’em In an ideal world, we would want all of the query traffic for our online databases to be, well, online traffic. That’s why, for example, we don’t run our analytics queries against our online databases. However, there are scenarios where you have no choice: you have to run a job against an online database, because you want to effect some sort of change to the online data. This is a great example: here the job has to be run on the online database because the goal of the job is to make a change to that data. The double-edged sword of cleanup When it comes to the topic of “cleanup”, both for data and code, once you’ve seen enough incidents, you’ll notice a pattern: cleanup is a frequent risk. Sometimes incidents happen because a cleanup job was attempted, and the thing being cleaned up was still in use. And sometimes incidents happen because some entity that wasn’t even used anymore interacted in an unexpected way with a different part of the system. In other words: an absence of cleanup contributed to the incident. So, not cleaning up unused stuff is a risk: it can lead to incidents. But the act of cleanup is itself a change to the system that carries risk. It’s all risk tradeoffs. The asymmetry of misconfigured timeouts Timeouts are an essential tool in building reliable distributed systems. But there’s an asymmetry in the timeouts-are-too-short problem versus the timeouts-are-too-long problem. If your timeouts are too short, you’ll notice this during the normal operation of your system, by seeing excessive timeouts happening. And, so, when that happens, you adjust the timeouts to be longer. However, timeouts that are too long are harder to detect. If you’re lucky, you might notice some sluggishness in your user interface and diagnose the problem as being related to timeouts being too long. But, in many cases, you won’t have any signal at all that timeouts are too long until they contribute to an incident. As we saw here, long timeouts create a saturation risk by increasing the number of in-flight requests in a service, thereby depleting its resources. You probably have timeouts that are configured to be too long in your system, and nobody has noticed them yet. Retries help, until they hurt Retries are yet another essential tool in building reliable distributed systems. Because transient failures are common, retries can effectively mask these failures from your users: they’ll never know about that one bad pod in your deployment. But retries increase load, and in a saturation failure mode, you can get a pathological behavior where retries exacerbate the incident. That’s what happened in this case, where a particular service token creation kept retrying over and over again, leading to unending load on the database. You need retries But, like misconfigured timeouts, it can be hard to catch misconfigured retries until the incident happens. That’s exactly why our industry has a specific term for failure modes where retries made things worse: retry storms . When you’re overloaded, you need to reduce the load Let’s look at the last quoted paragraph again, since it has some details about the incident response emphasis mine . Monitoring declared the incident at 08:50 UTC, but due to the broad impact and amplification from token creation, it took time to identify the source of the load . First responders mitigated by shedding internal load and pausing the job, and all services recovered by 10:44 UTC. Those of us who do incident response know that our first priority is always getting the system back to healthy. If you can quickly identify the source of the problem, that’s great Go ahead and roll back that problematic deploy. But often we can’t tell what’s causing the problem. In a saturation failure mode, we can often identify which service has gotten overloaded and, very often, it’s a database , but it isn’t obvious what the source of the problem actually is. In scenarios like this, you want the responders to be able to: - quickly identify the sources of load which queries, and who is making them? - manually shed load block specific queries/sources To have that operational tooling at your fingertips during an incident, you need to have built it in advance. During the incident, it’s too late, you’re stuck with whatever tooling is available. Your systems will eventually become overloaded, this I promise you. Your online databases at particular risk. The time to prepare for this is now. The solution, as always, is to increase complexity Here’s the last paragraph of the write-up: To prevent recurrence, we are rate-limiting background jobs against shared, customer-serving databases by default, and adding automatic pausing and paging on primary-server load rather than replication lag alone. We are also surfacing running background work directly alongside database health signals so responders can see and pause it without leaving those dashboards, bounding retries in the token-issuing path, and adding request-level timeouts so one unhealthy database cannot consume shared web server capacity. In addition, we are breaking apart this database cluster to remove the single point of failure. We will be moving various service-specific data, including the authorization data, out of this shared cluster in the next two weeks. The proposed new interventions include: - adding rate-limits - monitoring additional metrics primary-server load - adding request-level timeouts - sharding the database cluster These all look good interventions to me. But note how these all increase complexity They all involve adding new stuff to the system, as post-incident interventions always do. I am on record as stating that increasing complexity in order to improve reliability is inevitable https://surfingcomplexity.blog/2026/01/31/ashby-taught-us-we-have-to-fight-fire-with-fire/ . But we should be sanguine about this. By increasing complexity, we eliminate known failure modes at the cost of introducing novel failure modes. And introduce them we do.