A Central US autoscaling failure hit authentication, Actions and Copilot as GitHub races to move production traffic onto Azure.
By Ryan Merket · Published
Primary source: GitHub Blog
Why it matters #
GitHub is infrastructure for code hosting, authentication, CI and AI development. A single scaling error can now halt several stages of software delivery at once.
GitHub said Thursday that a capacity failure in its Central US data center, followed by a retry storm that multiplied Copilot traffic, caused the widespread August 17th outage that lasted 7 hours and 47 minutes.
In a postmortem published August 20th, GitHub Chief Technology Officer Vlad Fedorov said traffic reached a new peak while a critical infrastructure component failed to scale. The resulting pressure spread into authentication and disrupted GitHub.com, APIs, pull requests, issues, Actions and Copilot.
Fedorov joined GitHub after co-founding the data governance startup UserClouds. He previously spent 12 years at Facebook, now Meta, where GitHub says he led engineering organizations across privacy, advertising and platform infrastructure. The August post is his third public reliability update since March, putting GitHub's infrastructure response directly under its top engineering executive.
An autoscaler watched the wrong limit
GitHub's detailed root cause analysis traces the initial failure to an Istio service-mesh sidecar that reached its concurrency limit. Its autoscaling policy monitored the host service rather than the sidecar's capacity, so new resources did not arrive as traffic increased.
That bottleneck spread until four HAProxy nodes exhausted their flow limits. The gateway authentication path then developed severe latency and failures, affecting services that depended on it. At the peak, GitHub measured error rates of about 20% for web and API traffic and roughly 50% for archive and raw-content downloads. SAML and OIDC authentication, SCIM provisioning and Team Sync were also affected.
Most GitHub services recovered by 16:36 UTC, just over three hours after the incident began. Actions remained degraded until about 18:03 UTC, while the Copilot Token Service did not fully recover until 21:02 UTC.
The Copilot delay came from a second failure mode. Slow responses from one internal endpoint activated a latent retry bug in VS Code, increasing Copilot Token Service traffic from its usual 7,000 to 9,000 requests per second to between 70,000 and 100,000. GitHub temporarily blocked token requests with HTTP 403 responses, reduced gateway retries and restored traffic gradually. GitHub also said scraping attacks against code-download endpoints complicated the recovery.
Capacity spending has not ended the capacity failures
GitHub described the August 17th event as its second significant incident of the month. On August 6th, GitHub Actions was degraded for more than nine hours after a routine deployment exposed an existing capacity and concurrency weakness. At the peak of that incident, 71% of workflow runs experienced infrastructure failures, while 75% of the remaining runs were delayed by more than five minutes.
The pattern extends beyond August. GitHub's July availability report counted eight incidents that degraded services during that month. GitHub had already acknowledged in March that rapid load growth, tightly coupled infrastructure and weak defenses against misbehaving clients were allowing localized problems to spread across critical services.
GitHub says monthly commits rose from 1.4 billion in April to 2.9 billion by August. Its charts also show merged pull requests approaching 130 million a month and new repository creation reaching about 24 million a month.
GitHub has responded with a large infrastructure buildout. Fedorov said GitHub added more than 3 million CPU cores and 120 petabytes of high-speed storage, installing as much hardware as the available power in its existing data centers allowed. GitHub is also accelerating its migration to Microsoft's Azure cloud.
Azure now handles roughly 58% of GitHub's platform load and half of its Git operations, according to the postmortem. Azure carried 12% of platform load in May. GitHub said in July that it was working toward moving GitHub.com production traffic out of its own data centers by the end of 2026, with a second Azure region intended to provide regional resilience.
The August outage shows the risk during that transition. GitHub has added enormous amounts of compute, yet individual dependencies, autoscaling policies and retry behavior can still turn a traffic peak into a failure spanning authentication, code review, automated builds and AI coding tools.
GitHub rewrites its retry rules
GitHub said it is correcting the sidecar autoscaling policy, auditing Istio concurrency and scaling limits, and improving load-balancer monitoring and regional failover safeguards. Engineers are also addressing the VS Code behavior that amplified Copilot token requests.
Across GitHub's services, teams will apply consistent retry limits, retry budgets and variable timeouts to reduce the chance that one slow dependency creates a cascading load spike. GitHub is also reviewing lower-priority CPU and memory alerts for components that could fail under sudden demand.
The longer-term work involves separating critical systems and removing shared dependencies so an overloaded component cannot take unrelated products with it. GitHub also plans to roll out an architecture that increases repository read capacity with the number of readers, beginning with its largest monorepos.
Those projects were already underway before August 17th. The outage established that GitHub's traffic growth is still moving faster than parts of the reliability program designed to contain it.