{"slug": "codex-broke-git-openai-rebuilt-the-engine", "title": "Codex Broke Git. OpenAI Rebuilt the Engine", "summary": "OpenAI built Pando, a new Git backend engine, after commits to its main branch surged 1,378% by late 2025 — with projections of 2,400% by year-end — and monorepo reliability fell below 90%, sometimes to 87.1%, under Codex agent traffic. Pando processes updates to separate branches independently and caches Git packfiles with over a 99% hit rate, cutting median push latency by nearly 80% (roughly 5x) while serving almost 1 petabyte of Git data daily and handling peaks above 57 terabytes of packfiles per hour. Written primarily in Rust by a single engineer with significant implementation by Codex agents, Pando went from kickoff to production in 10 weeks and keeps standard git push and git pull commands, preserving existing workflows and CI pipelines.", "body_md": "## When agents turn commits into a traffic jam\n\nCoding agents are fundamentally shifting the cadence of development. Where human developers might push a handful of commits daily, AI agents can generate bursts of branch updates and CI activity that far exceed what traditional **Git infrastructure** was built to handle. OpenAI experienced this firsthand, seeing commits to its main branch surge by 1,378% by late 2025, with projections of 2,400% by year-end.\n\nThis sheer volume of concurrent operations creates a bottleneck. Unrelated branch updates compete for shared server resources, transforming concurrency into debilitating contention. At OpenAI, this dynamic caused monorepo reliability to plummet below 90%, sometimes hitting 87.1% due to timeouts and reference conflicts.\n\nThe problem isn't the individual code changes, but the system's inability to process them in parallel. If hundreds of agents push changes to different branches simultaneously, all those pushes can stall, even if the changes are entirely independent. This cascades into stalled pushes and unreliable source control, slowing every agent and human developer in the loop.\n\nThis operational friction highlights a critical scaling challenge. Git servers couldn't't keep up with **[Codex](https://www.stork.ai/en/codex)**'s demands, demonstrating that as agentic development matures, the underlying infrastructure must evolve to match its unprecedented velocity and throughput.\n\n## Pando keeps Git’s face and swaps its engine\n\nOpenAI’s solution, **Pando**, is a new backend engine for Git, not a new version control system. Engineers and Codex agents continue to use familiar `git push` and `git pull` commands. This backward compatibility was crucial, minimizing disruption to existing developer workflows, CI pipelines, and extensive tooling while radically changing the underlying infrastructure.\n\nKey to Pando’s design is its handling of write paths. Unlike traditional Git servers that often serialize updates or lock the entire repository, Pando processes updates to separate branches independently. This means a push to branch A no longer waits for unrelated pushes to branch B, eliminating shared bottlenecks that previously degraded monorepo reliability to below 90%.\n\nPando also aggressively caches **Git packfiles**, which are compressed bundles of Git objects. When dozens of CI jobs request the same base code, the server doesn't regenerate identical data repeatedly. This caching achieves over a 99% hit rate, drastically reducing compute overhead and saturating multi-gigabit network links, accommodating peaks exceeding 57 terabytes of Git packfiles per hour.\n\nThis architectural shift delivered significant performance gains. Median push latency dropped by nearly **80%**, representing a roughly 5x speedup. Pando now serves almost 1 petabyte of Git data daily, a testament to its scalability. Designed primarily in Rust by a single engineer, with significant implementation by Codex agents, Pando went from kickoff to production in just 10 weeks.\n\n## The cache trick behind the speedup\n\nCore to Pando's performance is a sophisticated caching strategy for Git **packfiles**. These compressed bundles efficiently package Git objects—like commits, trees, and blobs—for transfer between repositories and clients. In a high-throughput environment, repeated CI requests for the same code base would ordinarily force Git servers to continually re-generate identical packfiles, consuming significant compute cycles and time.\n\nPando addresses this by aggressively caching these packfiles. When dozens of CI jobs request the same code, the system reuses pre-generated packfiles instead of recalculating them. This dramatically reduces server computation, serving data much more efficiently and at network saturation speeds.\n\nThis caching, combined with Pando's decoupled branch updates, has yielded impressive results. OpenAI reports median push latency fell by nearly **80%**, a fivefold speedup that transformed their development experience. Pando now serves close to a **petabyte of Git data per day**, accommodating peak demands exceeding 57 terabytes of packfiles per hour. For further technical details, explore [Scaling Git at OpenAI](https://openai.com/index/scaling-git-at-openai/).\n\nEnjoying this? Get one like it in your inbox each morning.\n\none email a day · unsubscribe in two clicks · no third-party tracking\n\n## Faster writes come with a consistency bill\n\nFaster writes come with a consistency bill. Pando achieves its impressive throughput by making a crucial architectural trade-off: reads across multiple branches are not guaranteed to reflect one perfectly synchronized, repository-wide snapshot. While individual branch histories remain consistent, the system prioritizes immediate write availability over atomic multi-branch read consistency.\n\nThis compromise works for OpenAI’s agent-driven development, where hundreds of parallel tasks frequently push changes to isolated branches, and the immediate state of a single branch is often more critical than a perfectly synchronized, global view. However, systems expecting atomic, repository-wide reads across disparate branches would need to account for this eventual consistency model.\n\nPando’s rapid development — designed by one engineer, implemented largely by **Codex agents**, and moving from kickoff to production in just 10 weeks — underscores a broader lesson. Agent-scale development is pushing developer infrastructure choices into the spotlight, demanding a fundamental rethinking of tools that were designed for human-centric workflows. Pando's ability to serve almost a **petabyte of Git data** every day, while cutting median push latency by nearly 80%, demonstrates both the opportunity and the pressure to innovate.\n\n## Frequently Asked Questions\n\n### What is Pando?\n\nPando is OpenAI’s proprietary Git backend, built to handle heavy concurrent reads and writes while preserving familiar Git commands and workflows.\n\n### Does Pando replace Git?\n\nNo. Pando replaces the backend serving Git data; developers and agents can continue using standard commands such as git push and git fetch.\n\n### How did Pando improve Git performance?\n\nIt processes updates to different branches independently and caches Git packfiles, reducing contention and repeated work for CI requests.\n\n### What trade-off does Pando make?\n\nReads spanning multiple branches may not show one perfectly synchronized repository-wide view, trading that guarantee for more available, concurrent writes.", "url": "https://wpnews.pro/news/codex-broke-git-openai-rebuilt-the-engine", "canonical_source": "https://www.stork.ai/blog/codex-broke-git-openai-rebuilt-the-engine", "published_at": "2026-11-08 01:16:54+00:00", "updated_at": "2026-11-09 01:55:48.044297+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["OpenAI", "Codex", "Pando", "Git", "Rust"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/codex-broke-git-openai-rebuilt-the-engine", "markdown": "https://wpnews.pro/news/codex-broke-git-openai-rebuilt-the-engine.md", "text": "https://wpnews.pro/news/codex-broke-git-openai-rebuilt-the-engine.txt", "jsonld": "https://wpnews.pro/news/codex-broke-git-openai-rebuilt-the-engine.jsonld"}}