We Let AI Agents Rewrite a 92M-Message-a-Day Service in Go

Checkly, a monitoring platform, rewrote its Results Daemon from Node.js to Go using Claude Code, an AI agent, and the service now processes 92 million messages a day with zero incidents and a 70% reduction in running pods. The rewrite succeeded because Checkly built a black-box test harness with golden files and real boundary services before the agent wrote code, enabling byte-to-byte parity with the legacy system.

Our Results Daemon processes about 92 million messages a day. We recently rewrote it from Node.js to Go, and we let Claude Code write it. We wanted to know whether we could trust an agentic rewrite for a critical, high-throughput production service rather than a prototype. It shipped with zero incidents, a 70% reduction in running pods, and a lighter database load. Go's stronger type system also proved a better fit for agents than JavaScript, adding protection against regressions and letting us ship faster and with more confidence. What made it work was the test harness we built before the agent wrote a line. Here's how we designed it, and the principles you can reuse on your own legacy services. The problem the-problem Checkly is a monitoring platform that runs synthetic checks, automated scripts that emulate real users, and uptime checks that confirm a system component is operational. A runner component executes all of these and produces a result that has to be processed, stored, and alerted on. Since we introduced uptime checks, and with the company's overall growth, the volume of checks run on our platform has doubled over the last year. Some components started degrading under that load. The most notable one was Results Daemon, a Node.js component written in vanilla JavaScript. Results Daemon is a background worker. It consumes results from our runner, writes them to databases, determines the check outcome, issues alerts, and schedules retries as needed. It also publishes WebSocket updates to our CLI and UI. In total, this component processes approximately 92,000,000 messages every day , around 40,000,000 of them check results and the rest WebSocket publishes. At that scale, it was becoming a bottleneck. It paged our on-call engineers more often, and limited type safety made every change harder to land safely. So we decided to rewrite Results Daemon in Go using agentic engineering. Designing a test harness designing-a-test-harness We built the harness before we started the rewrite. If an agent is going to write the code, something other than a human reviewer has to define what correct means. We built it on these design principles: - The harness tests the component as a black box. There is zero coupling between the code or language of the system under test and the harness itself. - Every test case provides an input and expects a deterministic output, with all outputs recorded in "golden files." These are generated against the legacy system and later used by the rewrite to assert byte-to-byte parity. - Non-deterministic fields, such as UUIDs or timestamps generated during the test itself, are written as <uuid or <timestamp and are still type-checked, to minimize the risk of differences in behavior slipping through. - Surrounding components databases, queues, caches, other services are categorized as boundaries. These are managed strictly by the harness, and the system under test is only pointed at them using environment variables. - Boundaries use real instances of the service in testing. If data is written to PostgreSQL, the harness uses a real PostgreSQL container rather than an emulated one. - For simpler boundaries such as SQS queues, we built our own emulator instead of using ElasticMQ or LocalStack. We found it more performant and simpler, both in our tests and in our assertions. - A set of "oracle" classes fetches the test output and asserts whether the test failed. For example, PostgresOracle.expectResultToMatchSnapshot testId fetches the relevant output and asserts its byte-level accuracy against the established golden file. Three technical choices carried the harness: Playwright , a testing framework built for reliability, with strong tooling for network interception and parallel test execution. Our own synthetic monitoring offering is built on Playwright, so we already knew it well, and its black-box model fit what we were doing here. Docker Compose , the simplest way to start and tear down containers for our boundaries, both locally and in CI. Toxiproxy , a TCP proxy that emulates network conditions and let us test how the system behaves when surrounding infrastructure fails. Building test cases building-test-cases With the architecture settled, the next step was building actual test cases. Results Daemon's outputs depend on two things: - The check result, the data object representing the outcome of a check execution. It holds the outcome state of the check run Failed | Degraded | Success and other metadata. - The check configuration at the time the result was received: retry rules for rescheduling, alert rules for notifications, and so on. From there it followed quickly that behavior coverage depends directly on the diversity of the inputs. A harness that covers every combination of check result and configuration covers every possible code path, with zero coupling to the implementation. That gave us our first principle: - The quality of the harness depends on the quality of the inputs you can provide. The more diverse and realistic the inputs, the greater the coverage of code paths and behaviors. In our case, the number of outcomes is represented by accountConfigs × groupConfigs × checkConfigs × resultOutcomes . We generated those inputs from our internal data lake. We extracted all account, group, and check configurations along with every result outcome from the last 24 hours, which is the longest interval we schedule checks at. We loaded all of it into a ClickHouse instance, and each dataset was "collapsed" into a unique set of configurations and outcomes, each tagged with its number of occurrences. We then used that data to seed realistic scenarios and pin them to business rules. For example, a re-dispatch takes its runtime from the account when the job pins none . After generating the test cases, we used code coverage reports to estimate how effective the harness was, targeting between 90% and 100%. We also fed those reports back to the agent to review, identify gaps, and propose test cases for anything missed. We expected some gaps to remain, but with all relevant files sufficiently covered and every scoped feature included, we considered the harness ready for a test run. - Code coverage is a great metric to track as you start out with your initial set of test cases, since high code coverage means the core behaviors are covered. However, it does measure direct system actual behavior and is therefore likely to miss edge cases. We also built a separate suite of tests that caught failure modes when infrastructure failures occur, e.g., PostgreSQL going down, using Toxiproxy. These were focused on how behavior changes and what data is lost when a piece of infrastructure goes down. The agentic rewrite the-agentic-rewrite A prerequisite for this approach was an earlier migration to a monorepo running on Tilt https://tilt.dev/ , which lets us spin up the whole platform end to end in a local development environment. We dispatched an instance of Claude Code, using Fable https://www.anthropic.com/claude/fable , and gave it one instruction: "build a Go service that consumes data from input queues, processes the messages, and writes the outputs to downstream applications such as databases, caches, and other queues," with the legacy implementation available as a reference. The main acceptance criterion was that the test harness passed against the new implementation. The agent ran overnight and produced a deployable service of about 13,000 lines of application code, architecturally mirroring the legacy implementation. It also kept token usage within the daily limits of a $200 subscription. We ran an earlier attempt with the same instructions using Opus https://www.anthropic.com/claude/opus . That implementation did not meet our bar and was discarded. After reviewing the implementation, we deployed the application to every environment except production, using an internal account to push results so we could get feedback quickly. The review also surfaced anti-patterns we wanted gone and improvements we wanted in: - Removing configuration generated at runtime. The legacy application derived its configuration from partial values provided via environment variables. That has been a pain point in the past when debugging and making configuration changes, especially under pressure during incidents. - Improving end-to-end observability. The legacy service had high-level observability, but given the increase in throughput, there was clear room to do better. We implemented both with human supervision, which gave us two more principles: - Human intervention is sometimes required, especially to find and fix anti-patterns an agent inherited from the legacy application or introduced itself. For us this meant removing all configuration derivations and making static environment variables the only way the application is configured, plus improving observability. That improved the system's operability and simplified the code in both the rewrite and the harness. - Take the rewrite as an opportunity to improve things. In our case that meant expanding end-to-end observability. At this message volume, recording low-level metrics such as database connection utilization, CPU, memory, and timings of individual code execution steps was a necessary addition for long-term operational excellence. With those in place, the next step was production. First deployment: what we got wrong first-deployment-what-we-got-wrong Before deploying, we had to decide how to migrate customers. The simplest approach was to spin up separate infrastructure queues and patch the consumers to push data to the new queue for accounts with a specific feature flag enabled. Once the infrastructure was ready, we deployed the Go application across all environments and migrated internal accounts. We ran into issues almost immediately, mainly retries failing. The root cause was a critical gap in the harness: our local environment's surrounding infrastructure did not match production. The harness modeled a simplified local queue topology instead of the real one. Locally, retries were routed to 3 queues based on check type alone. In production, routing also depends on factors like priority and hosting type, for a total of 18 possible queues per region. Because the harness assumed the local topology, the agent built its retry logic around it, and the gap made its way into the new application. We made that mistake during design. The initial boundary classification was too high-level and assumed production behavior that proved wrong. We updated the harness to align with production, removed all code branches that only ran in the local environment, and followed up with a human-supervised refactor of the retry module. A day later, the daemon was processing results for our internal accounts, around 3% of total platform load. The retry-queue gap left us with three principles for the harness going forward: - The harness environment must align with production as closely as possible. - Every boundary must be established down to its lowest unit. Every single queue, every DB table, accounted for. - Every assumption about the surrounding infrastructure must be written down and verified, not inferred. Rolling out to customers rolling-out-to-customers The migration strategy was deliberately boring. Before migrating any customer, the new daemon had already been through the harness and a full rollout across our own internal accounts on real production traffic. On top of that, we migrated customer accounts in stages to reduce blast radius, so each cohort benefited from what we learned on the previous one: - Free accounts, on a trial period or hobbyists. - Paid accounts, on a monthly subscription. - Enterprise accounts, with signed enterprise deals. Migrating a cohort meant flipping a feature flag, which instantly switched their traffic from the legacy daemon to the new one. After each flip, we monitored for 24 to 48 hours to confirm no results were dropped, failure rates across the platform stayed consistent, and no support escalations pointed back to the daemon before moving on. During this period, both daemons were processing real traffic, which meant both had to stay in sync. A change in one needed to land in the other. To enforce that without slowing anyone down, we updated CI to run the harness against both the legacy application and the new daemon, blocking any pull request that failed for either. When customers found or flagged bugs, we used a simple TDD loop to fix them: reproduce the gap in the harness first, then fix the underlying issue. That let us close an issue in about an hour, including CI and deployment time. The issues we hit were minor, and nearly all of them were edge cases the harness hadn't covered. Beyond those, none of our customers noticed they had been migrated. After a week of migrating customers, monitoring dashboards, and fixing small bugs, the migration was complete, and we decommissioned the legacy workload. Results results Better database performance better-database-performance - The tooling we chose to talk to the database, sqlc https://sqlc.dev/ , produced more performant queries. - Row locking dropped because Go's concurrency model executes operations inside transactions faster. - Together, that gave us 60% fewer total average active sessions AAS on our database and a roughly 15% reduction in database CPU . Improved efficiency and operability improved-efficiency-and-operability - The new daemon freed up around 15 vCPU and 45GB of memory, as the new application needed far fewer pods to support the workload. - The additional observability built into the new application helps with triaging during any alerts or incidents and enables us to make informed decisions when refactoring the code. Improved developer experience improved-developer-experience - Go's type system works much better with agents and adds another layer of protection against regressions. - The test harness guarantees consistent behavior before any rollout. That confidence shows up as faster feature and bug-fix shipping, more frequent deployments, and fewer alerts for our on-call engineers. With a harness that validates behavior at this level of rigor, we can now ship agent-written code faster than before , trusting the harness to catch regressions instead of relying on manual review alone. Summary summary Rewriting legacy applications is never easy, even with agents doing the heavy lifting. We got there by designing and building a strict test harness first, then letting the agent work inside it. Rigorous testing, strict boundary definition, and knowing when to step in as a human are what made this work, and those principles will hold up as the tooling keeps changing. If you want to see the testing philosophy behind this in a product rather than a blog post, it is the same one behind Checkly: Monitoring as Code https://www.checklyhq.com/product/monitoring-as-code/ , real Playwright scripts running against production, and results you can trust enough to act on. Start monitoring for free https://www.checklyhq.com/ , or read the docs https://www.checklyhq.com/docs/ to see how it fits your stack.