{"slug": "what-a-room-of-engineers-asked-me-about-realtime-resilience", "title": "What a room of engineers asked me about realtime resilience", "summary": "Ably designed its realtime messaging platform so client SDKs automatically fall back to endpoints on a separate domain served by a different DNS provider when the primary CloudFront path fails, a resilience measure the company says dates to its early years and was documented after the 2016 Dyn DNS attack. Speaking at the AWS Cloud Resilience Roadshow in Dublin on 22 September 2026, an Ably engineer described the company's four dependability pillars — performance, integrity, reliability and availability — and said Ably runs disaster recovery drills on the same automation that handles real failures. The engineer said Ably accepts added latency when a client lands on a non-nearest datacentre, choosing availability over optimal latency every time.", "body_md": "I spoke about Ably's approach to resilience at the AWS Cloud Resilience Roadshow in Dublin on 22 September 2026, and the questions afterwards covered CDNs, regional failover, outages, message ordering, and capacity. They got me reflecting on which parts of our approach are common across the industry, which are less so, and how choices made early, well before my time at Ably, have put us in a strong position today.\n\n## Why Ably treats resilience as more than uptime\n\nAbly is the realtime layer inside other companies' products, powering experiences like AI agent streaming, chat, live sports data, and collaboration. Our customers build on us rather than around us, so if we are down, their product is down, which makes them entirely reliant on our service.\n\nThat premise shapes everything we do. We frame dependability around four pillars: performance, integrity, reliability, and availability, and these pillars are not siloed. Resilience is not only whether the service is up, it is whether every message arrived, in order, even through disruption.\n\nThe rest of the talk covered three points:\n\n- **Build for failure at every layer.** Our SDKs, our software, and our infrastructure are each designed to lose parts of themselves without the end user noticing. Retrofitting that later is possible, but it's genuinely hard.\n- **Verify rather than assume.** We run disaster recovery drills on the same automation that handles real failures, so the drill and the incident are the same code path. If a failure mode worries you, rehearse it.\n- **Resilience is relational.** The hardest problems are not lost instances or degraded regions. Those are the happy-path failure domains Ably is built for. Subtle degradations at the boundary between our software and AWS are more challenging and more interesting. A strong relationship with the AWS account team is what gets those solved, and it is far better built before the incident than during it.\n\nThe questions that followed went deeper into specific parts of that picture. Each section below reflects on one of them and looks at where our approach matches common practice and where it differs.\n\n## Plan for your CDN, DNS, and domain failing\n\nOne question was whether we had considered an alternative CDN provider, for example Cloudflare rather than CloudFront. The question was framed around performance, though I think resilience is the more interesting angle.\n\nA common pattern is to treat the CDN, DNS provider, and domain as fixed infrastructure, chosen once on performance and cost, and to rely on each provider's own redundancy. That is reasonable, though it leaves each of them as a single point of failure.\n\nOur inbound path has several layers. [__CloudFront__](https://ably.com/docs/platform/architecture/infrastructure-operations#inbound-connectivity) is the outermost one, providing DDoS protection and a nearby edge location for clients. Behind it, Network Load Balancers spread connections across frontend instances, and latency-based DNS sends each client to the nearest available region.\n\nEvery one of those layers is a dependency, so what happens when one of them fails? Our answer sits largely in our Ably client SDKs:\n\n- If a client cannot reach the primary endpoint, our SDKs automatically try [__fallback endpoints__](https://ably.com/docs/platform/architecture/edge-network) , which connect directly to specific datacenters and bypass the CloudFront routing path.\n- Those fallbacks live on a completely separate domain, served by a different DNS provider, so a DNS or domain-level failure cannot take out both. We [__wrote about this in 2016__](https://ably.com/blog/routing-around-single-point-of-failure-dns) , after the attack on Dyn's DNS took down many popular websites.\n\nOne trade-off is latency. A client on a fallback may land on a datacentre that is not the closest, which can add latency, but we choose availability over optimal latency every time. This layered resilience was designed in during Ably's early years rather than added after an incident, and coupling infrastructure redundancy with the fallback behaviours in our SDKs makes it very hard to take down.\n\n## Make regional failover a client and platform concern\n\nI was also asked how traffic moves between regions when something fails. Many multi-region setups rely on DNS alone to move traffic, and treat a reconnecting client as a brand new one. That works, though it is slow to react and leaves the application to rebuild whatever state it had.\n\nEach Ably region runs as a [__semi-autonomous site__](https://ably.com/docs/platform/architecture/fault-tolerance), capable of handling connections and processing messages on its own, while sharing data with the others. Most disruptions start in a single site and stay there.\n\nMoving traffic away from a failing region happens in a few ways:\n\n- **Automatically** , when a region is determined to be unhealthy, traffic is routed to the next closest healthy one.\n- **From the client** , when SDKs detect errors and try other regions through their fallback endpoints, without waiting for DNS to catch up.\n- **Deliberately** , when on-call engineers decide a problem will not resolve quickly and[__disable the site__](https://ably.com/docs/platform/architecture/infrastructure-operations#regional-disabling) . DNS and load balancer changes send new connections and reconnections elsewhere, and the site is reintroduced gradually once fixed.\n\nReconnecting is only half the job. A client that is disconnected for less than two minutes [__resumes its connection__](https://ably.com/docs/platform/architecture/connection-recovery) rather than starting again. It tells the server the last message it saw on each channel, and the server delivers everything it missed, in order, as if nothing had happened.\n\nNone of this works without spare capacity. The receiving regions have to absorb the traffic, which means keeping enough headroom during normal operation.\n\nPutting part of failover in the client, and making resumption a platform feature rather than an application problem, are the less typical choices here, and both are very hard to retrospectively add to a platform.\n\n## Make a global outage close to impossible\n\nSomeone asked when our last outage was (I should have mentioned this in my talk but somehow forgot), and it deserves a fuller answer than there was time for on the day.\n\nRegional disruption is almost guaranteed to happen at internet scale, whether in our own infrastructure or in a dependency like AWS. Ably's architecture is designed to absorb it, as earlier sections describe.\n\nFor many architectures, losing a region is effectively a global outage, because that region holds state or capacity nothing else can replace. For us, a global outage is a much higher bar. With regions operating independently, we would need severely degraded service in multiple AWS regions at the same time. Realistically, only two things could cause that:\n\n- a catastrophic deployment or configuration change that reached most regions before anyone noticed\n- those regions all becoming unviable at once\n\nThat is why so much of our process is about change. Deployments replace instances gradually rather than updating them in place, configuration changes are versioned so they can be rolled back, and changes move through development and staging before production. Ably has now gone more than seven years without a global outage.\n\n## Put message ordering in the platform, not the application\n\nAnother question was how we guarantee message ordering. Many messaging systems only guarantee order within a single partition or region, and leave anything wider to the application.\n\nWhen messages are published over a realtime connection, [__subscribers anywhere in the world receive them in the order they were published__](https://ably.com/docs/platform/architecture/message-ordering).\n\nOrdering gets more interesting when publishers are in different regions. Messages travel peer-to-peer between regions by the most direct path, so two messages published at almost the same moment in different regions can arrive in a different order in each region. We handle this with two orderings:\n\n- **Realtime order** , the order a given region observed messages, which every subscriber in that region sees consistently\n- **Canonical global order** , a single order every region agrees on, used for features like history\n\nEvery message carries an identifier that fixes its position in both. The same identifiers are what let a reconnecting client tell the server exactly where it left off. Building this into the platform means most customers never have to write their own sequencing logic.\n\n## Don't depend on a single instance type\n\nThe last question was whether we struggle to get particular instance types in some regions. Standardising on a single instance family is a common choice, as it simplifies capacity planning and performance tuning. We used to feel the downside of this, and the answer was to stop depending on a single instance type. Running Ably across multiple types means a shortage of one is no longer a practical problem.\n\n## Measure availability the way customers feel it\n\nNobody asked this one directly, though it sits underneath all of the others. Availability is rarely binary. [__Disruption usually shows up__](https://ably.com/docs/platform/architecture/infrastructure-operations#service-status-and-incident-communication) as higher error rates for some operations, slower responses in one region, or problems for specific accounts or features. A service can be \"up\" while some customers are having an issue.\n\nMany teams still monitor averages and report status as simply up or down, which hides exactly these problems. We alert on percentiles rather than averages, so problems affecting a subset of users do not disappear into aggregate numbers.\n\nThe core of this is our quality of service (QoS) system. It runs synthetic Ably clients inside every region of every production cluster. Several times a second, in each of those regions, they connect, attach to channels, and exchange messages, recording whether each operation succeeded and how long it took. If our QoS service were a paying customer, it would be one of our largest.\n\nThe results enable three things. They drive alerting and on-call paging, so a failure rate rising in one region of one cluster is picked up even when the platform as a whole looks healthy. They are also how we measure availability against our SLA, with any interval that crosses a failure threshold counted as degraded. Finally, they let us confirm service is restored before re-enabling a region or component. In other words, availability is measured from the client side of the connection, region by region, rather than inferred from server health.\n\nAlongside QoS, independent external monitoring checks latency and availability from outside, with no dependency on any Ably internals.\n\n## Resilience is decided before anything fails\n\nLooking back across the questions, the common thread is that the hard parts of resilience are decided long before anything fails:\n\n- **Dependencies need a plan B.** A CDN, a DNS provider, or a whole domain can fail, and clients need somewhere else to go when they do.\n- **Moving traffic is only half of failover.** Clients also need to pick up where they left off, with nothing missed and nothing out of order.\n- **A global outage should be close to impossible.** With independent regions, the realistic risk is a bad change reaching everywhere at once, so that is where the safeguards belong.\n- **Guarantees belong in the platform.** Ordering and resumption handled by the platform are logic every customer would otherwise have to write themselves.\n- **Capacity is a resilience decision.** Spare headroom and flexibility over instance types are what let healthy regions absorb a failing one.\n- **Availability is what customers experience.** Synthetic clients in every region, measured on percentiles, catch the partial failures that averages hide.\n\nNone of these are easy to add once a platform is running at scale. Many of ours were made in Ably's early years, and they are why regional failure is a routine operational event for us rather than a disaster scenario. That is the bar I would encourage any team running critical realtime infrastructure to aim for.\n\nWe are open about how all of this works, and the [__platform architecture section of our docs__](https://ably.com/docs/platform/architecture) goes into far more detail than I could in fifteen minutes.", "url": "https://wpnews.pro/news/what-a-room-of-engineers-asked-me-about-realtime-resilience", "canonical_source": "https://ably.com/blog/aws-engineers-realtime-resilience", "published_at": "2026-10-20 15:42:33+00:00", "updated_at": "2026-10-20 16:17:55.687720+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents"], "entities": ["Ably", "AWS", "CloudFront", "Cloudflare", "Dyn", "AWS Cloud Resilience Roadshow"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-a-room-of-engineers-asked-me-about-realtime-resilience", "markdown": "https://wpnews.pro/news/what-a-room-of-engineers-asked-me-about-realtime-resilience.md", "text": "https://wpnews.pro/news/what-a-room-of-engineers-asked-me-about-realtime-resilience.txt", "jsonld": "https://wpnews.pro/news/what-a-room-of-engineers-asked-me-about-realtime-resilience.jsonld"}}