Netflix dumps gRPC for SSE, eyes more AI

Netflix is overhauling search and discoverability using large language models, with generative AI workflows used in roughly 300 of its titles. The company has built its own foundation model for personalized recommendations and a real-time service map to troubleshoot engineering issues, while also abandoning gRPC in favor of server-sent events (SSE) for its service dependency map.

Netflix /tag/netflix/ Netflix – which is spending “billions of dollars every year” on technology according to co-CEO Ted Sarandos this week – is overhauling search and discoverability, using large language models LLMs . That’s one of the many efforts the company is making to improve operations, executives said on a July 16, Q2 call, in which they described tech as one of three pillars central to the firm’s success. These were “entertainment value,” “technology,” and “improved monetisation,” and technology also sits across them; e.g. on entertainment value, generative AI workflows “have been used in roughly 300 of our titles,” Sarandos told investors on the call. It was not immediately clear when Netflix would make natural language search – also mentioned by Sarandos on the call – available globally. Modelling the future? Netflix has built its own https://netflixtechblog.com/foundation-model-for-personalized-recommendation-1a0bd8e02d39?ref=thestack.technology foundation model for user recommendations. Its machine learning team in March 2025 described this as in “direct use as a predictive model to generate user and entity embeddings for other applications and can be fine-tuned for specific canvases…” That’s in line with a strong “build it” ethos at the entertainment firm. Netflix has publicly documented how it built its own graph database system for example, after dismissing products like Neo4j and Amazon Neptune as not scalable enough for the amount of data it handles. Its homegrown system now handles 10 million operations per second across 650 TB of graph datasets “with low latency and cost efficiency.” That system dubbed KVDAL and documented here https://netflixtechblog.medium.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-2-building-a-scalable-storage-layer-ff4a8dbd3d1f?ref=thestack.technology , is a two-level map architecture built on top of the open-source NoSQL Apache Cassandra database, and designed to emulate complex graph relationships. Cassandra, Kafka, Pekko... It’s also built a real-time service map designed to help engineers troubleshoot issues across its increasingly complex stack; building heavily on open-source tools like Akka fork Apache Pekko, eBPF, open-source databases like Cassandra, and streaming software Kafka. Netflix has added live streaming, gaming and built its own ads engine in recent years; all of these have posed real engineering challenges and over the past four years, a review of its support tickets showed that five key questions cropped up regularly, a May engineering blog https://netflixtechblog.com/from-silos-to-service-topology-why-netflix-built-a-real-time-service-map-0165ba13a7bc?ref=thestack.technology showed. These are “What are my upstream and downstream dependencies?”; “Is this failure in my service, or is something I depend on broken?”; “Which services will be impacted if I take this down for maintenance?”; “Why is this service showing as ‘Unknown’ in my metrics?”; “What changed in my call path recently that could explain this behavior?” Why Netflix dumped gRPC Writing this week, the team of Netflix engineers behind the real-time service map gave a warts-and-all overview of what had been working and what hadn’t, across a system split into three core components. Netflix has split its real-time service dependency map into three layers with different storage optimised for each, its engineers explained. - Network Layer: eBPF flow logs in graph database partition. - IPC Layer: Application metrics in an isolated graph database - Tracing Layer: Distributed traces in Parquet columnar storage. Apache Pekko streams meanwhile acts as the central processing engine that cleans and organises data before it goes into the databases. One of the team's notable recent decisions was to abandon the Google-created, open-source gRPC remote procedure call framework. Netflix had been using gRPC to expose this massive, multilayered service map and earlier admitted https://netflixtechblog.com/from-silos-to-service-topology-why-netflix-built-a-real-time-service-map-0165ba13a7bc?ref=thestack.technology faintly ominously that the “technical details of building this at Netflix scale — handling Kafka lag, managing memory and garbage collection, optimizing distributed processing, debugging reactive streams — deserve their own discussion.” Ultimately, its team said, gRPC became a performance bottleneck. “Serialization overhead, connection pool management, and memory pressure for streaming responses consumed more CPU than business logic. Message queues added infrastructure complexity without benefit for our use case,” its engineers wrote in an update on the system. They’ve now swapped gRPC for the Server-Sent Events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent events/Using server-sent events?ref=thestack.technology SSE protocol. “SSE proved ideal: lightweight HTTP-based protocol with minimal serialization, natural backpressure integration with reactive streams, and simpler connection model. The lesson: industry best practices like ‘use gRPC for service communication’ don’t apply universally,” Netflix’s engineers said this week. They added that the positive outcomes from the switch included CPU usage becoming far more evenly distributed across instances “no more hot nodes with 10x the load of others” and network bandwidth usage dropping “significantly due to better aggregation and lighter-weight protocol.”