Apache Spark vs Apache Flink: which should you use?

Databricks' published benchmark found Apache Spark's Real-Time Mode matched Apache Flink on stateless transformations at 3 ms p99 latency and beat Flink on stateful aggregation (14 ms vs 45 ms p99) and stream-static joins (9 ms vs 95 ms p99), which Databricks summarized as up to 92% faster than Flink. The benchmark ran at 10,000 records per second for 30 minutes measuring Kafka input-to-output latency, with Spark on Databricks and Flink on AWS Managed Service for Apache Flink 1.18, and Databricks cautions the results compare Spark's worst run with Flink's best run. The comparison argues Spark's millisecond streaming largely removes the latency gap that historically pushed teams to add Flink as a second engine alongside Spark for batch, SQL and machine learning workloads.

How Spark Real-Time Mode changes the streaming decision, with latency benchmarks, a feature comparison and a guide to choosing an engine. For most teams, Apache Spark™ is the better choice. Spark runs batch ETL, SQL analytics, machine learning and streaming on one engine, and Real-Time Mode https://www.databricks.com/product/data-engineering/spark-real-time-mode now brings Spark streaming to millisecond, end-to-end latency. Apache Flink is a stream-native engine that fits specialized event-processing workloads, but choosing it usually means running a second engine next to Spark. The latency gap used to be the main reason teams added Flink. With Real-Time Mode, that reason has largely gone away, and the real question becomes how many engines you want to build, operate and govern. Spark is a unified engine for batch, streaming, SQL and AI, and Flink is a specialized stream processor. Both are open source Apache Software Foundation projects with exactly-once processing, stateful operators and SQL APIs. The difference is scope. Spark started as a batch engine and added Structured Streaming https://www.databricks.com/glossary/what-is-structured-streaming , which treats a stream as a table that keeps growing. The same DataFrame and SQL code runs as a scheduled batch job, a micro-batch stream or a Real-Time Mode stream. Flink was designed for streaming first and treats batch as a bounded stream. This table compares the two engines on the dimensions that usually decide the choice. | Dimension | Apache Spark | Apache Flink | |---|---|---| | Engine scope | Unified: batch, streaming, SQL, ML and AI data prep | Stream-first, with batch as a bounded stream | | Streaming latency | Seconds with micro-batch, milliseconds with Real-Time Mode | Milliseconds | | Batch and large-scale ETL | Industry default for petabyte-scale batch | Supported, less common in practice | | Languages | Python, SQL, Scala, Java, plus Go, Rust and Swift clients via Spark Connect | Java, SQL, Python PyFlink | | Stateful processing | Aggregation, deduplication, joins, transformWithState, RocksDB state store | Keyed state, timers, disaggregated state ForSt in Flink 2.x | | Complex event processing | Custom logic with transformWithState | Built-in CEP library and SQL MATCH RECOGNIZE | | Machine learning and AI | MLlib, feature pipelines, vector search in SQL, LLM data prep | Limited native ML, usually paired with another system | | Lakehouse integration | Native Delta Lake and Apache Iceberg read and write | Connectors for Iceberg and other formats | | Managed options | Databricks, AWS EMR and Glue, Google Cloud, Microsoft Fabric | AWS, Confluent, Ververica and others | Table 1. Apache Spark and Apache Flink compared. Sources: Apache Spark Real-Time Mode docs , Apache Flink 2.0 release and Flink CEP docs . In Databricks' published benchmark, Spark Real-Time Mode matched Flink on stateless work and had lower p99 latency on stateful aggregation and stream-static joins. The benchmark code https://github.com/databricks-solutions/latency-benchmarks is public, uses customer-derived feature-computation workloads and runs at 10,000 records per second for 30 minutes, measuring Kafka input-to-output latency. Databricks summarized the results as up to 92% faster than Flink https://www.databricks.com/blog/announcing-general-availability-real-time-mode-apache-spark-structured-streaming-databricks . | Workload | Spark Real-Time Mode p99 | Apache Flink p99 | |---|---|---| | Stateless transformation | 3 ms | 3 ms | | Stateful aggregation | 14 ms | 45 ms | | Stream-static join | 9 ms | 95 ms | How to read these numbers: Databricks ran the benchmark. Spark ran on Databricks and Flink ran on AWS Managed Service for Apache Flink 1.18. The results compare Spark's worst run with Flink's best run. Treat them as evidence for these workload patterns, and test your own pipeline before deciding. Production results tell the same story. MakeMyTrip https://community.databricks.com/t5/announcements/customer-story-makemytrip-achieved-millisecond-personalization/td-p/153882 considered Flink, chose Spark Real-Time Mode to avoid a second engine and cut P50 latency to 44 ms and P99 from more than a minute to about 500 ms. Coinbase and DraftKings also run Real-Time Mode in production. For how the engine achieves this, read Breaking the microbatch barrier https://www.databricks.com/blog/breaking-microbatch-barrier-architecture-apache-spark-real-time-mode . You change the trigger. Real-Time Mode is a trigger on an existing Structured Streaming query, so the same code moves between latency tiers without a rewrite. Real-Time Mode needs update output mode and reads from and writes to Kafka or a row-level foreach sink. Stateful operators arrive in open source with Apache Spark 4.3 and are generally available on Databricks today. On Databricks, it runs on classic compute https://docs.databricks.com/aws/en/structured-streaming/real-time/setup or in Lakeflow Spark Declarative Pipelines https://docs.databricks.com/aws/en/ldp/real-time . See the Real-Time Mode reference https://docs.databricks.com/aws/en/structured-streaming/real-time/reference for the full support matrix. Flink can be the better fit for a small set of specialized streaming workloads. Consider it when most of these are true: If the stream also feeds analytics, ML features or lakehouse tables, Spark avoids duplicating logic, operations and governance across two engines. Every second engine adds a second codebase, a second skill set and a second set of governance rules. Teams that pair Spark for batch with Flink for streaming often rewrite the same business logic twice and debug drift between them. Pipelines can also be declared instead of hand-wired. Spark Declarative Pipelines https://www.databricks.com/blog/bringing-declarative-pipelines-apache-spark-open-source-project manage dependencies and incremental refresh for streaming tables. Databricks was founded by the original creators of Apache Spark and runs Real-Time Mode as a generally available feature, including stateful operators and transformWithState https://www.databricks.com/blog/introducing-transformwithstate-apache-sparktm-structured-streaming . It adds Lakeflow Spark Declarative Pipelines, Unity Catalog governance and serverless pipelines. Code written for open source Spark runs on Databricks unchanged. To get started, follow the Apache Spark tutorial: Real-Time Mode https://www.databricks.com/spark/getting-started-with-apache-spark/real-time-mode . In Databricks' published benchmark, Spark Real-Time Mode matched Flink on stateless workloads and had lower p99 latency on stateful aggregation and stream-static joins. Results depend on the workload, so benchmark your own pipeline. Not only. Spark Structured Streaming defaults to micro-batch, and Real-Time Mode processes records continuously with millisecond latency. You choose per query by setting the trigger. For most workloads, no. Real-Time Mode gives Spark millisecond latency on the same engine you use for batch, SQL and ML, so you run one system instead of two. Flink can fit specialized workloads such as very large keyed state, complex event processing or teams that already run a Flink platform for standalone streams. Yes. Most Flink SQL and DataStream logic maps to Spark SQL and DataFrames in Structured Streaming, and Real-Time Mode covers the low-latency path. Start with stateless and aggregation jobs, then move stateful logic to transformWithState. Subscribe to our blog and get the latest posts delivered to your inbox.