Spotify Builds External Index to Enable Low Latency Point Queries on Its Data Lake

Spotify introduced Random Access Parquet (RAP), a storage architecture that adds an external indexing layer over Apache Parquet files to enable low-latency point queries directly against its data lake, allowing online services and AI applications to retrieve individual records without replicating datasets into operational databases. The company said RAP maps lookup keys to Parquet file locations, reducing the overhead of query planning and metadata traversal, and supports the same datasets for analytics, machine learning, and online serving. Spotify stores petabytes of online data in Bigtable and exabytes in its Google Cloud Storage-based data lake, making large-scale replication costly.

Spotify has introduced Random Access Parquet RAP https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries/?utm source=chatgpt.com , a storage architecture that enables low-latency point queries directly against data stored in its data lake, allowing online services and AI applications to retrieve individual records without replicating datasets into operational databases. RAP adds an external indexing layer over Apache Parquet files, enabling interactive lookups while continuing to use the same datasets for analytics, machine learning, and online serving. Spotify explained that modern data lakes have become the central repository for analytical and AI workloads, but retrieving individual records remains inefficient because distributed query engines such as Trino https://trino.io/ and BigQuery https://cloud.google.com/bigquery are optimized for analytical scans rather than key-based lookups. Although cloud object stores such as Google Cloud Storage https://cloud.google.com/storage now provide millisecond access latency, query planning, metadata traversal, and file discovery can add significant overhead for point queries. Spotify noted that it stores petabytes of online data in Bigtable while exabytes reside in its Google Cloud Storage-based data lake, making large-scale replication into serving databases increasingly costly. RAP addresses this challenge by introducing an external index that maps lookup keys, such as user IDs, directly to Parquet files and row locations. Instead of scanning thousands of files, a query resolves the key through the index before issuing a targeted ranged read against object storage. As new data is written into Apache Iceberg https://iceberg.apache.org/ tables, an index builder generates append-only index fragments without modifying immutable Parquet files. Spotify said the approach allows the same datasets to support analytical processing, machine learning pipelines, notebooks, AI agents, and latency-sensitive online applications without maintaining duplicate storage systems. Spotify's announcement follows broader efforts to extend open data lake technologies beyond analytical processing. Google Cloud recently described an Apache Iceberg based lakehouse architecture https://cloud.google.com/blog/products/data-analytics/the-future-of-data-lakehouse-for-the-agentic-era?utm source=chatgpt.com for AI applications that similarly seeks to reduce data duplication while enabling operational access to data. Unlike that approach, RAP introduces a dedicated external indexing layer optimized for point lookups while remaining compatible with existing Parquet files and Iceberg tables. The architecture also generated discussion within the data engineering community. Andrew Lamb https://publish.x.com/?url=https://twitter.com/andrewlamb1111/status/2083460112555888799 highlighted RAP as an example of extending open data formats for interactive workloads. In a separate LinkedIn discussion https://www.linkedin.com/posts/vikas-singh-bril everyones-data-lake-is-already-fast-enough-activity-7490989209868980224-R5gJ , Vikas Singh argued that improvements in cloud object storage performance have shifted more of the latency associated with point queries toward query planning and metadata access, an area that RAP is designed to reduce through precomputed indexes. Spotify also described several storage layout optimizations that reduce point query latency. These include sorting data by lookup key to reduce the number of files accessed, grouping related records together, interleaving value columns so multiple attributes can be retrieved through a single contiguous read, and using covering indexes that can satisfy some queries without reading Parquet files. According to Spotify, these techniques trade modest increases in file or index size for fewer storage operations, allowing some point queries to be served through a single ranged read of only a few kilobytes. Interleaved value column layout enables related values from multiple columns Source: Spotify Blog Post Spotify also supports secondary indexes, enabling efficient querying across multiple lookup dimensions, such as buyer ID or seller ID, without rewriting Parquet files https://parquet.apache.org/ . Hash-based indexes support exact lookups, while sorted indexes enable range queries. Spotify said secondary indexes are managed at the serving layer, allowing new access paths without changing data pipelines while continuing to use the same Parquet datasets for both analytical scans and interactive point lookups. Storage layout techniques such as Z ordering https://en.wikipedia.org/wiki/Z-order curve and Hilbert curves https://en.wikipedia.org/wiki/Hilbert curve can further improve data locality for secondary lookup dimensions.