{"slug": "a-fast-path-for-fixed-length-lists-in-parquet", "title": "A Fast Path for Fixed-Length Lists in Parquet", "summary": "Apache Parquet's Dremel encoding imposes a roughly 3× overhead for fixed-length lists like vector embeddings, but the upcoming Hardwood release introduces a fast path that detects data pages with uniform list lengths and bypasses standard record reconstruction, achieving speed-ups of up to 3.7× for 768-element lists and matching flat-column performance, according to the Parquet community.", "body_md": "In its current form Apache Parquet isn’t a great fit for storing fixed-length lists, such as coordinates, RGB(A) colors, or—an increasingly common case—vector embeddings driving search and retrieval workloads.\nA 768-dimensional embedding is just a list of floats that always has the same length,\nyet Parquet’s Dremel machinery encodes it as if that length could vary from row to row, spelling out and reconstructing each vector’s structure on read.\nThat costs roughly 3× more than a purely flat columnar representation of the same data (see [apache/arrow#34510](https://github.com/apache/arrow/issues/34510)).\n\nThe Parquet community has recognized this problem and is actively working on addressing it.\nSeveral options for adding fixed-length list support are [currently under discussion](https://lists.apache.org/thread/qhq33wg1loxhymsyqjnsfsrd82qnv43m),\nfor instance in the form of a new logical type `FIXED_SIZE_LIST`\n\n.\nEventually, this should bring decoding performance of fixed-length list data to the level of a fully flat schema.\nHowever, until that’s the case, is there anything we can do to narrow or even close the gap by retrieving effectively fixed-length lists in a smarter way?\n\nTurns out, yes we can.\n\nBy examining the *encoded* data streams for Parquet’s definition and repetition levels it is possible to detect data pages which only contain lists of the same lengths and put these onto a fast path which bypasses the regular Dremel record reconstruction.\nWe’ve implemented this optimization for the upcoming Hardwood release,\nwith striking results.\nThe exact impact depends on the chosen Hardwood reader (row vs. column) and list length.\nThe following chart shows the results from parsing ZSTD-compressed files with two list widths, each with 128M four-byte float values.\n\nThe short 3-element lists (say, 3D coordinates) see a modest speed-up of 1.1× for the row reader and a solid 2.5× for the column reader. For the larger 768-element lists (for instance, vector embeddings), the speed-up is 3.7× and 2.5×, respectively, getting on par with a flat column storing the same number of values. The row reader’s advantage grows with the list length: it has to materialize one list object per record, an overhead the fast path can’t remove. At short lists (many records), that cost dominates and the win is minor, whereas for long lists (few records) it essentially vanishes and the fast path brings the row reader down to the flat-column floor.\n\nIn the following, we’ll take a closer look at why Parquet in its current form isn’t optimal for storing fixed-length lists and how Hardwood’s fixed-length list fast path makes up for it.\n\n## Parquet’s Dremel Encoding\n\nFirst, let’s explore how exactly Parquet stores nested and repeated data.\nThe storage representation is described in the seminal [Dremel paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36632.pdf) from 2010.\nAn optional list, i.e. a list which itself can be `null`\n\n, of optional elements has the following schema:\n\n```\n1\n2\n3\n4\n5\noptional group mylist (LIST) {\n  repeated group list {\n    optional float element;\n  }\n}\n```\n\nThis three-level encoding is required to tell apart `null`\n\nand empty lists from `null`\n\nelements.\nParquet uses the notion of *definition levels* to encode which elements in the serialized data are `null`\n\n.\nIn the case of `mylist`\n\n, the maximum definition level is 3:\n\n-\nthe entire list is null (definition level = 0)\n\n-\nthe list is present but empty (definition level = 1)\n\n-\nthe list contains a null element (definition level = 2)\n\n-\nthe list contains an actual value (definition level = 3)\n\nThe notion of *repetition levels* is used to express at which level of the (potentially deeply nested) document tree elements are repeated.\nIn our example, the maximum repetition level is 1:\na value either continues the list of the current record (repetition level = 1),\nor a new record’s list starts (repetition level = 0).\n\nAs an example, let’s consider a Parquet file with four records which contain the following `mylist`\n\nvalues: `[1.0, null, 1.2]`\n\n, `null`\n\n, `[]`\n\n, `[4.0, 4.1]`\n\n.\nHere are the value stream as well as the definition- and repetition-level streams:\n\nThe value stream solely by itself would be ambiguous;\nwhen reading it back, you wouldn’t know to which record a given value belongs and where null values sit.\nThe definition-level stream encodes the `null`\n\nand empty values,\nwhile the repetition-level stream establishes the record boundaries.\nThis mechanism is great for encoding lists which actually are of varying lengths and which may contain `null`\n\nelements as well as may be `null`\n\nthemselves.\n\nNow let’s take a look at an example of required (i.e. non-optional) fixed-length lists of required elements (max def = 1):\n\nThe definition levels are now a constant stream of 1s;\nwhile this can be encoded very efficiently using run-length encoding,\nwe’d ideally not store anything for this at all, as the non-nullness already is part of the data schema.\nThe repetition levels are a recurring sequence of one 0 followed by `n - 1`\n\n1s, with `n`\n\nbeing the fixed length of our lists: `0,1,1,0,1,1,0,…`\n\n.\nThat’s about the most inefficient way you could think of for encoding a constant `n`\n\n,\nso it is understandable that the Parquet community is exploring alternatives for storing fixed-length lists more efficiently.\n\nNow, the disk footprint is not that much of an issue, as this data encodes tightly. The issue is the processing overhead: Definition- and repetition-level streams need to be decoded, arrays for them allocated, the reader needs to keep track when to start a new record with a new list, etc. All this adds up, leading to the 3× slow-down mentioned above.\n\n## Reading Effectively-Fixed-Length Lists Faster\n\nIn Hardwood, we have recently implemented [an optimization](https://github.com/hardwood-hq/hardwood/blob/main/core/src/main/java/dev/hardwood/internal/reader/FixedSizeListDetector.java) which avoids this overhead.\nBy scanning the encoded definition and repetition levels (def and rep levels from here on) we can detect that lists in a data set effectively are fixed-length, i.e. they all have the same size.\nIf that’s the case, we can bypass the regular Dremel record reconstruction machinery for the affected data pages, yielding a very nice speed-up.\nSo how does this work?\n\nFor def levels, it’s trivial.\nFor a given page, we’re looking for a stream of `max_def_level`\n\nwhose length matches the number of values in the page.\nBoth required lists (max def = 1) and optional lists (max def = 2) are handled.\nParquet stores def and rep levels using a [hybrid scheme](https://parquet.apache.org/docs/file-format/data-pages/encodings/#RLE) of bit-packing and run-length encoding (RLE).\nSay, we’re dealing with a page of 3D float coordinates, i.e. fixed-length lists with `n`\n\n= 3.\nWith 4-byte floats, a ~1 MB page holds roughly 87,000 records of 3 coordinates each—261,000 leaf entries in total (261,000 × 4 bytes ≈ 1 MB).\nFor a stream of all-present list elements, the encoder will serialize the def levels as one single RLE run,\nmade up of a varint header—the run length shifted left by one, with the freed low bit set to 0 to mark this as an RLE run rather than bit-packed—followed by a byte for the max def level:\n\nThis check for the absence of null values computes in O(1) and if it succeeds, the page is simply stamped with a \"no null values\" marker,\navoiding the decoding and processing of the def-level stream.\nIn fact, this optimization isn’t specific to fixed-length lists;\nit also speeds up the common case of a column whose schema permits null values but that happens to contain none.\nThat shortcut is a well-known one and is also implemented in other Parquet readers and engines, [including DuckDB](https://github.com/duckdb/duckdb/blob/2c2d62b247fdd87311b4f0a00bc7d3e209a0381b/extension/parquet/column_reader.cpp#L719-L723).\n\nDetecting the repeating 0,1,1,0,… pattern in the encoded rep-level stream is a bit more complex.\nHow it encodes depends on the value of `n`\n\n:\nParquet’s RLE/bit-packing hybrid bit-packs the 0 at the record start and up to seven subsequent 1s.\nIf the list length is longer—there are more 1s left—any remaining 1s are put into an RLE run.\nThis means we need to either look for a series of bit-packed runs (n ≤ 8) or for alternating bit-packed and RLE runs.\n\nIf the def-level gate has been passed,\nthe fixed-length list detector tries to detect small `n`\n\nrep levels first.\n`n`\n\nitself is the distance between the first zero and the next zero.\nBit-packing a periodic bit pattern yields a periodic *byte* pattern, with period `n / gcd(n, 8)`\n\nbytes:\na single byte when `n`\n\ndivides 8 (`n`\n\nin {1,2,4,8}), or 3/5/7 bytes otherwise.\nFor example, for `n`\n\n= 4 the stamp is one byte, `0xee`\n\n(`0 1 1 1 0 1 1 1`\n\n, read LSB-first),\ni.e. the lists of two records are covered by one byte:\n\nIf `n`\n\n= 3, the stamp is three bytes, `B6 6D DB`\n\n, spanning eight records (24 bits).\nBoth single-byte and multi-byte stamps are calculated once and then applied in batches,\nreading eight rep-level bytes at a time into one long.\nIf a given small-`n`\n\nrep-level stream doesn’t match the stamping pattern at some point, this means it contains lists of varying lengths and the fast path can’t be taken.\nThis technique (SWAR, SIMD within a register) proved fast enough during benchmarking1.\n\nNext, let’s discuss the case where `n`\n\n≥ 16;\nhere we need to look for the recurring sequence of a bit-packed run at the record boundary followed by an RLE run of `n`\n\n- 8 1s.\nIn the common case each record lands on a byte boundary, so every record is the *same* sequence of bytes.\n\nAs an example, for `n`\n\n= 16, we’d have four bytes per record:\n\nWe only need to parse the rep levels of the first record, so we can derive `n`\n\nand the stride (in bytes) and then can check whether the whole stream is that stride repeated. It is byte-periodic with period `strideBytes`\n\n, so the stream shifted left by one stride must equal itself.\nThat one bulk compare replaces per-value rep-level processing.\n\n|\nThe exact byte split shown, a bit-packed boundary group plus an RLE run, is what mainstream encoders emit, including parquet-java and Arrow C++. It is encoder-dependent, however. The Parquet specification itself only requires that the rep levels decode to the right sequence, not how the RLE/bit-packing hybrid chunks them into runs. Therefore, the detector doesn’t hardcode any specific pattern; instead, it derives the stride from the first record and verifies that the rest of the stream repeats it, deferring to the scalar fallback described below for any layout it doesn’t recognize. |\n\nThe third and final scenario is where 9 ≤ `n`\n\n≤ 15.\nIn this case, there’s no per-record stride to bulk-compare:\nFor instance, the rep levels of a 9-element list are nine bits; against eight-bit bytes its boundary drifts by one bit each record and only realigns every lcm(8, 9) = 72 values.\nA scalar fallback handles these cases, processing the rep levels run by run, ensuring equal distance between record boundaries.\nIt reads each run’s header and scans bit-packed groups in 64-bit words at a time, while skipping RLE runs of 1s in O(1) rather than expanding them.\nAs soon as two records disagree in length, the page is rejected from the fast path.\nBecause it assumes nothing about the layout, this path also catches the rarer irregular cases:\none-element lists, or writers that split runs unusually.\n\n## Performance Gains\n\nNow, what do we actually gain by applying this logic — what exactly is the \"fixed-length list fast path\" once we have determined that the lists in a given page are all of the same length? A number of optimizations can be applied in this case:\n\n-\nvalues are copied page-wise, rather than processed element by element\n\n-\noffsets are trivial—no rep-level scan is required to recover record boundaries\n\n-\ndef- and rep-level arrays are never materialized, reducing GC pressure\n\nThe effect is substantial.\nThe fast path is opt-in currently,\nso we can easily measure the parse times of fixed-length lists with and without the optimization.\nThe chart shows a sweep of `n`\n\nfrom 1 all the way up to 1536.\n\nFor the column reader the advantage hovers around 2.5× across most of the sweep.\nThe shaded band flags the scalar-fallback widths (`n`\n\n= 9–15), which aren’t byte-aligned and so make the detector’s rep-level scan fall back to a slower bit-by-bit walk—visible here as a slight dip in the column curve, and the detector’s most expensive case (which we quantify below).\nThe row reader has a more pronounced shape, climbing to a peak of around 3.9× in the mid-range (`n`\n\n≈ 64–256) before settling to 3.6× for large lists.\nThe rare pathological case of lists with a length of 1 is the one exception:\nwith a single element per record, list-object creation dominates and the fast path brings no advantage for the row reader—if anything, it’s a tiny bit slower2.\n\nOf course, running this detection logic over the encoded def-level and (in particular) rep-level streams has a cost of its own.\nSo does it penalize a regular variable-length `LIST`\n\ncolumn?\nThe computational cost for the detector heavily depends on the list size and the applied pattern logic:\nfor `n`\n\n≤ 8, where bit-packed runs of all rep levels need to be walked, it is up to ~1.7% of the regular list retrieval cost (latency).\nFor larger `n`\n\n, the highly efficient RLE runs bring detection down toward `O(records)`\n\ncomplexity, so the relative cost falls from ~0.2% (for `n`\n\n= 16) to under 0.1% (`n`\n\n= 768).\nAs expected, the most expensive case is the scalar fallback, with a relative detector overhead of ~17% for `n`\n\n= 15.\n\nHowever, these are the pure detection latencies [measured in isolation](https://github.com/hardwood-hq/hardwood-benchmarks/blob/main/results/2026-07-22-fixed-size-list/decode-benchmark.log) in a microbenchmark.\nWhen looking at actual end-to-end parsing times,\nthe fast-path detector cost is negligible for all values of `n`\n\n.\nEven in the absolute worst case, where only the very last list in a page has a different length than all the lists before—i.e. the detector’s O(1) def-level gate passes and it then scans the entire rep-level stream before finding the odd list at the very end and falling back to the regular decode path—there’s [no measurable impact](https://github.com/hardwood-hq/hardwood-benchmarks/blob/main/results/2026-07-22-fixed-size-list/fallback-benchmark.log) on the overall processing time for all list sizes but one.\nOnly for `n`\n\n= 15 (an uncommon width, and the most expensive scalar fallback case) is there an overhead of ~2%, just outside the 99.9% confidence intervals.\n\n|\nThe benchmarks in this post are based on |\n\n## Summary\n\nUntil the fixed-length list support is added to the Parquet format,\nit’s sub-optimal for storing data such as vector embeddings,\ntriggering a costly retrieval of list boundaries from the file’s repetition-level stream.\nHowever, by examining the *encoded* repetition levels and looking for the patterns clearly identifying actually fixed-length lists,\nHardwood’s Parquet parser significantly reduces the reading times for such lists,\nyielding improvements of up to 2.7× for the column reader and up to 3.9× for the row reader, depending on the specific list lengths.\n\nWhile the definition-level optimization discussed above is relatively common,\nI’m not aware of any mainstream Parquet parser that applies the encoded repetition-level inspection\n(if you know of any parser doing this, I’d love to learn about it in the comments below).\nThis optimization is part of the upcoming Hardwood 1.1.0.Beta1 release.\nIt currently must be explicitly enabled by setting the parser option `hardwood.fixed-list-fast-path`\n\nto `true`\n\n,\nbut we expect to enable it by default for the 1.1 release barring any unforeseen issues.\n\nSo if you’re storing embeddings, coordinates, or other fixed-length vectors in Parquet, give it a try and see for yourself how much of the list-decoding overhead falls away in your case. Note that the numbers reported above come from relatively large (~450-500 MB) files; on smaller inputs the wins tend to be not quite as large, yet still substantial. Once the Parquet format has a proper fixed-length list type, that optimization should become unnecessary for newly written files. But until then, and for all the list-encoded data already out there, a little inspection of the encoded levels goes a long way.", "url": "https://wpnews.pro/news/a-fast-path-for-fixed-length-lists-in-parquet", "canonical_source": "https://www.morling.dev/blog/fast-path-for-fixed-length-lists-in-parquet/", "published_at": "2026-07-27 09:26:03+00:00", "updated_at": "2026-07-27 09:53:52.762636+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Apache Parquet", "Hardwood", "Dremel"], "alternates": {"html": "https://wpnews.pro/news/a-fast-path-for-fixed-length-lists-in-parquet", "markdown": "https://wpnews.pro/news/a-fast-path-for-fixed-length-lists-in-parquet.md", "text": "https://wpnews.pro/news/a-fast-path-for-fixed-length-lists-in-parquet.txt", "jsonld": "https://wpnews.pro/news/a-fast-path-for-fixed-length-lists-in-parquet.jsonld"}}