{"slug": "apache-data-lakehouse-weekly-july-1-to-july-8-2026", "title": "Apache Data Lakehouse Weekly: July 1 to July 8, 2026", "summary": "The Apache data lakehouse community focused on governance this week, with Parquet voting on versioned releases and Polaris aligning with the incubating Ossie semantics project. Iceberg Rust released RC3 after verification issues, while contributors debated how to handle table statistics across multi-engine deployments.", "body_md": "The lakehouse community spent this week deciding how change itself should work. Apache Parquet opened a formal vote to adopt versioned releases for breaking changes, borrowing a governance model that Iceberg refined over years. Apache Polaris canceled a vote on its semantic model API so it can align with Apache Ossie, the freshly incubating semantics project that opened its dev list this week. And across Iceberg, Arrow, and Parquet, contributors debated who owns statistics, which format features deserve to survive, and how far the specs should bend to serve AI and machine learning workloads. The connective tissue this week is governance. These communities are building the rules for evolving open formats without breaking the millions of tables that already depend on them. That work is invisible when it goes well, which is exactly why it deserves a close read while it happens. Releases also kept pace with the design debates: Polaris shipped 1.6.0, Arrow Rust shipped 59.1.0, and both Iceberg Rust 0.10.0 and Arrow 25.0.0 entered their final voting rounds.\n\nThe Rust implementation dominated release activity this week, and the path was not smooth. For readers who mostly touch Iceberg through Spark or a query engine, iceberg-rust deserves a moment of framing. The Java implementation remains the reference, but the Rust library has become the foundation for a second generation of tooling: pyiceberg-core binds it into Python, DataFusion integrates it for query processing, and a wave of lightweight services use it to read and write tables without a JVM. When this library ships a bug, the blast radius crosses language ecosystems, which explains the ceremony you are about to read. Danny Jones and Shawn Chang [opened the vote for Iceberg Rust 0.10.0 RC2](https://lists.apache.org/thread/m35vm2j7z9pdv2o3jtyvd5vls0bko0xj) on July 1, drawing verification from L. C. Hsieh, Matt Butrovich, Neelesh Salian, Renjie Liu, and Xin Huang across a twelve message thread. Issues surfaced during verification, and by July 8 Jones was back with [a vote on RC3](https://lists.apache.org/thread/h8ohxh44g1zc32clg3sgxr80xjc4nnr9). The willingness to cut a third candidate rather than wave through a flawed second one says something about how seriously the Rust community takes its release checklist, which covers everything from ASF license headers to a clean build of the pyiceberg-core bindings. The Rust library now sits underneath a growing stack of Python and query engine integrations, so the caution pays forward.\n\nThe week's most consequential design debate concerned table statistics, a topic that sounds dry until two engines start fighting over the same files. Some background helps here. Query engines rely on statistics, things like distinct value counts and data distributions, to choose good execution plans. Join two tables in the wrong order and a query that should take seconds takes minutes, so statistics quality translates directly into compute cost. Iceberg lets engines write statistics files and attach them to a snapshot in table metadata. That design works cleanly when one engine owns a table. The trouble starts in the multi-engine deployments Iceberg was built for, where Spark handles ingestion, Trino or Dremio serves interactive queries, and Impala or Flink sits somewhere in the mix, each with its own idea of which statistics it wants. Dzeri96 raised concerns about [how Iceberg handles existing statistics files](https://lists.apache.org/thread/hb4xz96257gkqwxc4khyjzsfm2d8v9h0), and Gábor Kaszab pushed back on several proposed remedies across a nine message exchange. The core problem: when engine A writes statistics for a snapshot and engine B later computes its own, B can overwrite A's work. One proposal would create a new snapshot whenever statistics get computed. Kaszab argued this breaks the mental model, since snapshots today mark data changes, not metadata additions, and a snapshot X holding stats for snapshot Y confuses more than it clarifies. Another proposal would bind multiple statistics files to one snapshot, keyed by engine. Kaszab questioned how readers match files to engines in practice. Does Impala version X.Y know whether it can read stat files written by Spark version A.B? Would engine IDs live in the Iceberg spec or in tribal knowledge? The thread closed the week without resolution, which is fine. Multi-engine interoperability is Iceberg's whole reason for existing, and the statistics gap is a real hole in that story that deserves a careful fix rather than a fast one.\n\nSecurity and access control produced a second substantial discussion. A quick primer for readers newer to the REST catalog: when a client asks the catalog for a table, the catalog can vend temporary storage credentials scoped to that table, so the client reads data files directly from object storage without holding permanent cloud keys. This delegation model is one of the REST catalog's best ideas, since it centralizes access decisions in the catalog while keeping the data path fast. William Hyun [proposed extending it with file-level access delegation](https://lists.apache.org/thread/odof6m2npvktwd51cz8qnrxjv95ws4wm), with Kevin Liu joining the exchange. Today, delegated access in the REST catalog operates at table scope, and that granularity is the limitation Hyun wants to fix. If a consumer should only see a subset of partitions, administrators either over-provision access or fragment tables to match access boundaries, and both options create operational pain. Hyun's proposal uses pre-signed URLs during scan planning to make partition-scoped sharing practical without restructuring tables. Concretely, a catalog planning a scan for a restricted consumer returns signed links only for the files that consumer is entitled to see, and the storage layer enforces the boundary because unsigned paths simply fail. The idea builds on an earlier delegation thread and moves Iceberg closer to the fine-grained sharing models that commercial platforms offer on top of the format. That last point carries strategic weight. Fine-grained access control is one of the few remaining capabilities where proprietary lakehouse platforms hold a clear edge over the open spec, and standardizing it in the REST catalog narrows that gap for every open implementation at once. Nevin Zheng added a related thread on [updating the Read Restrictions proposal](https://lists.apache.org/thread/bvb36z39p04322vtr1j1bgk8gvllf6h7) to adopt the Expressions Spec and IDReference, which points toward a more unified security model across these proposals.\n\nThe primary key conversation resurfaced with new energy. Anyone coming from the database world finds this gap surprising, so it deserves a plain statement: Iceberg tables have no primary keys. The format grew up serving analytical scans over immutable files, where keys matter less, and it handles row-level changes through delete files rather than key-addressed updates. That works, but change data capture pipelines, which replicate every insert, update, and delete from an operational database into the lakehouse, spend enormous effort reconstructing key semantics the format never promises. Chandra Sekhar K continued the [discussion on first-class primary key tables](https://lists.apache.org/thread/z6rtyds23g81hwgpxjgcjl4yv5zhzdpl), describing production experiments with primary-key-oriented semantics for CDC and mutable-data workloads. His team treats the primary key as table metadata and then builds behavior on top of it: key-aware write semantics, storage organization, compaction strategy, and changelog generation for incremental processing. He framed this as complementary to the existing constraints proposal, which limits PRIMARY KEY and UNIQUE to informational metadata and leaves enforcement out of scope. The distinction matters because it sketches a two-layer future. The spec standardizes how keys are declared, and engines opt into richer key-aware behavior above that line. Anyone watching the upsert and CDC space, where formats like Apache Hudi and Paimon built key handling in from the start, should track this thread closely. A related note from [st...@steveis.com](mailto:st...@steveis.com) asked about [row-delta commits and multi-table transactions in iceberg-rust from a CDC producer's perspective](https://lists.apache.org/thread/7d4mzx3ft3b0x1kb5g1jk91wkxnm0qf0), showing the same pressure arriving through the Rust door.\n\nGeospatial work is quietly becoming one of Iceberg's most active frontiers. Sunmin Lee proposed [declaring row-level bounding box covering columns in Iceberg metadata](https://lists.apache.org/thread/6qfom12527mwkgohrlf8wmhn1x31rqcy), which lets readers prune data files using spatial bounds. Seyed Muhammad Mahdi Hoseini followed with a separate proposal for [QuadTree-inspired physical spatial partitioning](https://lists.apache.org/thread/k09hbrtmzvd9tdjcrv5yb9d0wmx7l6nt), with Tanmay Rauth engaging on the bbox thread. Together these ideas sketch a spatial stack for the format: partition data by spatial cells on write, prune by bounding boxes on read. The economics mirror what min-max statistics did for numeric columns years ago. A query asking for events inside a city polygon should never open files whose contents sit on another continent, and today that pruning depends on engine-specific tricks rather than portable metadata. Iceberg v3 introduced geometry and geography types, so the type system groundwork exists, and these proposals show the community moving from type support to the performance engineering that makes spatial workloads economical. Fleet telemetry, logistics, climate data, and location-based applications all stand to benefit, and they represent exactly the data volumes where file pruning changes the bill.\n\nThe variant type effort keeps a steady drumbeat. Variant is the shredded, binary-encoded type for semi-structured data that lets JSON-shaped payloads live in tables with columnar performance, and it spans both the Iceberg and Parquet specs, which makes coordination the hard part. A field promoted to a shredded column by one writer has to remain readable by every engine, and the type touches file format, table format, and engine layers at once. That is why the community runs a dedicated sync for it. Neelesh Salian [posted notes and a recording from the July 2 Variant Sync](https://lists.apache.org/thread/z9wfmvjfc7c3x5j4rqjvp4hmy1qjxy8l), the recurring gathering that coordinates work on semi-structured data support across implementations, with action items now tracked in a shared document. The cadence of these syncs since April shows the feature moving through the unglamorous middle stage of standardization, past the exciting design documents and into the grind of cross-implementation agreement. Anurag Mantripragada opened a related design question about [column update file representation](https://lists.apache.org/thread/bj0sgk79yx0yt0wnr8jww2dsjsjc5tqj), which touches how partial column updates get written to storage.\n\nSpec hygiene rounded out the Iceberg week. Daniel Weeks opened threads on [clarifying schema JSON type string serialization](https://lists.apache.org/thread/bw8pvv1mghgfl96ny0sbqkx3ct33lco6) and [clarifying valid source types for the identity transform](https://lists.apache.org/thread/dpcg8qbt57t0b09bdm7xdwstvstk6zyb). Sung Yun raised a [write-path gap for field-id-bound policy during schema evolution](https://lists.apache.org/thread/8t1mp6o390jm1n4t48nm8rw0v627q6lc) and a question about [field id handling in the REST spec](https://lists.apache.org/thread/3mdjz2ogqdmgzzpsqbxl2gpnh6rk3ng9). These threads rarely make headlines, but they are the reason independent implementations of Iceberg in Java, Rust, Python, Go, and C++ produce the same answers on the same tables. Every ambiguity closed on the list is a bug that never ships, and the current volume of clarification work reflects the v3 and v4 feature waves pushing the spec into corners nobody had to define precisely before. Spec text that a single reference implementation can leave fuzzy becomes load-bearing the moment a second implementation reads it differently.\n\nA few more items deserve a sentence each. Szehon Ho called a [vote to add specific-name to the UDF spec](https://lists.apache.org/thread/br8bfgbd68z3tmmzrpkm5fqm4zqsvd4b), advancing the effort to make user-defined functions portable across engines. Alexander Löser continued the [collation support discussion](https://lists.apache.org/thread/n7gmlbq1yoz9v43ob4zjn78vhf5rc5rl), which matters for anyone whose sort order needs to survive engine boundaries. Daniel Weeks flagged an [upcoming Iceberg Terraform provider release](https://lists.apache.org/thread/mhyx8g420qtq3rc57m84xgcnqfl0dp15) with Alex Stephen, bringing infrastructure-as-code workflows to table management. Tanmay Rauth proposed a [table_properties_log metadata table](https://lists.apache.org/thread/zw4t6tpf64270vtf1d4n7hggtpjfctn8) to expose property history for audit and debugging. And Renjie Liu suggested [trimming CI runner time by running JDK 21 checks only on main and nightly builds](https://lists.apache.org/thread/jy1qlv7qk1gw078c9cn1wzqbxtmovxxx), a small change that respects contributor time on every pull request.\n\nThe Polaris story of the week is a vote that did not happen, and the cancellation matters more than most votes that pass. First, the context. A semantic model defines business concepts, metrics like revenue or monthly active users, dimensions like region or customer segment, and the logic connecting them to physical tables. Today those definitions live scattered across BI tools, dbt projects, and application code, and every copy drifts from the others. The industry calls the result semantic drift, and it is why two dashboards in the same company report different numbers for the same metric. The Open Semantic Interchange effort, backed by a broad vendor coalition, produced a specification for expressing these definitions in a portable, machine-readable format. Catalogs are the natural home for such definitions, since they already serve as the shared source of truth for tables, and that is where Polaris comes in. Yufei Gu had called a [vote to accept the OSI Semantic Model API Specification](https://lists.apache.org/thread/hd5hp3591to4smz6ofpmlfp4249v02jo), the proposal that would let Polaris store and serve business metric definitions alongside its catalog duties. Jean-Baptiste Onofré asked the community to pause. His reasoning: the Open Semantic Interchange specification is transitioning into the Apache Ossie project, which entered incubation on June 22, and an initial Ossie spec release is expected soon. Implementing the OSI spec now and the Ossie spec later means doing the work twice. Robert Stupp and Adam Christian weighed in across the thirteen message thread, and Gu canceled the vote after gathering the feedback. He noted the practical middle path along the way: most of the semantic model work in Polaris can proceed in parallel, since the main dependency on Ossie is a JSON validator that arrives later, and the API can ship marked as beta with an explicit plan to converge on the Ossie specification.\n\nGu then did exactly that kind of parallel work, opening a focused design thread on the [semantic model REST API payload representation](https://lists.apache.org/thread/5nm0440bs0n9slvmhxhzg1mbnnvm5o7h). The question sounds technical but shapes years of API stability: should Polaris represent a semantic model as a raw string, as an opaque JSON document, or as a fully modeled structure in the REST specification? Gu argued for the first or second option. His concern with full modeling is coupling. The Ossie schema is versioned and expected to evolve, and baking its structure into the Polaris REST spec means every Ossie schema change ripples through the REST specification, generated clients, and client applications. Keeping the payload opaque separates the long-term REST contract from the fast-moving semantics spec. This is the kind of API design judgment that determines whether integrations built in 2026 still work in 2029.\n\nMeanwhile, the release train kept rolling. Jean-Baptiste Onofré [prepared and opened the vote for Apache Polaris 1.6.0 rc1](https://lists.apache.org/thread/g2rg50df4y2sff0rksv0yqtj9mo8k44c), which drew verification from Dmitri Bourlatchkov, Yong Zheng, Francois Papon, Robert Stupp, and Ajantha Bhat. The [vote passed on July 8](https://lists.apache.org/thread/yf55j176f9t2o400rrw04l23svt9k570), keeping Polaris on the steady release cadence it has maintained through the year. For a project that graduated to top-level status only in 2025, the rhythm of regular, verified releases is itself a signal of maturity. Release votes at Apache are more than ceremony. Each binding vote represents a committer who downloaded the artifacts, checked signatures and licenses, and built from source, which is why the roll call of verifiers in these threads reads like a project health report. Six people independently verifying a release candidate within days tells you a community has depth beyond its most visible names.\n\nPersistence flexibility got real engineering attention. Yufei Gu shared a [proof of concept for a Polaris-managed JDBC datasource](https://lists.apache.org/thread/7f36lxd9dz7rv6p22p42m9zy9ck4152r), which lets Polaris create its own Hikari connection pool from configuration instead of depending on the Quarkus-managed datasource. Three details make this more interesting than typical plumbing. Different configurations can create independent datasources, which lays groundwork for future per-realm datasource routing, a meaningful capability for multi-tenant deployments. JDBC drivers can load at runtime from a jar rather than living on the build-time classpath, which solves a genuine licensing problem for drivers like MySQL that carry Apache-incompatible licenses. And the change is opt-in, since Polaris keeps the existing Quarkus path when no JDBC URL is configured. Dmitri Bourlatchkov, Robert Stupp, and Onofré worked through the design across seven messages.\n\nConfiguration cleanup and operational clarity threaded through the rest of the week. Gu proposed [deprecating ALLOW_EXTERNAL_TABLE_LOCATION](https://lists.apache.org/thread/t2pn7t2h983qjpkzq7n08cxxv941n8pm) after code review showed it acts mostly as a metadata-location escape hatch that overlaps with ALLOW_EXTERNAL_METADATA_FILE_LOCATION while inviting confusion with ALLOW_UNSTRUCTURED_TABLE_LOCATION. His proposal keeps the old flag as a backward-compatible alias and clarifies the documentation, including a doc mismatch he found where one setting claims to require a flag the implementation never checks. Location flags in a catalog are security boundaries, so naming confusion here is not cosmetic. In the same spirit, Alexandre Dutra proposed [deprecating TreeMapMetaStore and related classes for removal](https://lists.apache.org/thread/mnf4gk1lf8o7c9j97by2lofr6rs3ghft), trimming legacy persistence code, and Dmitri Bourlatchkov opened a discussion on [standardizing vended credential property names](https://lists.apache.org/thread/kox8co2o30sl1wvvf2lzp64lz8lt3ogy), which affects how engines consume the temporary storage credentials Polaris hands out.\n\nTwo threads pointed at the operational future. Bourlatchkov floated a [Kafka events publisher](https://lists.apache.org/thread/ohboxb4c9wrrjqjly4rkvpmlz3y77x12), building on the earlier discussion Gu, Bourlatchkov, and Dutra held about [REST endpoints for table metrics and events](https://lists.apache.org/thread/3nyxf15cqm8q7v9bsn5wgk2psvc0hptm). Catalogs sit at the choke point of lakehouse activity, so streaming catalog events into Kafka opens the door to audit pipelines, cache invalidation, and reactive workflows. And the Terraform wave that hit Iceberg reached Polaris too, with Alex Stephen, Gu, Nándor Kollár, and Sung Yun discussing a [Polaris Terraform provider](https://lists.apache.org/thread/xdp4tnyrdf1pcrnjj0fs0l9b58b989tr) across six messages. The same contributor raising Terraform providers on two project lists in one week is worth pausing on. It signals that lakehouse components have crossed a maturity threshold in the eyes of platform engineering teams, who now expect to manage catalogs, principals, grants, and table definitions the way they manage VPCs and Kubernetes clusters: declared in code, reviewed in pull requests, applied by CI. When the infrastructure-as-code ecosystem starts building first-class providers for your project, it means production adoption arrived ahead of the tooling, and the tooling is catching up.\n\nCommunity texture rounded things out. Rich Bowen interviewed project members for the ASF's [PlusOne series](https://lists.apache.org/thread/5ljoockhvw1x8bqtyjz4nsvyzxwl6doj), Prithvi S pushed forward on [clearer 403 authorization messages](https://lists.apache.org/thread/39693slxbh0skolj3lnwqt42z56qyjzz), Ayush Saxena asked about [staged creates in multi-table transactions](https://lists.apache.org/thread/2x25mj2pv981ocj83l8wfpr8yqvpplmg), and Gu proposed [entity-level filtering for list operations](https://lists.apache.org/thread/c4cjzx5x4stlssv1c0pynd40t939ffnv), which matters once catalogs hold thousands of entities and clients need server-side filtering rather than full listings.\n\nArrow entered release season on two fronts at once. Raúl Cumplido [opened the vote for Apache Arrow 25.0.0 RC1](https://lists.apache.org/thread/dv7obxo3rn6yv94jo6t0vgc02zg9fcy8) on July 6, drawing verification reports from Bryce Mecum, Sutou Kouhei, Gang Wu, Adam Reeve, and L. C. Hsieh. On the Rust side, Andrew Lamb ran the [vote for Arrow Rust 59.1.0 RC1](https://lists.apache.org/thread/2vpxdt6n7kzo72sxpr7q8yyby4495gnk) and [announced its passage on July 7](https://lists.apache.org/thread/46rmn75b2hq88plsvsb7mpl5g3ot69sx) with binding approvals from Hsieh, Mecum, and Kevin Liu among others. The arrow-rs crate underpins DataFusion and a wide slice of the Rust data ecosystem, so its steady minor-release cadence quietly services a lot of downstream projects. The two release lines also illustrate how differently the same project can move in different languages. The C++ line ships large coordinated releases spanning a dozen language bindings, while the Rust line ships small and often, and both models work because each matches its ecosystem's expectations. Downstream users planning upgrades should note that Arrow major releases occasionally carry ABI-relevant changes for the C data interface consumers, so the 25.0.0 release notes deserve a read before rollout.\n\nThe format-level discussion of the week asked a question every mature project eventually faces: which features deserve to keep living? Arrow's IPC format is the wire and file protocol that moves Arrow data between processes, and it was designed primarily around RecordBatch messages, the tabular unit that nearly every Arrow user touches. Alongside those, the format carries two specialized message types for N-dimensional arrays. Antoine Pitrou proposed [deprecating the Tensor and SparseTensor IPC messages](https://lists.apache.org/thread/z2kl4g1353mbtgpt3qo7j3bzohq0q7fv). His case is thorough. The Tensor message arrived in 2017 and SparseTensor in 2019, both remain marked experimental, neither appears in the cross-implementation integration test suite, neither is reachable from Flight RPC or the C++ Dataset API, and a GitHub code search turned up no third-party usage. Meanwhile the messages carry maintenance cost, including recent security reports about missing validation on untrusted input. Arrow now has canonical extension types that carry dense tensors as ordinary RecordBatch columns, with room to add sparse variants if demand appears. Rok Mihevc and Weston Pace joined the discussion. Deprecating unused format surface is unglamorous work, but every retired feature shrinks the attack surface and the implementation burden for the ecosystem.\n\nKent Wu opened a discussion that fills a long-standing usability gap: [a canonical JSON representation of Arrow schemas](https://lists.apache.org/thread/rbqmskk3omdfdhmsmtgz7v0vhmppgyw1). Arrow schemas today only serialize canonically as IPC binary, which is awkward wherever humans or JSON APIs get involved. Try describing an expected schema in an API contract, or hand-writing one in a test fixture, or diffing two schemas in a code review, and the binary-only reality starts to chafe. The immediate motivation comes from ADBC, Arrow's database connectivity standard, where the 1.2 milestone includes new metadata APIs and contributors found returning schemas as IPC blobs unsatisfying. Wu drafted a proposal document and invited comments, and Dewey Dunnington engaged from the ADBC side. Wu also noted the topic has surfaced repeatedly over the years, which is usually the tell that a gap is real rather than hypothetical. A readable schema format helps API contracts, hand-authored fixtures, and configuration files, and it lowers the barrier for tools that want to speak Arrow without linking an Arrow library. Anyone who remembers how much JSON Schema did for the JSON ecosystem understands the shape of the win.\n\nThe rest of the Arrow week was community maintenance in the best sense. Robert Kruszewski asked for a [spec clarification on field names](https://lists.apache.org/thread/v2fvqjtfrvsrbm133yc7yjrxvjh0x3l5), drawing responses from Felipe Oliveira Carvalho and Weston Pace. Pitrou and Mihevc discussed the [status of Arrow conbench data and the conbench OSS project](https://lists.apache.org/thread/rzyyqhq3oh6076372q9h4ffsnz9g2xmy), the continuous benchmarking infrastructure that guards against performance regressions. Nic Crane asked for [extra hands closing out old issues](https://lists.apache.org/thread/5xpt00zxlx57bwdm0v02rn7m1yj2olr9), and Pitrou collected contributions for the quarterly board report. Ian Cook hosted the July 1 community call.\n\nParquet produced both the busiest thread and the most consequential vote of the week, and the two stories reinforce each other.\n\nThe busiest thread first. The [discussion on introducing a FIXED_SIZE_LIST logical type](https://lists.apache.org/thread/95nkoj58y6k5y1jm2jhcd42kkmjgfpfs) ran to twenty four messages, making it the most active conversation across all five projects this week. The motivation is the AI workload wave. Machine learning embeddings, feature vectors, and scientific arrays are fixed-length by nature. An embedding column holds, say, exactly 1,536 floats in every single row, forever. Yet Parquet today stores those values as variable-length lists using its Dremel-derived repetition and definition levels, the bookkeeping that records where each list starts and ends and which values are null. That bookkeeping is essential for genuinely variable data and pure waste for arrays whose length never changes. Readers pay to decode structure that carries zero information, and vector search and feature store workloads read a lot of these columns. The thread weighed three design options, and Andrew McCormick made a detailed case for what participants call Option C. Option A breaks backward compatibility and loses encoding quality. Option B is the design a greenfield format would choose but demands a rewrite-scale effort in Parquet's layered reader and writer stack. Option C stays fully backward compatible, keeps the existing encodings, and needs only modest reader and writer changes. Old readers see a normal list, new readers see the fixed-size annotation and take a fast path. The remaining cost is that repetition levels still hit the disk, but under run-length encoding a constant-length list compresses to almost nothing. McCormick brought receipts: on a benchmark of one hundred thousand rows of four-thousand-element float arrays, the fast reading path decompressed 2.3 times faster than the baseline, with roughly another 1.5x available when the writer records the fixed-size hint so readers skip verification. Russell Spitzer, Adam Reeve, Antoine Pitrou, Alkis Evlogimenos, and Philipp Fischbeck all engaged. Watch this one. Vector-heavy workloads are becoming a defining Parquet use case, and this type closes a real efficiency gap against newer formats designed with ML in mind.\n\nNow the vote. Ryan Blue moved the long-running versioning debate to a decision, opening a [vote to use version numbers for releasing forward-incompatible changes](https://lists.apache.org/thread/o8bwmqdy1wf0z459h65x43lnsrldrvhq). The mechanics are straightforward. Forward-incompatible changes accumulate against the next major version of the Parquet spec, new breaking changes automatically target the version after the current one, and the community votes to close and adopt each major version as a unit. Blue noted the lineage with a smile: Iceberg contributors call this the Iceberg model, though Iceberg originally inherited the idea from Parquet. His [summary in the discussion thread](https://lists.apache.org/thread/g285pbt7losnhp9p9r03pj01sxogysgg) captured the state of debate honestly, acknowledging that Micah Kornfield and Antoine Pitrou hold differing views on the ideal mechanism while observing that nobody argued versions cannot work. The community sync surfaced no dissent either. Micah Kornfield, Russell Spitzer, Daniel Weeks, Gunnar Morling, and Pitrou participated as the vote thread gathered ten messages. This decision unblocks a queue of format work, starting with specifying how files carry the format version. Andrew Lamb kept a related question moving in the [SemVer for parquet-format releases thread](https://lists.apache.org/thread/rtz9mr89y564c4f4m01vbndcf89zmypq), which concerns versioning the specification artifacts themselves.\n\nWhy does versioning deserve this much ink? Because Parquet sits at the bottom of nearly every analytics stack on earth, and it has historically evolved through feature flags and reader capabilities rather than clean version boundaries. Forward incompatibility is the scary direction of change. Backward incompatibility breaks old files, which communities avoid at all costs. Forward incompatibility breaks old readers, meaning a file written with a new feature fails, or worse, silently misbehaves, in software that predates the feature. With Parquet readers embedded in everything from Spark clusters to embedded databases to decade-old ETL jobs nobody dares touch, the community needs a way to ship new capabilities without playing compatibility roulette. Recent additions like Variant, Geometry, and new encodings strain the old model past its limits. A predictable major-version mechanism means query engines can advertise Parquet 3 support as a coherent unit, vendors can test against a fixed target, and users can reason about compatibility without memorizing a feature matrix. The INT96 story this week shows the flip side of format ambiguity: Micah Kornfield [closed the vote defining ordering for INT96 timestamps](https://lists.apache.org/thread/4kq2gfgx3cmow26h8ofd93653z9zhs2t), pinning down semantics for a legacy type that engines interpreted loosely for years.\n\nThe ecosystem around the format had its own headline. Gunnar Morling announced [Hardwood 1.0, a new Parquet reader for the JVM](https://lists.apache.org/thread/8o3dw262j17ykt9688lg5jqj72kjdsmx). Hardwood targets a gap that has annoyed JVM users for a decade: parquet-java carries heavy Hadoop dependencies, and plenty of applications want to read Parquet without dragging in a distributed filesystem stack. A microservice that reads a few Parquet files from S3 should not inherit hundreds of megabytes of Hadoop transitive dependencies and their security advisories. The C++, Rust, and Go ecosystems solved this with clean standalone readers years ago, and Hardwood brings the JVM the same option. Community response was warm and practical. Pritam Pan asked about Spark integration, and Morling sketched a plausible path where an engine keeps its own decoders but adopts a Hadoop-free metadata, IO, and filtering layer underneath. Steve Loughran connected Hardwood to his pending parquet-testing fixtures for malformed files, and Morling reported the results candidly: Hardwood rejected nearly everything, but several rejections happened incidentally rather than for the right reasons, a few fixtures exposed missing validation, and one surfaced a latent bug. That exchange is open source working as designed, with a new implementation and a shared test corpus hardening each other in public. It also echoes the security thinking in Loughran's separate thread asking [how deep a realistic Variant value nests](https://lists.apache.org/thread/jtgndn5tndyo3rykj8y2m8xfy33kzb4k), part of an ongoing effort to harden Variant handling against hostile inputs. Kevin Liu, Loughran, and Ed Seidl also worked through [expected behavior when older parquet-java readers encounter VARIANT columns](https://lists.apache.org/thread/7hmgy6v73dp7xfkqzsqlksop726zlsm), another compatibility question the new versioning policy will make easier to answer in the future.\n\nEncodings research added a forward-looking note. Prateek Gaur opened threads on [ALP encoding for floating point data](https://lists.apache.org/thread/6mqpmv70ygnk4xkldy7comycjnxl7z48) and [PFOR encoding](https://lists.apache.org/thread/952t4hqdm1kvywkwvkhg9o60z9l9pv2k). ALP, short for adaptive lossless floating point, exploits the fact that most real-world floats are decimals in disguise, encoding them as scaled integers that compress far better than raw IEEE bits, with an exact fallback for values that resist the trick. It has shown strong results in the research literature and in modern engine formats, and floats dominate ML feature data, so the pairing with the FIXED_SIZE_LIST discussion is natural. Between the fixed-size type, float-native encodings, and the footer redesign work, a picture emerges of Parquet systematically retooling for the vector era rather than ceding that ground to specialized formats. The healthy sign is that each piece arrives through the normal proposal process, with benchmarks attached, rather than as a rushed response to competitive noise. Divjot Arora proposed [extended precision nanosecond timestamps](https://lists.apache.org/thread/l41qxghl5wwtt58m40gxk34gjnqz7jjd), Daniel Weeks continued the [complete FILE proposal](https://lists.apache.org/thread/tyv6oxj0fcj5t3g6pm35ztj523g2fjn9), Jiayi Wang convened [session four of the Parquet Footer Working Group](https://lists.apache.org/thread/4ww185sm1l4khf257ksgk9hjd2qdg15f), and Eduard Tudenhöfner and Gunnar Morling discussed [adopting AssertJ for test assertions](https://lists.apache.org/thread/t5qfbmvsd1gz9kb81s7volw3kqw59x4j) in parquet-java. Julien Le Dem gathered input for the July board report. Add it up and Parquet had one of its most substantive weeks of the year.\n\nApache Ossie opened its doors this week, and readers of this newsletter got a preview of why it matters in the Polaris section above. Ossie comes from Open Semantic Interchange, and it defines a vendor-neutral specification for expressing business metrics, dimensions, and their relationships, so a definition like monthly active users means the same thing to every tool that touches it. The design philosophy matters as much as the format. Rather than point-to-point field mappings between tools, Ossie standardizes the ontology and lets systems read semantic metadata straight from the source, which means the meaning travels with the data instead of being retranslated at every boundary. The project ships two main components, the specification itself plus bindings and converters from existing formats, so teams with definitions locked inside today's tools have a migration path. Ossie entered the Apache Incubator on June 22 with a mentor bench that signals how seriously the data community takes it: Jean-Baptiste Onofré, Zili Chen, Russell Spitzer, and Holden Karau, names readers of this newsletter will recognize from Iceberg, Polaris, and Parquet threads.\n\nThe dev list came alive on July 8. Onofré posted the [welcome message](https://lists.apache.org/thread/04gkfodyr4hh0vtyj2f5b4hwbz9j9zno), announcing that the website is live at ossie.apache.org and the [GitHub repository](https://github.com/apache/ossie) has been populated. He followed with a [proposal for a bi-weekly community meeting](https://lists.apache.org/thread/x61r9whdjbh1mc8tg7t879rc2rbjz5wk) on Wednesdays at 6pm Pacific, and the proposed norms deserve notice: meetings host discussion only, decisions happen on the mailing list, sessions get recorded, and summaries return to the dev list. That is the Apache Way applied cleanly from day one, and it keeps the project legible to contributors in every time zone. Quigley Malcolm joined the thread, an early signal that the semantic layer community beyond the founding group is paying attention. With Polaris explicitly holding its semantic API in beta until the Ossie spec lands, the podling starts life with a waiting integration partner, which is a rare and useful forcing function for a specification project. Most standards efforts spend their first year searching for an implementer willing to bet on them. Ossie has one on day one, plus a coalition of semantic layer and BI vendors already invested in the OSI work it inherits.\n\nFor readers wondering whether to invest attention in a four-committer podling, consider the pattern this newsletter has tracked before. Polaris itself went from incubation to top-level project to production deployments in under two years because it standardized something the ecosystem had already decided it needed. Ossie targets the same kind of gap. Every data team maintains metric definitions somewhere, none of those definitions travel, and AI agents querying the lakehouse have made the cost of ambiguous metrics suddenly concrete. An agent asked for monthly active users has to resolve that phrase to logic, and today the answer depends on which tool it asks. A shared specification is the boring, correct fix, and boring, correct fixes are what Apache projects do best. This section will cover Ossie weekly as list activity warrants.\n\nThree threads of connective tissue stood out this week. The first is that format governance is converging on a shared playbook. Parquet voted to adopt versioned releases for breaking changes, the model Iceberg has used to sequence v2, v3, and the emerging v4 work. Arrow moved to deprecate format surface that never found users, applying the discipline of subtraction, and its stated policy of extension types over new message kinds is the same lesson expressed structurally. Iceberg spent the week closing spec ambiguities one thread at a time. These communities share contributors, and the people carry the lessons across projects. Ryan Blue, Russell Spitzer, Daniel Weeks, Antoine Pitrou, Micah Kornfield, Kevin Liu, and Andrew Lamb all appeared in multiple project discussions this week, and the versioning vote text itself acknowledged the cross-pollination with its nod to the Iceberg model that Iceberg originally borrowed from Parquet. For practitioners, the payoff of this convergence is predictability. When every format in the stack evolves through the same version-and-vote mechanism, upgrade planning becomes one skill instead of four.\n\nThe second thread is that semantics is becoming lakehouse infrastructure. Polaris pausing its own API vote to wait for Ossie is a small procedural act with a large implication: the community wants one shared semantic specification, not a catalog-specific dialect. Add the week's AI-adjacent format work, from fixed-size vectors and float encodings in Parquet to variant hardening and geo partitioning in Iceberg, and the direction is clear. The open lakehouse stack is being fitted for a world where the primary consumers of data include agents and models, and where meaning has to travel with the data rather than living in a BI tool's private configuration.\n\nThe third thread is that fresh implementations are stress-testing old specifications, to everyone's benefit. Hardwood 1.0 ran a shared corpus of malformed Parquet files and exposed validation gaps in a week-old 1.0 release, in public, with the author reporting the findings himself. Iceberg Rust burned two release candidates rather than ship a flawed one. Arrow Rust shipped 59.1.0 on schedule while the C++ line worked through 25.0.0 verification. Every additional independent implementation of these formats converts spec ambiguity from a latent hazard into a visible bug report, and the volume of clarification threads across Iceberg, Arrow, and Parquet this week shows that feedback loop running at full speed. Multiple implementations are not a maintenance burden for open formats. They are the immune system.\n\nWatch for the Parquet versioning vote result and the first concrete steps toward carrying a format version in files, since that decision unlocks the queue of pending format work including FIXED_SIZE_LIST. The Iceberg Rust 0.10.0 RC3 vote should conclude within days, and the Arrow 25.0.0 vote is in its verification window, so expect two release announcements shortly. On Polaris, the payload representation discussion should settle the shape of the semantic model API, and the newly released 1.6.0 starts reaching production deployments. The first Ossie community meeting is expected once the calendar details land, and the podling's first spec draft is the artifact the whole semantics conversation now waits on. The Iceberg statistics ownership debate has no obvious resolution yet, which makes it the design thread most worth reading closely next week, and the file-level access delegation proposal deserves the same attention from anyone running multi-tenant lakehouses. Quarterly board reports for Arrow and Parquet land this month, which usually surface useful project health summaries for both communities.\n\n**Get Started with Dremio**\n\n**Free Downloads**\n\n**Books by Alex Merced**", "url": "https://wpnews.pro/news/apache-data-lakehouse-weekly-july-1-to-july-8-2026", "canonical_source": "https://dev.to/alexmercedcoder/apache-data-lakehouse-weekly-july-1-to-july-8-2026-3non", "published_at": "2026-07-09 00:00:30+00:00", "updated_at": "2026-07-09 00:11:24.625111+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Apache Parquet", "Apache Iceberg", "Apache Polaris", "Apache Ossie", "Apache Arrow", "Iceberg Rust", "Danny Jones", "Shawn Chang"], "alternates": {"html": "https://wpnews.pro/news/apache-data-lakehouse-weekly-july-1-to-july-8-2026", "markdown": "https://wpnews.pro/news/apache-data-lakehouse-weekly-july-1-to-july-8-2026.md", "text": "https://wpnews.pro/news/apache-data-lakehouse-weekly-july-1-to-july-8-2026.txt", "jsonld": "https://wpnews.pro/news/apache-data-lakehouse-weekly-july-1-to-july-8-2026.jsonld"}}