The lakehouse community spent this week deciding what gets carried forward and what gets left behind. Iceberg voted to forbid new equality deletes in V4 and debated whether the format still needs Avro manifests at all. Parquet shipped 1.18.0 and then spent the back half of the week chasing two data corruption bugs that block adoption of that same release. Arrow, DataFusion, and Iceberg all wrestled with the same governance question from different angles: what do you do when AI-generated pull requests and AI-generated review comments start outpacing the humans who have to read them? Add a fresh DataFusion major release, a wave of Ossie converter contributions, and Polaris hardening its persistence and encryption story, and you get one of the busiest weeks on the Apache dev lists this summer.
Every claim below links to the source thread on lists.apache.org, so you can read the full discussions yourself.
The V4 spec work dominated the Iceberg list this week, and the biggest single development was Huaxin Gao's vote to deprecate equality deletes in V4. The proposal has three parts. Writing new equality deletes becomes forbidden in V4 tables because the V4 metadata will not define them as an allowed entry type. Reading them stays supported for backward compatibility, both for existing V2 and V3 tables and for equality deletes carried into upgraded V4 tables. And the upgrade itself stays metadata-only, with no synchronous rewrite of data or delete files required. The rationale Huaxin laid out is the one this community has been circling for two years: equality deletes impose an asymmetric cost paid on every read, they complicate the format, and they block features like CDC, row lineage, and incremental materialized view maintenance. Deletion vectors turn deletion into a flat, one-time cost, and the Flink ConvertEqualityDeletes work proves a viable replacement path exists. The thread drew 27 messages, with Manu Zhang pressing on whether a V2 or V3 upgrade to V4 requires a manifest rewrite, and Ryan Blue, Anurag Mantripragada, and Junwang Zhao weighing in on the mechanics. This is the clearest signal yet that streaming writers need a migration plan for their equality delete pipelines before V4 lands.
The upgrade question got its own dedicated thread when Shawn Chang opened a discussion on V3 to V4 upgrade expectations and migration practices. Shawn's concern is operational rather than technical. An implementation can perform a lightweight upgrade by creating a V4 root manifest that references existing pre-V4 manifests, then writing new metadata in V4 format going forward. What stays undefined is the lifecycle of those legacy manifests. Shawn worries the format will technically support a clean migration while the practical path relies on users running optional maintenance jobs they historically skip. His comparison to the equality delete situation landed with the thread's participants, including Russell Spitzer, Amogh Jahagirdar, and Manu Zhang. He proposed the community either spec the expected lifecycle or publish explicit guidance, and floated eager conversion of cheap metadata while leaving expensive data migration alone. Expect this to become a recurring theme as V4 firms up.
Steven Wu asked a question this week that sounds small and is not: should V4 manifests be Parquet-only? During the column update sync, the initial inclination was to keep the Avro option because it already exists, even though Avro cannot support projection reads on manifest files. With both formats available, every engine and integration has to choose, and most will pick Parquet for projection-read support anyway. Steven's argument is that requiring Parquet reduces the cognitive and decision burden on integrations while aligning with Iceberg's priority on scan planning performance, where projecting column stats from manifests matters. Manu Zhang, Russell Spitzer, Anoop Johnson, and Péter Váry all engaged, and the sync recording is public for anyone who wants the full context. If this direction holds, V4 becomes the version where Parquet takes over Iceberg's metadata layer, not just its data layer.
The column update work itself kept moving. Leonid Lygin followed up the earlier Column File representation thread with a proposal for a row group alignment optimization. The idea: supporting writers align all row groups in a Column File with the Base File, enabling supporting readers to do simple zero-copy reads. Péter Váry pushed back constructively, asking how update writers obtain the base file's row group boundaries in practice and whether readers even need an alignment flag, since a reader can seek to the nearest row group and discard leading rows whether or not alignment holds. Daniel Weeks, Gianluca Graziadei, and Ryan Blue joined the design work. This is the kind of detail that decides whether column-level updates become a practical feature or a spec curiosity.
Two spec votes moved to conclusion. Russell Spitzer called a vote to clarify content file uniqueness in the table spec, making explicit as a snapshot invariant what scan planning has assumed since PR 4272: duplicate live file paths in a snapshot produce undefined scan results. The vote gathered quick +1s from Matt Butrovich, Huaxin Gao, Junwang Zhao, and Maninder Parmar. Gábor Kaszab opened a vote to add key-id to table and partition statistics and deprecate key-metadata. The current spec stores the encryption key for table statistics as raw key-metadata inside unencrypted table metadata, which defeats the purpose. The fix points statistics at an encrypted key in the table metadata's encryption-keys list, mirroring how manifest list encryption already works. Alexander Bailey, Ryan Blue, and Russell Spitzer participated in the review.
Release trains kept rolling too. The 1.12.0 release discussion that Neelesh Salian is coordinating picked up two significant sub-threads. Felix Perez Diener from Stripe asked whether Flink 2.3 support makes the cut, noting Stripe has already started its Flink upgrade, and Péter Váry confirmed the community wants to settle the Flink version question for the next release. Cheng Pan raised a bigger question: with V3 features implemented in Iceberg Java, when does Spark switch its default table version from 2 to 3? Neelesh answered that Variant and Geo type gaps make that unlikely within the 1.12 timeline, and committed to starting a separate tracking thread. Meanwhile Kevin Liu moved PyIceberg 0.12.0rc1 through its release candidate vote, and Matt Topol opened the vote for the Apache Iceberg Terraform Provider v0.1.0 RC2, which will give infrastructure teams a first official path to managing Iceberg resources declaratively.
Two community threads deserve attention. Sung Yun announced early planning for Iceberg Summit 2027, with a sponsorship interest form open now and a call for Lead Sponsors who want to help fund and organize the event. The organizers want the PMC proposal to reflect a broad set of interested companies, so if your organization wants in, this is the moment to raise a hand. And Manu Zhang started a discussion on AI review comments that captured something every maintainer is feeling. Lengthy AI-generated review comments take real time to dissect, different AI reviewers operating on different context produce conflicting feedback loops, and it is unclear humans always read what their tools post. Manu's own practice is to read AI findings, rephrase the valid points, and post them manually. Junwang Zhao agreed with the principle while doubting it can be enforced: the key is not publishing review comments without understanding them yourself. Manu disclosed his email was polished by AI, which is either irony or proof of the point.
Rounding out the week: Shangqing Yang proposed Parquet Page Index pruning in Iceberg's custom reader, Neelesh Salian and Sung Yun continued the shared conformance fixtures discussion for cross-implementation testing, and Xiening Dai surfaced a V4 spec question about null_value_count on optional fields that pulled in Eduard Tudenhöfner and Anoop Johnson.
Polaris spent the week on the unglamorous work that makes a catalog trustworthy: transactional consistency, key management, and spec-level guarantees.
The deepest technical thread was the ongoing discussion of consistent multi-object changes in Polaris persistence. Robert Stupp and Dmitri Bourlatchkov are working through what a backend-agnostic change-set primitive needs to guarantee. Dmitri's analysis cut to the hard part: the state read by validation code is not necessarily reflected in the change set. Unchanged entities considered by validation can change in a parallel request, and some validation code talks to the MetaStore directly, outside the Resolver's data. For JDBC backends, he sketched a request-wide transaction at SERIALIZABLE isolation as one solution, weighing it against manually tracking all reads and redoing them in a small commit transaction. The design question is how to get these guarantees on JDBC without leaking transaction concepts into the NoSQL persistence layer, which will use different mechanisms. Prithvi S joined the thread as well. This work decides whether Polaris can promise atomic multi-entity operations across all its backends, which matters for everything from tags to grants.
That same consistency thinking showed up in EJ Wang's revised Polaris Tag Spec design proposal. After feedback from Robert Stupp, EJ made the consistency guarantees explicit backend conformance requirements rather than implications of a proposed JDBC layout. The contract now states that overlapping tag operations must behave as if one happened before the other, that a successful operation becomes fully visible while a failed one changes nothing, and that detach-all is all-or-nothing to API callers. Implementations that cannot provide the required result must reject the operation rather than report success with weaker semantics. The mechanism stays open: transaction, CAS, atomic batch, or provider-native operation. EJ also kept reverse lookup catalog-wide intentionally, framing the privilege model as an explicit disclosure contract instead of a serving optimization.
The semantic layer integration story advanced in the Semantic Model REST API payload discussion, which now directly connects Polaris to Apache Ossie. Dmitri Bourlatchkov accepted JSON response payloads following the Ossie JSON structure for the v1 API, with other payload types deferred. The open question is version signaling: if Ossie's JSON representation is not explicit about its spec version, Polaris has to indicate it somehow, probably with an envelope, because revising the whole Polaris API for every Ossie spec change is impractical. Yufei Gu sketched what format-and-version envelopes look like for both JSON and encoded payloads. Watch this thread if you care about catalogs serving semantic models to BI tools and agents, because the decisions here will shape how every engine consumes Ossie documents from Polaris.
Security work landed on two fronts. ITing Lee's proposal to add decrypt-only access for legacy AWS KMS keys fixes a real key rotation gap: today every configured KMS key receives encryption permissions when Polaris vends write-capable credentials, so an old key retained for reading existing data can still sign new writes. The proposed legacyKmsKeys configuration grants only DescribeKey and Decrypt, with validation rejecting keys that appear in both legacy and encrypt-capable categories because AWS combines Allow statements. Dmitri Bourlatchkov reviewed the PR. And the long-running Iceberg table encryption discussion reached a working consensus, with Yufei Gu backing Dmitri's position that PR 5060 is a valid incremental step, letting Polaris use encryption key IDs from metadata files when the operator trusts the linked object storage. Robert Stupp's earlier point stands as follow-up work: Iceberg's spec requires catalogs to protect encryption.key-id from tampering and verify metadata integrity, and Polaris still owes a complete answer there.
Operations and adoption threads rounded out the week. Eundo Lee made a direct appeal for reviewer attention on making the Relational JDBC schema name configurable, arguing schema inconfigurability blocks new users whose database conventions do not match the hard-coded POLARIS_SCHEMA, while walking through why shipped defaults preserve existing deployments untouched on upgrade. EJ Wang posted notes from the metrics architecture sync, where the module layout settled into core/ holding only the entity data model and a new spi/ module taking every other shared contract, with PR 5068 merged and 5204 being reshaped onto the split. Sung Yun returned from vacation to push the Polaris Terraform Provider repository creation forward through ASF infra friction. And a user question about Aliyun OSS credential vending drew a response from Yufei Gu, a reminder that cloud storage coverage requests keep arriving from every region.
Arrow had a quieter week by volume and a meaningful one by substance. Raúl Cumplido announced the Apache Arrow 25.0.1 release, a patch with 9 resolved issues since 25.0.0. Raúl also announced a new Arrow committer, Tadeja Kadunc, and the congratulations thread became the most active on the list, with David Li, Alenka Frim, Ruoxi Sun, and others welcoming her aboard.
On the format side, Mandukhai Alimaa opened the formal vote for the Canonical BigDecimal Extension Type. The proposed arrow.big_decimal canonical extension provides high-fidelity representation and transport for variable-scale numeric data, the kind that PostgreSQL NUMERIC, Trino DECIMAL, and Oracle NUMBER produce, without forcing a uniform scale across an entire column. Draft implementations already exist in both arrow-go and arrow-rs. Curt Hagenlocher and Micah Kornfield weighed in during the vote window. Anyone who has fought decimal scale mismatches while moving database data through Arrow knows exactly why this matters: it removes a whole class of lossy casts at the boundary between transactional systems and the analytics stack.
Governance took center stage in Nic Crane's discussion on limiting concurrent open PRs for non-committers. After a brief reprieve, AI contributions ticked up again, with some contributors not responding to feedback and leaving stale PRs open that block others from picking up the work. Nic did the analysis: non-committers average 1.43 concurrent open PRs with a median of 1, so a limit around 3 constrains the long tail without hurting productive contributors. An ASF infrastructure PR to enable the corresponding GitHub setting is already open, and Nic proposed Arrow push for it while agreeing on its own interim policy. Jeffrey Vo and Rok Mihevc joined the discussion, and as you will see below, Jeffrey carried the same question to DataFusion days later.
The community calendar filled in too. Ian Cook hosted the Arrow community meeting on August 12, and Nic Crane announced an Arrow Hackathon at Community Over Code Glasgow on October 13, open to everyone with curated issues from documentation to involved code changes and committers on hand to help newcomers get set up.
Parquet shipped its biggest release of the year and then immediately demonstrated why release announcements are the start of a story rather than the end. Fokko Driesprong announced Apache Parquet 1.18.0 on August 11 after the RC2 vote closed with support from Russell Spitzer and others.
Within days, Yiming Li from Broadcom's VMware Tanzu Greenplum team filed a blocker report of silent data corruption in 1.18.0. The bug sits in ByteBufferBackedBinary.getBytes() when reading repeated or array columns: shared page-wide buffers get clobbered during lazy record assembly. The sting is in the motivation. Yiming's team is upgrading to 1.18.0 specifically to resolve critical Jackson CVEs, so the corruption bug blocks a security upgrade. The fix duplicates the buffer before adjusting limits and positions, with regression tests added, and the ask is a fast review so a 1.18.1 patch release can unblock adoption. Then it got worse. Aaron Niskode-Dossett dug into the performance PR the first bug traced back to and found a second, similar corruption path: BytesInput.copy() promises a copy in its Javadoc but now returns a reference in some circumstances, and he posted a failing test that proves dictionary page copies alias source bytes. Aaron noted he did the deeper analysis with Codex's help, an AI-assisted review catching what human review of a broad performance PR missed. If you are planning a 1.18.0 upgrade for the Jackson CVEs, wait for 1.18.1.
The format side of the project delivered a milestone. Julien Le Dem closed the vote on using versions to release forward-incompatible changes with 5 binding +1s, 9 non-binding +1s, and no vetoes. Fokko Driesprong, Ryan Blue, Daniel Weeks, Micah Kornfield, Gang Wu, Ed Seidl, Matt Topol, Kevin Liu, Amogh Jahagirdar, Prateek Gaur, and Russell Spitzer all participated across the vote's life. This settles a question that has constrained Parquet evolution for a decade: how the format ships changes that old readers cannot process without breaking the ecosystem's trust. Julien opened a follow-up thread on finalizing the versioning proposal to complete the spec text. Every encoding discussed below moves faster because this passed.
Speaking of encodings, the new-encoding pipeline is full. Arnav Balyan announced the FSST proposal has finished final design review and is moving to implementation. FSST brings random-access string compression to Parquet, and implementation is already underway with Devan Benz building the arrow-rs version and Arnav's own Arrow C++ proof of concept. The call is out for owners of Parquet Java and Arrow Go implementations to enable cross-language interoperability testing. Gunnar Morling and Curt Hagenlocher joined the review discussion. Meanwhile the ALP floating-point encoding hit the interoperability phase: Andrew Lamb asked for verification of his proposed ALP test dataset covering varied vector sizes, distributions, and exceptions. Curt Hagenlocher and Vinoo Ganesh confirmed the C# and Java implementations read the file, Andrew verified Rust, and a blog post introducing ALP is in the works with Kosta and Prateek Gaur. Andrew also proposed moving the ALP spec to its own document page. Divjot Arora's extended precision nanosecond timestamps proposal advanced too, with Micah Kornfield reviewing the split-out spec change for how readers handle unsupported logical and physical type combinations and proposing a second implementation before a vote.
One small note with large implications: Julien Le Dem convened the regular Parquet sync on August 12, and Jiayi Wang canceled the August 18 footer sync. The footer redesign work continues on its own track alongside everything above.
DataFusion pushed a major release across the line. Tim Saucer ran the 55.0.0 release votes, with an RC2 that surfaced issues, extra backports, and an RC3 that gathered binding +1s from Andrew Lamb, Andy Grove, and Adrian Garcia Badaracco, who verified on Apple Silicon with Rust 1.97. Community members including Kumar Ujjawal, Gabriel Musat, and Martin Grigorov tested the candidates. The willingness to cut a third candidate rather than ship a known-flawed second one says something about where this project's quality bar sits as its embedder ecosystem grows.
The other DataFusion thread of note connects directly to Arrow's governance conversation. Jeffrey Vo opened a policy discussion on the uptick of LLM-generated PRs from new contributors, pointing to a GitHub discussion about new contributors submitting multiple apparently LLM-generated PRs at once. Jeffrey participated in Nic Crane's Arrow thread on the same problem days earlier, so the two communities are now working the question in parallel and can be expected to converge on compatible policies.
Ossie, the semantic layer spec project, had another week that shows why it has become the fastest-moving list in this newsletter's roster. The activity splits into three streams: a foundational debate about the query interface, a converter ecosystem filling out at speed, and core spec refinements.
The big debate arrived in stereo. Justin Talbot and Chris Eubank posted parallel discussions proposing SQL with measures as the Ossie BI and semantic layer interface. They agree with the goal of common queryable semantics that engines implement and BI tools query, but they raised structured concerns with the proposed foundational semantics in PR 246 and the compliance suite in PR 237. Their core argument centers on BI vendor buy-in: engines with multi-table query interfaces similar to the proposal already exist, and BI tools integrating with them typically ship lists of broken or unsupported features, because BI tools emit and optimize complex SQL for features like level-of-detail calculations, and semantic interfaces with non-SQL behavior break core assumptions. Their alternative is a smaller, less opinionated semantics built on top of the existing SQL standard, specifically SQL with measures. Will Pugh responded from the PR 246 side. This is the kind of architectural fork that determines whether a spec gets adopted by the tools it needs, so expect this debate to run for weeks.
The converter ecosystem keeps compounding. Ding Ye (Kunwu) from Alibaba proposed contributing a bidirectional converter for Alibaba Cloud Hologres Semantic View, mapping Hologres's dimensions-and-metrics model onto Ossie's FK-pair relationship semantics, isolated under converters/hologres/ with no spec changes. Mikhail Nitsenko from Cube asked for a final maintainer review of the Cube to Ossie converter PR 289, which follows the pattern of the recently merged WisdomAI and NVIDIA GSF converters, and Jean-Baptiste Onofré replied from a hiking trail that he will review when he returns August 20. Best of all, real-world validation arrived: a practitioner from a MetricFlow and dbt-databricks shop posted detailed feedback from two independent converter tests, a round-trip fidelity test of Databricks Metric Views through Ossie and back, and a generation test from dbt semantic manifests. The verdict: where the converters run, they are numerically faithful, with every converted Metric View returning the same values as hand-built ones, and round-trips preserving vendor specifics through custom_extensions. That is exactly the evidence an interchange spec needs.
Microsoft's Markus Cozowicz drove two core-spec threads. His proposal to register MICROSOFT as a well-known vendor token resolves a live conflict where PR 250 adds MICROSOFT while separate converter work uses POWER_BI. His argument: one object model backs Power BI, Fabric, Azure Analysis Services, and SQL Server Analysis Services, a model.bim does not record which product produced it, and the project's precedent names organizations, with SALESFORCE already covering Tableau. One canonical token, no aliases, five previously disagreeing vendor enumerations reconciled. He also proposed allowing ai_context and custom_extensions on the document root, fixing an asymmetry where every node except the root carries those fields, so document-wide agent guidance has nowhere to live and producers copy shared instructions into each model.
Community design discussions kept humming alongside: the metrics trees big idea drew a production report from the agentic-data-contracts project describing a two-edge-kind design that separates deterministic identity edges from labeled influence edges with explicit confidence levels, the entity and grain proposal continued, verified_queries as a core spec element gathered support, and Ankit Tandon posted notes from the Ossie Ontology working group sync. New introductions from Harel Shein of Datadog's OpenLineage team and Kyoung Min Kim reading the spec from the catalog side show the contributor funnel is healthy.
Three threads ran through every list this week. The first is AI contribution governance. Iceberg debated AI review comments, Arrow moved toward concurrent PR limits after an uptick in unresponsive AI contributions, DataFusion opened a policy discussion on LLM-generated PRs from new contributors, and in Parquet, an AI-assisted review by Aaron Niskode-Dossett found a real data corruption bug that human review missed. The picture is nuanced: AI is generating maintainer load through low-accountability contributions and simultaneously catching bugs when wielded by accountable experts. The policies these communities converge on in the next month, likely some combination of PR limits and understand-before-you-post norms, will become the template for the wider ASF.
The second theme is Parquet becoming the metadata substrate, not just the data substrate. Iceberg is seriously discussing Parquet-only V4 manifests for projection reads, Iceberg's readers are looking at Parquet Page Index pruning, and Parquet's own versioning vote gives the format a sanctioned path to evolve for exactly these new metadata workloads. The stack is consolidating around one columnar format at every layer.
The third theme is the semantic layer becoming load-bearing across projects. Polaris is designing its Semantic Model REST API around Ossie's JSON structure, Ossie is debating the query semantics BI tools will consume, and vendors from Alibaba to Microsoft to Cube are contributing converters in the same week. A year ago the semantic layer conversation was speculative. This week it looked like protocol engineering.
Watch for the equality delete vote result and whether Shawn Chang's V3 to V4 migration concerns turn into spec text. Parquet needs a 1.18.1 patch release fast, and the shape of that release will tell you how the project handles security-driven urgency. The DataFusion 55.0.0 announcement should land any day. In Ossie, the SQL-with-measures debate and JB's return from vacation on August 20 both promise movement. And Iceberg Summit 2027 sponsorship interest is open now, which is worth acting on if your company wants a seat at that table.
If you want to go deeper on any of this, from Iceberg internals to lakehouse architecture to agentic analytics, I have written a full shelf of books on these topics. Browse the complete catalog at books.alexmerced.com.