{"slug": "an-in-depth-overview-of-the-apache-iceberg-1-11-0-release", "title": "An In-Depth Overview of the Apache Iceberg 1.11.0 Release", "summary": "Apache Iceberg 1.11.0, released on May 19, 2026, introduces major structural changes to enhance security and file format flexibility. Key improvements include native encryption for manifest lists using the GCM stream cipher to protect sensitive metadata, and a new pluggable File Format API that decouples Iceberg's core metadata management from specific storage formats like Parquet and ORC. These changes address critical vulnerabilities in open data lakehouses and enable easier integration of future file formats.", "body_md": "Apache Iceberg 1.11.0 was officially released on May 19, 2026, marking a major milestone in the evolution of open data lakehouse architectures. While minor point releases often focus on small bug fixes and dependency bumps, this release introduces fundamental structural changes. The community has completed major initiatives to improve security, extend file format capabilities, and optimize query planning overhead.\n\nThis release represents a convergence of two development focuses. First, it introduces structural changes to the core metadata specification to support advanced security features and lay the groundwork for future format revisions. Second, it stabilizes several feature sets in the Iceberg format specification, moving them from experimental status to fully stable defaults.\n\nThis post analyzes the most critical improvements in the Apache Iceberg 1.11.0 release. We will examine the specific GitHub pull requests, explain the underlying mechanics of each feature, and review what these changes mean for data engineers and platform architects.\n\n## Manifest List Encryption (PR #7770, #15813)\n\nSecurity in open data lakehouses has historically focused on encrypting the actual data files stored in object storage. While file-level encryption prevents unauthorized users from reading raw Parquet or ORC data, the table metadata has remained exposed. In a default setup, anyone with read access to the storage bucket could inspect the metadata JSON, manifest lists, and manifest files.\n\nThese metadata files contain sensitive structural details. An attacker scanning an unencrypted manifest list can extract file paths, column names, partitions, partition bounds, and exact null value counts. In highly regulated industries such as healthcare or financial services, this structural exposure constitutes a major data leak.\n\nTo resolve this vulnerability, PR #7770, introduced by @ggershinsky, adds native encryption for manifest lists. This change works alongside follow-up improvements in PR #15813. Manifest lists can now be encrypted using the Galois/Counter Mode (GCM) stream cipher.\n\n```\nMetadata JSON (Contains encryption state references)\n       │\n       ▼\nManifest List (Encrypted via GCM Stream Cipher) ◄── Decrypted in-memory during planning\n       │\n       ▼\nManifest Files (Point to encrypted Parquet data files)\n```\n\nThe table encryption configuration can be defined during table creation or updated via table properties:\n\nWhen a query engine plans a scan against an encrypted table, it performs the following sequence:\n\n- The client queries the catalog to fetch the table metadata.\n- The catalog returns the metadata location along with the required decryption keys.\n- The query engine reads the encrypted manifest list from object storage.\n- Using the catalog keys, the engine decrypts the manifest list in-memory.\n- The engine processes the decrypted partitions and statistics to prune manifest files.\n\nThis approach ensures that the manifest list is never written to disk in plain text. It implements a model of envelope encryption: each metadata file is encrypted with a unique data encryption key (DEK), and these DEKs are encrypted using the table's master key managed by the Key Management Service (KMS). Even if an attacker gains raw access to the storage bucket, they find only encrypted bytes, protecting both the table contents and its structural metadata.\n\n## Pluggable File Format API and V4 Spec Foundations (PR #15049)\n\nHistorically, Apache Iceberg hardcoded its support for data file formats. The core library contained format-specific code paths for Parquet, ORC, and Avro. If you wanted to query or write a table, the engine executed internal code blocks tailored to those exact structures.\n\nThis hardcoded design created a major bottleneck for format innovation. If a team wanted to test a next-generation format, they had to modify the core engine codebase, extending complex switch statements and format-dependent utilities.\n\nPR #15049, introduced by @anoopj, restructures this architecture. It introduces a pluggable File Format API that decouples Iceberg core metadata management from physical storage layouts.\n\n```\n┌────────────────────────────────────────────────────────┐\n│                  Iceberg Core Engine                   │\n└───────────────────────────┬────────────────────────────┘\n                            │\n                            ▼\n┌────────────────────────────────────────────────────────┐\n│            File Format API Interface Layer             │\n└──────┬────────────┬─────────────┬─────────────┬────────┘\n       │            │             │             │\n       ▼            ▼             ▼             ▼\n  ┌─────────┐  ┌─────────┐   ┌─────────┐   ┌─────────┐\n  │ Parquet │  │   ORC   │   │ Vortex  │   │  Lance  │\n  └─────────┘  └─────────┘   └─────────┘   └─────────┘\n```\n\nThe File Format API provides a clean plugin interface. A file format is defined as a plugin that implements standard reader and writer interfaces. Iceberg core negotiates table transactions, schemas, and partition specs, while delegating the physical file access to the registered plugin.\n\nThis decoupling makes it practical to support next-generation formats:\n\n-\n**Vortex:** A general-purpose, modular format designed as a successor to Parquet. It is optimized for high-performance analytics, utilizing fixed-width columns with bitmap masks for nulls. This enables Single Instruction Multiple Data (SIMD) filtering directly on memory-mapped files without CPU decompression cycles. The community is actively using the new API to build a Vortex-backed Iceberg plugin. -\n**Lance:** A layout built for machine learning and AI workloads. It is optimized for high-dimensional vector search and random access to nested embeddings, implementing index structures such as Inverted File with Product Quantization (IVF-PQ) directly in the file format to enable fast query planning. -\n**Nimble:** A format optimized for wide tables containing thousands of feature columns. Nimble prioritizes fast decoding over high compression ratios, opting for lightweight run-length and bit-packing compression schemes. This reduces the CPU overhead of ML training loops that consume millions of rows per second.\n\nAdditionally, PR #15049 introduces the foundational Java interfaces and types for the upcoming V4 manifest specification. These changes prepare Iceberg for format-agnostic manifest storage, ensuring the metadata layer can scale to tables with millions of files without hitting Java memory overhead limits.\n\n## REST Client Protocols and Extended Headers (PR #12194)\n\nThe REST Catalog protocol has become the standard interface for managing Iceberg tables across multiple processing engines. It isolates clients from catalog catalog details and provides a unified API for schema management, snapshot commits, and credential vending.\n\nHowever, as deployments scale inside large enterprises, catalogs need to process custom client context. For example, a platform team might want to track which business unit submitted a query, pass custom security tokens, or inject correlation IDs for distributed tracing. In previous versions, the standard `RESTClient`\n\ndid not allow clients to send custom HTTP headers.\n\nPR #12194, written by @gaborkaszab, solves this constraint by extending header support inside the `RESTClient`\n\nimplementations.\n\n```\n┌────────────────────────────────┐\n│      Iceberg REST Client       │\n│  (Spark, Flink, Trino, etc.)   │\n└───────────────┬────────────────┘\n                │\n                │  POST /v1/namespaces/db/tables/events\n                │  Custom-Headers:\n                │    - X-Trace-Id: trace-98421\n                │    - X-Tenant-Id: finance-billing\n                │\n                ▼\n┌────────────────────────────────┐\n│      REST Catalog Server       │\n│  (Parses headers for auditing)  │\n└────────────────────────────────┘\n```\n\nWith this update, client engines can configure and inject custom headers into every REST call. The client-server handshake follows this sequence:\n\n- The client initializes the REST catalog using the properties map.\n- The client specifies static custom headers using the prefix\n`header.custom.`\n\n:\n\n```\n   header.custom.X-Tenant-Id=finance-billing\n   header.custom.X-Trace-Id=system-trace-99\n```\n\n- During request execution, the\n`RESTClient`\n\nintercepts the HTTP call and injects these custom headers. - The REST catalog server processes the headers to apply dynamic authorization, audit logging, or request routing.\n\nThis change enables the following capabilities:\n\n-\n**Auditing and Governance:** Engines can pass tenant identifiers or user profiles in the HTTP headers, allowing the REST catalog server to log catalog operations with full user context. -\n**Distributed Tracing:** Tracing headers such as W3C Trace Context can propagate from client engines through the catalog server, providing end-to-end trace visibility for query planning operations. -\n**Dynamic Authorization:** Clients can send custom authorization tokens that the REST catalog server evaluates dynamically to enforce fine-grained access control.\n\nThe properties are configured during catalog initialization using the standard configuration map, making it simple to roll out headers across existing query platforms.\n\n## Overwrite-Aware Table Registration (PR #15525)\n\nIn multi-tenant data platforms, multiple engines frequently access and modify the same table metadata. When registering a new table or importing an existing table state into the catalog, concurrency conflicts can occur.\n\nIf two separate processes attempt to register or overwrite a table reference at the same location simultaneously, a naive catalog might register the second request, silently overwriting the first. This creates data inconsistencies where the catalog points to an outdated or incorrect `metadata.json`\n\nfile.\n\nPR #15525, written by @sririshindra, adds overwrite-aware table registration to the catalog API.\n\n```\nWriter 1: Commits events_v1 ────► [Catalog Table Pointer] ◄──── Writer 2: Commits events_v2\n                                            │\n                                            ├────────► If conflict: Catalog rejects Writer 2\n                                            └────────► Prevents silent metadata overwrites\n```\n\nThis implementation leverages Optimistic Concurrency Control (OCC) at the catalog level. The conflict resolution sequence proceeds as follows:\n\n- Writer A and Writer B both read the current table state pointing to snapshot v1.\n- Writer A writes new data files, generating metadata version\n`metadata_v2.json`\n\n. - Writer B writes new data files in parallel, generating metadata version\n`metadata_v3.json`\n\n. - Writer A calls the catalog's\n`/v1/namespaces/db/tables/events/register`\n\nendpoint, stating that the expected base location is`metadata_v1.json`\n\n. - The catalog verifies the base matches, registers the new pointer to\n`metadata_v2.json`\n\n, and updates the table version. - Writer B attempts to register its state, listing\n`metadata_v1.json`\n\nas its expected base. - The catalog detects that the current pointer is now\n`metadata_v2.json`\n\n. - The catalog rejects Writer B's request, returning a HTTP 409 Conflict. Writer B must re-read the updated table state, resolve any overlapping partition commits, and retry the registration.\n\nThis validation ensures that table registration is safe and prevents silent metadata overwrites in highly active environments.\n\n## Deletion Vector Pruning in Snapshot Validation (PR #15653)\n\nOne of the major highlights of the V3 format specification is the stabilization of deletion vectors. Deletion vectors improve row-level delete performance by replacing positional delete files with Roaring bitmaps. Instead of writing a new delete file for every minor update, the engine updates a binary bitmap linked directly to the data file.\n\nThese deletion bitmaps are stored in the Puffin file format. You can inspect active deletion vector locations using metadata system tables:\n\n```\nSELECT file_path, pos, row_position, deletion_vector\nFROM TABLE(table_files('my_catalog.schema.events'));\n```\n\nHowever, as tables grow to hold millions of data files, validating these deletion vectors during query planning can introduce latency. During scan planning, the query engine must ensure that the deletion vectors linked in the metadata are valid and match the corresponding data files.\n\nIn earlier versions, this validation was executed across the entire table snapshot during plan initialization. If you had a 50 TB table and queried a single day, the planner still spent time validating deletion vectors for the entire table.\n\nPR #15653, introduced by @anoopj, optimizes this process. It adds manifest partition pruning to deletion vector validation inside the `MergingSnapshotProducer`\n\n.\n\n```\nQuery Filter: WHERE event_date = '2026-05-23'\n       │\n       ▼\nPartition Pruning Step\n       │\n       ├─► Skip Partition '2026-05-22' ──► Skip Deletion Vector Validation\n       │\n       └─► Read Partition '2026-05-23'  ──► Run Deletion Vector Validation\n```\n\nWith this change, the query planner matches the query filter predicates against partition bounds before executing deletion vector checks. If a partition is pruned out, the engine skips validating the deletion vectors for the files in that partition. This change reduces planning CPU cycles and improves scan startup times for partitioned tables.\n\nFor a detailed look at how hidden partitioning helps the query engine perform partition pruning and reduce metadata scan sizes, read the [Apache Iceberg Hidden Partitioning Post](https://alexmerced.blog/blog/2026-04-29-iceberg-masterclass-05-hidden-partitioning.html).\n\n## Scheduled Credential Lifecycle Refresh (PR #15678, #15732, #15696)\n\nTo security-harden data lakehouses, platforms avoid using long-lived storage credentials. Instead, query engines authenticate using temporary tokens vended by the REST catalog or cloud identity providers. These credentials typically have short lifespans, often expiring after one hour.\n\nThis security model creates issues for long-running operations. If a massive query runs for 90 minutes, or a streaming Flink sink runs continuously, the temporary credentials expire mid-job. When the client attempts to write new files or fetch manifests after the expiration window, the storage client throws an authentication exception, failing the job.\n\nThe 1.11.0 release resolves this lifecycle problem. PR #15678 (by @danielcweeks) and PR #15732 (by @nastra) add scheduled refresh threads to the `S3FileIO`\n\nclient. A parallel change in PR #15696 (by @nastra) implements the same capability for `GCSFileIO`\n\n.\n\n```\nQuery Thread (Reads/Writes Data)\n       │\n       ├───────► Token Expiration Approaching (e.g. at 50 minutes)\n       │\nBackground Refresh Thread\n       │\n       ├───────► Send Request to Catalog ──► Fetch New Credentials\n       │\n       └───────► Update S3FileIO/GCSFileIO Credentials In-Memory\n       │\nQuery Thread (Continues without interruption)\n```\n\nThe credential refresh system runs a background daemon thread that tracks token expiration times. The lifecycle is controlled by the following properties:\n\nBefore the active credential expires, the background thread automatically polls the catalog's `/v1/tokens`\n\nendpoint for refreshed tokens and updates the file system client in-memory. The main query and write threads continue to run without interruption, eliminating query failures caused by expired credentials.\n\n## Spark Streaming Triggers and Z-Ordering (PR #13824, #15706)\n\nApache Spark remains the primary engine for heavy write workloads and batch compaction in Iceberg tables. Version 1.11.0 includes several updates to improve Spark streaming and layout optimization.\n\n### Trigger.AvailableNow Support (PR #13824, #14026)\n\nPR #13824, introduced by @alexprosak, adds support for the `AvailableNow`\n\ntrigger in Spark Structured Streaming. This change was also backported to Spark 4.0, 3.5, and 3.4 in PR #14026.\n\n``` php\nContinuous Trigger:\n[Read Batch 1] -> [Write] -> [Wait] -> [Read Batch 2] -> [Write] -> (Runs indefinitely)\n\nAvailableNow Trigger:\n[Scan All Available Data] -> [Process Batch 1] -> [Process Batch 2] -> [Write All] -> [Graceful Shutdown]\n```\n\nIn Spark streaming, the default trigger runs continuously in the background, consuming resources even when no new files are arriving. The alternative `Once`\n\ntrigger processes only a single batch and shuts down, which can leave data unprocessed if a large backlog has accumulated.\n\nThe `AvailableNow`\n\ntrigger combines the benefits of both approaches. It scans the source for all available data, splits the workload into consecutive micro-batches, processes them all in a single run, and then shuts down the streaming context. This is configured in PySpark as follows:\n\n```\n# Configure Trigger.AvailableNow with Iceberg source and sink\nquery = spark.readStream \\\n    .format(\"iceberg\") \\\n    .load(\"prod_catalog.db.events\") \\\n    .writeStream \\\n    .format(\"iceberg\") \\\n    .trigger(availableNow=True) \\\n    .option(\"checkpointLocation\", \"/mnt/checkpoints/events\") \\\n    .toTable(\"prod_catalog.db.events_compacted\")\n```\n\nThis trigger configuration allows data platforms to run streaming ingestion pipelines as scheduled cron jobs, reducing cluster idle time.\n\n### Z-Order Column Collision Validation (PR #15706)\n\nPR #15706, introduced by @YanivZalach, addresses a failure mode during Z-order layout optimization. Spark uses the internal column name `ICEZVALUE`\n\nduring Z-order sorting. If a user table already contained a column named `ICEZVALUE`\n\n, the compaction process failed or generated incorrect sort orders.\n\nThe update adds strict schema validation that checks for column name collisions before running Z-order compactions, preventing data corruption.\n\n## Flink Post-Commit Maintenance and Branch Compaction (PR #15566, #15672, #14148)\n\nApache Flink is the standard engine for real-time streaming ingestion into Iceberg tables. Streaming ingestion has different write characteristics than batch ingestion, often writing many small files at high frequency. Iceberg 1.11.0 adds features to manage these files directly within Flink pipelines.\n\n### Flink Post-Commit Maintenance (PR #15566, #15667)\n\nPR #15566, written by [@mxm](https://dev.to/mxm), adds support for arbitrary post-commit maintenance tasks inside the Flink `IcebergSink`\n\nbuilder. This is also backported to active Flink branches in PR #15667.\n\nDuring streaming ingestion, Flink commits data to the Iceberg table at every checkpoint. These frequent commits generate a large number of small manifest files. With the new post-commit interface, you can attach background maintenance tasks directly to the sink:\n\n```\n// Configure Flink sink with post-commit compaction\nIcebergSink.forRowData(dataStream, tableLoader)\n    .table(icebergTable)\n    .tableLoader(tableLoader)\n    .writeParallelism(4)\n    .distributionMode(DistributionMode.HASH)\n    .postCommitMaintenance(\n        PostCommitMaintenance.builder()\n            .optimizeDataFiles(true)\n            .rewriteManifests(true)\n            .build()\n    )\n    .append();\n```\n\nAfter a commit succeeds, Flink runs compaction and manifest cleaning tasks in the background, keeping the table structure optimized without requiring external scheduler jobs.\n\n```\nFlink Stream Ingestion\n       │\n       ▼\n[Commit Data File (Checkpoint)]\n       │\n       ├───────► Post-Commit Trigger\n       │\n       ▼\n[Background Maintenance Action (RewriteDataFiles / Compaction)]\n```\n\n### Flink Branch Compaction Support (PR #15672, #15690)\n\nPR #15672, also written by [@mxm](https://dev.to/mxm), adds branch support to the Flink `RewriteDataFiles`\n\naction.\n\nHistorically, Flink's background compaction actions could only run on the table's main branch. In modern architectures, engines often ingest experimental data or staging runs into separate table branches. Flink can now run file compaction directly on these non-main branches, keeping staging and experiment branches organized before they are merged back.\n\n### Flink Metadata Columns (PR #14148)\n\nPR #14148, introduced by @Guosmilesmile, exposes metadata columns to Flink readers.\n\nFlink applications can now read the `_row_id`\n\nand `_last_updated_sequence_number`\n\nsystem columns. This is useful for CDC (Change Data Capture) reconciliation pipelines that need to track the exact ingestion sequence of rows.\n\n## JSON to Variant Mapping and Spec Cleanups (PR #13137, #14045)\n\nThe Variant type is a key part of the Iceberg V3 specification, designed to store semi-structured data using a binary representation that supports predicate pushdown. Iceberg 1.11.0 refines this integration across multiple engines.\n\n### Variant Type Validation (PR #13137, #14081)\n\nPR #13137 (by @manirajv06) and PR #14081 (by @geruh) add schema validation and filtering rules for the Variant type in Parquet metrics.\n\nThese updates ensure that Parquet file readers can extract column-level statistics from nested variant structures. This allows the query engine to prune files based on nested variant fields, improving query performance.\n\n### Trino Variant Type Mapping\n\nIn parallel, query engine connectors are adopting these changes. Trino now maps its native `JSON`\n\ntype to Iceberg's Variant type in V3 tables. This means you can write JSON data from Trino and query it with predicate pushdown, avoiding the performance penalties of plain string JSON.\n\n### Positional Deletes with Row Data Deprecated (PR #14045)\n\nPR #14045, written by @pvary, deprecates positional delete files that embed row data.\n\nIn Iceberg V2, positional delete files could store the actual deleted row data alongside the file path and row offset. While this design saved a join step during reads, it duplicated data in the delete files, increasing storage costs and metadata complexity.\n\nThe community has deprecated this option in favor of Deletion Vectors, simplifying the V3 read path.\n\n## Table Upgrade Path and Connector Compatibility\n\nAll V3 features: manifest list encryption, deletion vectors, Variant types, geospatial types, and nanosecond timestamps: require upgrading your tables to format version 3.\n\n```\nExisting V2 Table\n       │\n       ├───────► Run: ALTER TABLE events SET TBLPROPERTIES ('format-version' = '3')\n       │\nUpgraded V3 Table\n       │\n       ├───────► New writes use Deletion Vectors and Variant type\n       └───────► Existing data files are left untouched (no rewrite required)\n```\n\nThe upgrade is a metadata-only operation executed using SQL:\n\n```\n-- Upgrade an existing table to Iceberg V3 format version\nALTER TABLE my_catalog.schema.events\nSET TBLPROPERTIES ('format-version' = '3');\n```\n\nThis operation updates the `format-version`\n\npointer in the table's metadata JSON. It does not rewrite your existing data files, which remain in place and continue to be readable.\n\nNew writes to the table will adopt V3 features automatically. For example, subsequent update or delete statements will write deletion vectors instead of positional delete files.\n\n### Lifecycle Status Updates\n\nBefore planning your migration to V3, review the engine compatibility changes in Iceberg 1.11.0:\n\n-\n**Java 11 Support Dropped:** Iceberg 1.11.0 drops support for Java 11. Core libraries and engine connectors now require**Java 17** or**Java 21**. -\n**Spark 3.4 Support Deprecated:** Support for Spark 3.4 is deprecated. Teams should migrate to Spark 3.5 or Spark 4.0+. -\n**Flink 1.19 Support Removed:** Flink 1.19 is no longer supported. The release adds support for**Flink 2.1.0**.\n\nMake sure all query engines and toolchains in your lakehouse deployment support Iceberg V3 and Java 17 before upgrading production tables.\n\nFor more on managing query performance optimizations and table format versions inside Dremio, refer to the [Dremio Autonomous Performance Blog](https://www.dremio.com/blog/how-dremio-keeps-agentic-analytics-fast-without-manual-tuning/).\n\n## Conclusion\n\nApache Iceberg 1.11.0 is a significant release for the project. It moves beyond incremental enhancements to deliver major architectural updates.\n\nThe unified File Format API restructures how Iceberg interacts with physical storage formats. This change makes it easier to integrate next-generation codecs designed for AI and high-performance workloads.\n\nAt the same time, the stabilization of V3 features provides a production-ready path for deletion vectors, Variant data, geospatial types, and nanosecond precision. These features help organizations optimize query performance and reduce operational overhead.\n\nIf you are running Iceberg V2 tables in production, evaluate your workloads to identify tables that will benefit from a V3 upgrade. In particular, tables with active update patterns or large JSON columns will see immediate performance gains.\n\n### Build Your Data Lakehouse Expertise\n\nIf you are designing, building, or managing modern data platforms, staying ahead of formatting specifications is critical. To deepen your understanding of these technologies, consider reading:\n\n-\n**\"Architecting an Apache Iceberg Lakehouse\"**: An architectural guide to designing open lakehouse platforms, managing catalog architectures, and optimizing table layouts. -\n**Other Data Lakehouse Publications**: Practical books covering hidden partitioning, schema evolution, and query acceleration engines.\n\nFind these books and other lakehouse learning resources at [books.alexmerced.com](https://books.alexmerced.com).\n\nTo query your Iceberg V3 tables with automatic file layout optimization, background compaction, and zero infrastructure management, start a free trial of Dremio Cloud at [dremio.com/get-started](https://www.dremio.com/get-started).", "url": "https://wpnews.pro/news/an-in-depth-overview-of-the-apache-iceberg-1-11-0-release", "canonical_source": "https://dev.to/alexmercedcoder/an-in-depth-overview-of-the-apache-iceberg-1110-release-4l1n", "published_at": "2026-05-23 16:52:51+00:00", "updated_at": "2026-05-23 17:01:54.114150+00:00", "lang": "en", "topics": ["open-source", "data", "cloud-computing", "developer-tools", "enterprise-software"], "entities": ["Apache Iceberg", "Apache Iceberg 1.11.0", "Parquet", "ORC"], "alternates": {"html": "https://wpnews.pro/news/an-in-depth-overview-of-the-apache-iceberg-1-11-0-release", "markdown": "https://wpnews.pro/news/an-in-depth-overview-of-the-apache-iceberg-1-11-0-release.md", "text": "https://wpnews.pro/news/an-in-depth-overview-of-the-apache-iceberg-1-11-0-release.txt", "jsonld": "https://wpnews.pro/news/an-in-depth-overview-of-the-apache-iceberg-1-11-0-release.jsonld"}}