{"slug": "maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup", "title": "Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup", "summary": "This article, part 10 of a 15-part Apache Iceberg Masterclass, explains the four key maintenance operations required to keep Iceberg tables healthy: compaction, snapshot expiry, orphan file cleanup, and manifest rewriting. It details how compaction merges small files into optimally-sized ones (128-512 MB) and can optionally re-sort data to improve query performance, while snapshot expiry removes old metadata to enable time travel retention policies. The article also covers three approaches to running these operations—manual scheduling, threshold-based automation, and autonomous platform handling—and provides code examples for Spark and Dremio.", "body_md": "This is Part 10 of a 15-part Apache Iceberg Masterclass. Part 9 covered how tables degrade. This article covers the four maintenance operations that keep Iceberg tables healthy and the three approaches to running them.\nCompaction reads small files, merges them into optimally-sized files (128-512 MB), and optionally re-sorts the data. It is the most impactful maintenance operation because it directly addresses the small file problem and restores sort order effectiveness.\nIn Spark:\nCALL system.rewrite_data_files('analytics.orders')\nIn Dremio:\nOPTIMIZE TABLE analytics.orders REWRITE DATA USING BIN_PACK\nCompaction with sorting rewrites files so that column values are ordered, tightening the min/max statistics and making file skipping far more effective:\nOPTIMIZE TABLE analytics.orders REWRITE DATA USING SORT (order_date, customer_id)\nSnapshot expiry removes old snapshots from the metadata. After expiry, the snapshot and its exclusive data files are eligible for cleanup. You typically retain snapshots for a window (e.g., 7 days) to support time travel, then expire everything older.\n-- Spark\nCALL system.expire_snapshots('analytics.orders', TIMESTAMP '2024-04-22 00:00:00')\n-- Dremio\nALTER TABLE analytics.orders EXPIRE SNAPSHOTS OLDER_THAN = '2024-04-22 00:00:00'\nAfter snapshots are expired, the data files they exclusively referenced become orphans. Orphan cleanup scans the storage directory, compares files against the current metadata, and deletes files that are not referenced by any snapshot.\n-- Spark\nCALL system.remove_orphan_files('analytics.orders')\nThis operation should run after snapshot expiry and with a safety delay (e.g., files older than 3 days) to avoid deleting files from in-progress writes.\nRunning orphan cleanup too aggressively can delete files from long-running write operations. A 3-day safety window ensures that any write operation has had time to complete before its files are considered orphans.\nOver many commits, manifests accumulate. A single snapshot's manifest list might reference hundreds of small manifests from individual commits. Manifest rewriting consolidates them into fewer, larger manifests.\n-- Spark\nCALL system.rewrite_manifests('analytics.orders')\nThis speeds up scan planning because the engine reads fewer manifest files. Each manifest file requires a separate I/O operation to read, so reducing the count from 500 to 20 eliminates 480 I/O round trips during query planning.\nStandard compaction (BIN_PACK) merges small files without changing the data order. Sort-order compaction rewrites files with data sorted by specified columns, which tightens the min/max statistics and makes file skipping more effective:\n-- Dremio sort-order compaction\nOPTIMIZE TABLE analytics.orders REWRITE DATA USING SORT (order_date, customer_id)\n-- Spark sort-order compaction\nCALL system.rewrite_data_files(\ntable => 'analytics.orders',\nstrategy => 'sort',\nsort_order => 'order_date ASC NULLS LAST, customer_id ASC NULLS LAST'\n)\nSort-order compaction is more expensive than BIN_PACK because it reads, sorts, and rewrites all data. However, the performance improvement for queries that filter on the sorted columns is substantial: file skipping can eliminate 90%+ of data files when the sort columns match common query filters.\nDecide how long to keep historical data accessible through time travel:\nLonger retention means more snapshots, more metadata, and more storage consumed by old data files. Shorter retention reduces costs but limits time travel capabilities.\nRun maintenance operations on a schedule using Spark, Trino, or Dremio. A typical pattern:\nPros: Full control over timing and configuration. Cons: Requires operational effort; forgotten or broken jobs lead to degradation.\nBuild a monitoring layer that checks table health metrics (Part 9 diagnostics) and triggers maintenance only when thresholds are exceeded (e.g., average file size drops below 64 MB).\nUse a platform that handles maintenance autonomously. Dremio's automatic table optimization runs compaction, expiry, and cleanup for tables managed by Open Catalog without any user configuration. AWS S3 Tables provides built-in compaction.\nFor most teams, starting with Dremio's autonomous optimization and only adding manual jobs for tables with unusual requirements is the most practical approach.\nRunning compaction during peak query hours: Compaction reads and rewrites data files, which competes with analytical queries for I/O bandwidth. Schedule compaction during off-peak hours, or use a separate compute cluster (Spark on EMR) that does not share resources with your query engine.\nExpiring snapshots too aggressively: If you expire snapshots while a long-running query is using one of them, the query can fail because the data files it needs might be cleaned up. Always keep snapshots for at least as long as your longest-running query.\nForgetting orphan cleanup: Many teams run compaction and snapshot expiry but forget orphan cleanup. Without it, compacted and expired data files accumulate indefinitely. Set up orphan cleanup as a weekly job with a 3-day safety window.\nNot monitoring after migration: Tables migrated from Hive or other formats (Part 15) often inherit poor file layouts. Run an immediate compaction pass after any in-place migration.\nPart 11 covers how to query the metadata tables that power diagnostics.", "url": "https://wpnews.pro/news/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup", "canonical_source": "https://dev.to/alexmercedcoder/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup-12en", "published_at": "2026-05-22 15:33:50+00:00", "updated_at": "2026-05-22 15:35:11.561279+00:00", "lang": "en", "topics": ["data", "open-source", "developer-tools", "cloud-computing", "enterprise-software"], "entities": ["Apache Iceberg", "Spark", "Dremio"], "alternates": {"html": "https://wpnews.pro/news/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup", "markdown": "https://wpnews.pro/news/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup.md", "text": "https://wpnews.pro/news/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup.txt", "jsonld": "https://wpnews.pro/news/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup.jsonld"}}