{"slug": "datasets-5-ruins-sharding-when-shuffle-is-called", "title": "Datasets >= 5 ruins sharding when shuffle() is called", "summary": "Hugging Face's Datasets 5.0.0 changed IterableDataset.shuffle() to fill the shuffle buffer from multiple input shards, which collapses an 8-shard Parquet stream into 1 logical shard and leaves only 1 effective DataLoader worker, down from 4 in v4.8.5, according to a reproduction by a Hugging Face forum user. The user recommends dataset.reshard() or the max_buffer_input_shards=1 parameter to restore pre-v5 behavior, while noting all 4096 rows were returned without duplicates in every tested configuration.", "body_md": "The shuffle change in v5 appears to be intentional, but whether this particular behavior is intended is much more questionable:\n\n[@lhoestq](https://discuss.huggingface.co/u/lhoestq)\n\nI can reproduce the `8 shards -> shuffle() -> 1 shard -> one effective DataLoader worker`\n\nbehavior independently.\n\nMy current read is:\n\n`IterableDataset.shuffle()`\n\nso that the shuffle buffer can be fed from multiple input shards at once;`num_shards`\n\nFor an 8-file Parquet stream, the most practical routes I would try are:\n\n```\n# Route A: keep the v5 multi-shard shuffle, but create more logical\n# shards first when the Parquet files contain multiple row groups.\ndataset = dataset.reshard()\ndataset = dataset.shuffle(seed=42, buffer_size=...)\n```\n\nor, if preserving the pre-v5 behavior is more important:\n\n```\n# Route B: documented compatibility path for the old shuffle behavior\ndataset = dataset.shuffle(\n    seed=42,\n    buffer_size=...,\n    max_buffer_input_shards=1,\n)\n```\n\nThere is also a useful middle ground for your specific 8-shard / 4-worker case:\n\n```\ndataset = dataset.shuffle(\n    seed=42,\n    buffer_size=...,\n    max_buffer_input_shards=2,\n)\n```\n\nIn my small reproduction, that retained **4 logical shards and all 4 DataLoader workers**, while still allowing the shuffle buffer to draw from more than one input shard at a time.\n\nI would probably try `reshard()`\n\nfirst for Parquet if it gives you enough logical shards, because that preserves the new v5 cross-shard mixing behavior. `max_buffer_input_shards=1`\n\nis the clean compatibility option if you specifically want the old semantics.\n\nA controlled reproduction I ran looked like this:\n\n| Datasets version / configuration | `num_shards` after shuffle |\nDataLoader workers that actually yielded examples |\n|---|---|---|\n4.8.5, default `shuffle()` |\n8 | 4 |\n5.0.0, default `shuffle()` |\n1 | 1 |\n5.0.1, default `shuffle()` |\n1 | 1 |\ncurrent main (`5.0.2.dev0` when tested), default |\n1 | 1 |\n5.x, `max_buffer_input_shards=1` |\n8 | 4 |\n5.x, `max_buffer_input_shards=2` |\n4 | 4 |\n5.x, `max_buffer_input_shards=4` |\n2 | 2 |\n5.x, `reshard()` then default shuffle |\n64 → 6 | 4 |\n\nThe test dataset was eight local Parquet files, 512 rows each, with eight row groups per file. Every tested configuration still returned all **4096 unique rows with zero duplicates**, so in that small case the thing that changed was the execution/sharding topology, not dataset coverage.\n\nI would not read much into throughput numbers from such a small local-file test; active worker count and correctness are the useful observations here.\n\nWhy this happens in v5So, for your original questions:\n\n**Why the change from Datasets 4 to 5?**\n\nTo improve streaming shuffle quality by filling the shuffle buffer from multiple input shards instead of effectively processing one input shard at a time. That change is intentional and documented in [the 5.0 release](https://github.com/huggingface/datasets/releases/tag/5.0.0) and [#8194](https://github.com/huggingface/datasets/pull/8194).\n\n**Is the 8 → 1 behavior itself a bug?**\n\nThe `8 -> 1`\n\nlogical-shard result follows from the current implementation and is reproducible. What is much less clear is whether losing DataLoader worker parallelism as a consequence was an intended part of that API change. Given the current worker documentation, I would consider this worth an upstream clarification/issue rather than assuming it is expected.\n\n**Do you need to change how the streaming dataset is instantiated?**\n\nProbably not fundamentally. For Parquet, I would first try:\n\n```\ndataset = dataset.reshard()\ndataset = dataset.shuffle(...)\n```\n\nIf you need exact pre-v5 behavior:\n\n```\ndataset = dataset.shuffle(\n    ...,\n    max_buffer_input_shards=1,\n)\n```\n\nAnd for eight source shards with four workers, `max_buffer_input_shards=2`\n\nis also a reasonable low-cost experiment if you want to keep some of the new cross-shard mixing without collapsing below four logical shards.", "url": "https://wpnews.pro/news/datasets-5-ruins-sharding-when-shuffle-is-called", "canonical_source": "https://discuss.huggingface.co/t/datasets-5-ruins-sharding-when-shuffle-is-called/179493#post_4", "published_at": "2026-09-01 21:06:38+00:00", "updated_at": "2026-09-01 21:24:56.527626+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Hugging Face", "Datasets", "IterableDataset"], "alternates": {"html": "https://wpnews.pro/news/datasets-5-ruins-sharding-when-shuffle-is-called", "markdown": "https://wpnews.pro/news/datasets-5-ruins-sharding-when-shuffle-is-called.md", "text": "https://wpnews.pro/news/datasets-5-ruins-sharding-when-shuffle-is-called.txt", "jsonld": "https://wpnews.pro/news/datasets-5-ruins-sharding-when-shuffle-is-called.jsonld"}}