21:06
2026-09-01
discuss.huggingface.co
developer-tools
Datasets >= 5 ruins sharding when shuffle() is called
Hugging Face's Datasets 5.0.0 changed IterableDataset.shuffle() to fill the shuffle buffer from multiple input shards, which collapses an 8-shard Parquet stream into 1 logical shard and leaves only 1 …