{"slug": "save-a-dataset-while-it-s-being-created-by-from-generator", "title": "Save a dataset while it's being created by from_generator()", "summary": "Hugging Face's datasets maintainer lhoestq confirmed there is no official way to resume an interrupted Dataset.from_generator() build, recommending instead to split work into multiple Dataset objects or process source shards independently so a failure only requires redoing the last unfinished unit. The guidance, posted in a Hugging Face discussion, suggests using natural source shards or creating bounded output chunks, and emphasizes that progressive Arrow writes are not a supported resume checkpoint.", "body_md": "It looks like there isn’t currently an official way to do this:\n\nJust in case, [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\n\nThe closest direct answer I found is from `datasets`\n\nmaintainer `lhoestq`\n\non an almost identical question about resuming an interrupted `Dataset.from_generator()`\n\nbuild: **“It’s not currently possible”**, with the suggestion to split the work into multiple `Dataset`\n\nobjects so that one failure does not invalidate everything.\n\nSo I would probably keep your overall streaming/preprocessing approach, but move the **failure/restart boundary** outside one giant `Dataset.from_generator()`\n\ncall.\n\nThe simplest default is:\n\n```\nsource shard/chunk\n    ↓\nexpensive processing\n    ↓\nindependently finalized output shard\n    ↓\nnext shard/chunk\n```\n\nThen a restart only needs to redo the last unfinished unit instead of rebuilding the entire dataset.\n\nIf your source already has natural shards/files, I would use those first. If it does not, I would create bounded output chunks myself.\n\nThe important distinction is:\n\n“some Arrow data has been written to disk” is not the same thing as “I have a resumable checkpoint.”\n\n`Dataset.from_generator()`\n\ndoes write progressively while building, but the public API does not expose those intermediate builder files as a supported resume point.\n\nA useful decision flow is therefore:\n\n```\nDoes the source already have stable shards/files?\n\n├── yes\n│   └── process one source shard\n│       → finalize one output shard\n│       → skip completed shards on restart\n│\n└── no\n    └── is source order stable and roughly 1 input → 1 output?\n        ├── yes\n        │   └── create bounded output chunks\n        │       → finalize each chunk independently\n        │\n        └── no\n            └── use stable IDs / a manifest / source state\n                instead of relying only on a row counter\n```\n\nThat is also close to a later suggestion from `lhoestq`\n\nfor streaming preprocessing: stream the dataset, apply `.map(...)`\n\n, then process it shard-by-shard and write each shard separately.\n\nSo, for your current case, my default order would be:\n\nThat preserves the basic idea of generating the dataset progressively; it just makes the unit of successful work smaller than the entire `Dataset.from_generator()`\n\nbuild.", "url": "https://wpnews.pro/news/save-a-dataset-while-it-s-being-created-by-from-generator", "canonical_source": "https://discuss.huggingface.co/t/save-a-dataset-while-its-being-created-by-from-generator/179254#post_2", "published_at": "2026-08-26 12:23:03+00:00", "updated_at": "2026-08-26 12:43:32.460592+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Hugging Face", "lhoestq", "Dataset.from_generator()", "Arrow"], "alternates": {"html": "https://wpnews.pro/news/save-a-dataset-while-it-s-being-created-by-from-generator", "markdown": "https://wpnews.pro/news/save-a-dataset-while-it-s-being-created-by-from-generator.md", "text": "https://wpnews.pro/news/save-a-dataset-while-it-s-being-created-by-from-generator.txt", "jsonld": "https://wpnews.pro/news/save-a-dataset-while-it-s-being-created-by-from-generator.jsonld"}}