Best way to convert long Korean novel TXT files into a Hugging Face dataset for causal LM fine-tuning?

A Stack Exchange user asked for the best way to convert long Korean novel TXT files into a Hugging Face dataset for causal LM fine-tuning, and the answer recommends separating corpus storage from sequence construction. The practical default is to store by book or chapter and later chunk into fixed-token blocks using the exact model tokenizer, with no overlap initially and optional metadata like book_id. The answer suggests using the Hugging Face 'text' loader with sample_by='document' or building explicit records, and advises checking token counts and decoded sequences before training.

Some parts seem fairly well established, while others seem less settled: For continued pretraining / causal-LM training on novel text, I would separate how you store the corpus from how you construct the fixed-length sequences that the model actually trains on . So, for your specific questions, my default answers would be: | Question | Practical default | | Paragraph, scene, chapter, or fixed tokens per Dataset row? | Book or chapter for storage; fixed tokens later for training. They do not need to be the same unit. | | Tokenizer-aware 1024/2048-token chunking? | Yes. Construct training blocks from token IDs produced by the exact model tokenizer. | | Overlap? | No overlap initially. Consecutive blocks are the simplest baseline. | | Artificial line breaks? | Repair verified formatting artifacts, but preserve real paragraph/dialogue structure. | | Is {"text": "..."} enough? | Yes. book id , chapter id , etc. are optional but very useful metadata. | | Are 20,000+ character rows a problem? | Not by themselves. A long storage row does not mean feeding the whole row to the model at once. | | Preserve chapter/scene boundaries? | Preserve them in the canonical corpus. Training block boundaries do not necessarily have to match them. | In other words, I would not make the irreversible dataset representation itself a collection of 2048-token fragments. I would keep a clean, reconstructable corpus first, and derive model-specific training blocks from it. A reasonable first pipeline would be: original TXT files ↓ conservative cleanup ↓ canonical book/chapter records + metadata ↓ train/validation split at an appropriate source boundary ↓ Qwen tokenizer ↓ explicit document-boundary token where appropriate ↓ 1024/2048/etc. fixed-token blocks ↓ causal-LM labels That keeps the original book/chapter structure available if you later change tokenizer, block size, model, packing strategy, or evaluation setup. The Hugging Face text loader actually supports this distinction directly: it normally samples TXT files line-by-line, but it can also sample by paragraph or by entire document using sample by . See Loading text data with Datasets https://huggingface.co/docs/datasets/main/nlp load . For example, if one TXT file is one book: python from datasets import load dataset from pathlib import Path files = str p for p in sorted Path "novels" .glob " .txt" ds = load dataset "text", data files=files, split="train", sample by="document", If you want reliable book/chapter IDs, provenance, ordering information, etc., I would probably build the records explicitly instead: python from datasets import Dataset from pathlib import Path records = for path in sorted Path "novels" .glob " .txt" : text = path.read text encoding="utf-8" Only conservative, corpus-verified cleanup here. text = text.replace "\r\n", "\n" .replace "\r", "\n" records.append { "book id": path.stem, "text": text, } ds = Dataset.from list records text is all the language-model objective needs. The extra fields are mainly useful for splitting, inspection, debugging, and rebuilding the tokenized dataset later. Before doing a long training run, I would check only a few inexpensive things: - token counts per book/chapter using the exact Qwen tokenizer ; - a few before/after excerpts from the line-break cleanup; - whether the intended train/validation source units are actually disjoint; - a few decoded packed sequences around book boundaries; - where the document-boundary tokens ended up; - labels at document-boundary and padding positions; - how many tokens the final packing procedure drops or pads. Those checks tend to reveal preprocessing mistakes much more cheaply than discovering them after training. Why “Dataset row”, “document”, “training block”, and “attention boundary” are different There are really four boundaries here, and they can be chosen independently. This is mainly an organizational boundary. For example: row 0 = Book A row 1 = Book B or: row 0 = Book A / Chapter 1 row 1 = Book A / Chapter 2 row 2 = Book A / Chapter 3 row 3 = Book B / Chapter 1 ... Both are reasonable Dataset representations. The chapter version can be convenient because you can inspect, filter, reorder, or split chapters easily. The book version is simpler and preserves the original source with less preprocessing. This means: where should the model be told that one independent document ended and another begins? That does not have to equal the Dataset row boundary. For example, you can store chapters separately: row: Book A, Chapter 1 row: Book A, Chapter 2 row: Book A, Chapter 3 while treating them as one continuous semantic document: Chapter 1 Chapter 2 Chapter 3 <end of Book A So choosing “one chapter per row” does not imply that every chapter needs a document-end token. For your source format—usually one complete book per TXT— book boundaries are a very natural first choice for independent-document boundaries . Chapter-level document boundaries are also possible if that is deliberately what you want to model, but it is a separate decision. This is the compute boundary: token 0 ........ token 2047 token 2048 ..... token 4095 ... A 2048-token block can start or end in the middle of a chapter. That is not automatically a preprocessing error. The simple Hugging Face causal-LM example does essentially: tokenize → concatenate → split into fixed block size sequences → labels = input ids See the current run clm.py https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run clm.py . This is yet another question: if two documents are packed into one physical training sequence, should tokens in the second document be allowed to attend to tokens from the first document? An end-of-document token provides a semantic boundary signal . Strictly blocking attention across documents requires a different attention-mask/packing mechanism. I would not make strict cross-document attention isolation a requirement for the first version of this pipeline. It is an advanced packing choice that can be evaluated later if it matters. This separation is why I would avoid trying to find one universal answer to: paragraph vs scene vs chapter vs fixed tokens? Those units solve different problems. Cleaning the artificial line breaks This is probably the most corpus-specific part of the problem. Since you already know that some physical line breaks are formatting artifacts, I would correct those before token packing , but only when the rule is reliable enough. For example, this: 그는 천천히 문을 열었다. 아무도 없었다. may genuinely be a hard-wrapped representation of: 그는 천천히 문을 열었다. 아무도 없었다. But this: 그는 문을 열었다. “누구세요?” 아무 대답도 없었다. contains useful structural information. Flattening all \n characters would erase paragraph and dialogue formatting that you probably want the model to learn. So I would avoid a blanket transformation such as: text = text.replace "\n", " " A better approach is: 1. keep the original TXT unchanged; 2. normalize basic file-level things such as CRLF/LF consistently; 3. inspect representative sections from several books; 4. identify patterns that reliably indicate hard wrapping; 5. repair only those patterns; 6. compare cleaned and original excerpts before processing the full corpus. Things that can help diagnose the pattern include: - whether the preceding line ends in sentence punctuation; - whether either line begins/ends with quotation marks; - blank-line patterns; - typical physical line lengths; - chapter headings; - scene separators. But I would treat these as corpus diagnostics, not as universal Korean grammar rules. There is at least a close Korean-novel precedent: the KoCoNovel https://github.com/storidient/koconovel project says its preprocessing corrected incorrect line breaks in its Korean novel corpus. That supports treating this as a real preprocessing issue, but it does not provide a universal rule that can safely distinguish every artificial line break from intentional literary formatting. Also, the tokenizer will not make this distinction for you. Literal newlines affect the resulting token sequence, so it is better to decide what the canonical text should look like before creating training blocks. Qwen document boundaries and <|endoftext| For Qwen specifically, I would pay attention to document boundaries rather than simply concatenating every book with no separator. The Qwen documentation describes <|endoftext| as the end-of-document EOD control token inserted between documents in packed pretraining sequences: Qwen3 Key Concepts — Control Tokens https://github.com/QwenLM/Qwen3/blob/main/docs/source/getting started/concepts.md So a simple book-level stream can look like: tokens for Book A <|endoftext| tokens for Book B <|endoftext| tokens for Book C <|endoftext| Then that token stream can independently be divided into 2048-token blocks. One important detail is that ordinary raw-text tokenization should not be assumed to add the pretraining document marker automatically. I would make it explicit: python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from pretrained "Qwen/Qwen3-0.6B-Base" eod id = tokenizer.convert tokens to ids "<|endoftext| " ids = tokenizer text, add special tokens=False, "input ids" ids.append eod id If your Dataset rows are chapters but you want book-level document boundaries, do not blindly append EOD after every row. Instead preserve book id / chapter order, concatenate the chapters belonging to one book, and append one EOD when that book ends. For example: Book A / Chapter 1 Book A / Chapter 2 Book A / Chapter 3 <|endoftext| Book B / Chapter 1 ... That is one reason why keeping metadata separate from the model-specific token representation is useful. Also, EOD does not have to coincide with a 2048-token boundary. A physical block can legitimately contain: ...end of Book A <|endoftext| beginning of Book B... That is normal packed-language-model structure. A simple tokenizer-aware packing implementation For a small/medium corpus, I would start with an intentionally simple implementation before introducing a specialized packing framework. Assume the train/validation split has already been made at the source level. python from datasets import Dataset from transformers import AutoTokenizer MODEL ID = "Qwen/Qwen3-0.6B-Base" BLOCK SIZE = 2048 tokenizer = AutoTokenizer.from pretrained MODEL ID eod id = tokenizer.convert tokens to ids "<|endoftext| " def pack books records, block size=BLOCK SIZE : """ Clear reference implementation, not a streaming-optimized one. records: iterable of {"book id": ..., "text": ...} """ stream = for record in records: ids = tokenizer record "text" , add special tokens=False, "input ids" stream.extend ids stream.append eod id n full blocks = len stream // block size n used = n full blocks block size blocks = stream i:i + block size for i in range 0, n used, block size dropped tail tokens = len stream - n used lm dataset = Dataset.from dict { "input ids": blocks, "labels": block.copy for block in blocks , } return lm dataset, dropped tail tokens This gives you: - exact tokenizer-aware blocks; - no character-count approximation; - one EOD after each book; - no overlapping targets; - no padding for full blocks; - explicit accounting of the final incomplete tail. For large corpora I would implement the same idea incrementally/streamingly rather than constructing one giant Python list, but the data semantics can remain the same. The final incomplete block has several possible policies: Simple and often fine if it is a tiny fraction of the corpus. But count it: print "Dropped tail tokens:", dropped tail tokens Useful if you implement batched/streaming preprocessing and do not want to lose a remainder at every processing batch. Also valid. In that case, create an attention mask and set labels to -100 only for actual padding positions . For example, conceptually: labels = input ids.copy for i, is real token in enumerate attention mask : if not is real token: labels i = -100 For completely full, equal-length blocks, precomputing labels = input ids.copy and using a simple/default collator is particularly easy. This is also close to the core idea in Hugging Face’s run clm.py https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run clm.py , which concatenates tokenized examples, splits them into block size chunks, and copies input ids into labels . One small difference worth knowing: the stock example drops the incomplete remainder of each batched group texts operation. The source currently notes that map batched=True normally processes groups of 1,000 texts. This is not necessarily a serious loss, but for a relatively small novel corpus I would simply count the discarded tokens rather than assuming it is negligible. Should the chunks overlap? For ordinary continued pretraining where every token in every block contributes to the causal-LM loss, I would start with: block 0: tokens 0..2047 block 1: tokens 2048..4095 block 2: tokens 4096..6143 ... rather than: block 0: tokens 0..2047 block 1: tokens 1792..3839 ... With the second scheme, the overlap is not merely extra context: if all labels are active, those repeated tokens become training targets multiple times. That changes their effective weighting. That may occasionally be intentional, but I would not make it the default. There is another valid design: overlapping prefix used only as context new target tokens where labels for the repeated prefix are masked. Hugging Face’s fixed-length perplexity guide https://huggingface.co/docs/transformers/main/perplexity uses exactly this distinction for sliding-window evaluation: overlapping tokens can be supplied as context while the already-accounted-for targets are set to -100 . That does not mean you need sliding-window training here. It just illustrates why: “using overlap for context” and “training twice on every overlapped token” are different designs. For a first CPT pipeline, consecutive non-overlapping blocks are much easier to reason about. Should fixed blocks respect chapter or scene boundaries? I would preserve chapter/scene structure in the canonical corpus , but I would not initially require every training sequence to end exactly at a chapter or scene boundary. Suppose a chapter is 5,300 tokens and your block size is 2,048. You could simply get: block 1: chapter tokens 0..2047 block 2: chapter tokens 2048..4095 block 3: chapter tokens 4096..5299 + following text... That is a normal fixed-block causal-LM representation. Trying to force every chapter into its own training sequence can instead produce: - truncation; - substantial padding; - highly variable sequence lengths; - inefficient packing. There is legitimate research motivation for preserving document integrity more carefully. For example, Fewer Truncations Improve Language Modeling https://proceedings.mlr.press/v235/ding24f.html studies ordinary concatenate-and-split pretraining and proposes Best-fit Packing to reduce unnecessary document truncation while retaining efficient packing. I would treat that as an advanced comparison , not as evidence that every Korean-novel chapter must be kept intact. So my order would be: 1. preserve all semantic boundaries in your canonical data; 2. establish a simple fixed-block baseline; 3. only then compare document-aware packing if preserving complete chapters/scenes seems important to your actual evaluation goal. This also leaves the door open to more specialized long-context training later without making the initial dataset pipeline depend on it. Train/validation splitting: split according to what you want validation to mean I would avoid randomly generating thousands of adjacent 2048-token blocks first and then assigning those blocks independently to train and validation. Otherwise neighboring excerpts from the same novel can easily occur on both sides, making validation less independent than it appears. The correct source boundary depends on what you want to measure. Hold out complete books first: books ├─ train books └─ validation books then tokenize/pack each split independently This is conceptually similar to long-form language-model datasets such as PG-19 https://github.com/google-deepmind/pg19 , where complete books are assigned to train, validation, and test partitions. There is also broader language-model evidence that train/evaluation overlap can distort evaluation; see Deduplicating Training Data Makes Language Models Better https://aclanthology.org/2022.acl-long.577/ . I would not infer from that that your corpus necessarily has a duplication problem—it is simply a good reason not to create avoidable adjacent-text leakage yourself. Holding out an entire book may remove too much training data. Then a reasonable compromise can be: - complete held-out chapters; or - one or more contiguous held-out spans. Just interpret that validation correctly: it measures held-out text within the same source/domain , not generalization to a completely unseen book. A within-book validation split can still be useful for loss monitoring / early stopping. It just answers a narrower question. So I would choose the split unit from the evaluation goal, rather than treating “book split” as a universal rule. A small Qwen/Transformers padding caveat There is one implementation detail I would check once if you use generic dynamic padding. In the current Qwen3-0.6B-Base tokenizer configuration https://huggingface.co/Qwen/Qwen3-0.6B-Base/blob/main/tokenizer config.json , <|endoftext| is exposed as both the eos token and pad token . Meanwhile, the current implementation of DataCollatorForLanguageModeling mlm=False https://github.com/huggingface/transformers/blob/main/src/transformers/data/data collator.py constructs labels and sets positions whose token ID equals pad token id to -100 . That combination matters if <|endoftext| is also an intentional real EOD inside the sequence : value-based masking cannot distinguish real <|endoftext| document boundary from <|endoftext| used as padding just from the token ID. This general failure mode has also been discussed in Transformers issue 23530 https://github.com/huggingface/transformers/issues/23530 . So if your pipeline uses that kind of collator, I would inspect one batch and verify: php real EOD position - label is still the EOD token ID actual padding position - label is -100 If you produce completely full fixed-length blocks yourself, the easiest route is to precompute: labels = input ids.copy and use a simple collator that only stacks the equal-size examples. If you keep padded tails, mask padding based on the actual attention/padding mask , rather than blindly masking every occurrence of the EOD token ID. This is a small implementation detail, not a reason to change the overall dataset design. What block size should you actually use? I do not think there is a generally established answer such as: Korean novels should use exactly 2048 tokens. 1024 , 2048 , 4096 , etc. are engineering/training choices. For your goal, I would first measure the corpus with the exact tokenizer: lengths = len tokenizer text, add special tokens=False "input ids" for text in ds "text" print "min:", min lengths print "max:", max lengths print "mean:", sum lengths / len lengths Then choose a block length based on: - available VRAM / training throughput; - how much local narrative context you want per update; - total corpus size; - whether longer sequences materially reduce batch size; - what you intend to evaluate. Starting with 1024 or 2048 is perfectly reasonable as a simple baseline; 4096 is also a natural comparison if the training setup handles it comfortably. I would not claim one is intrinsically best without actually comparing the resulting training/evaluation behavior. Also, do not choose the training length only from one tokenizer metadata number. At the moment, the HF files for Qwen/Qwen3-0.6B-Base expose: Those fields are useful configuration information, but neither says that 131,072—or even 32,768—is the sensible sequence length for your particular continued-pretraining run. “Model can represent a long context” and “this is the most useful/economical training block size for this corpus” are different questions. About the 20,000+ character rows I would not add an arbitrary character-count limit just to keep Dataset rows short. A Dataset row can hold an entire book while the training preprocessing later produces many fixed-token examples from it. This pattern is not unusual in long-form language modeling. For example, PG-19 https://github.com/google-deepmind/pg19 is explicitly organized around complete books as documents. So: Dataset row: 150,000 characters of a book does not imply: model forward pass: all 150,000 characters at once The latter is controlled by the tokenized training blocks. For ordinary novels, I would therefore monitor: number of tokens number of resulting blocks remainder/padding rather than enforcing a fixed maximum number of characters in the canonical Dataset. Extremely large individual files can eventually become a preprocessing-memory/streaming engineering issue, but that is separate from the model’s sequence-length limit. If by “Hugging Face Dataset” you also mean publishing the text on the Hub Creating a local datasets.Dataset and publishing the underlying novels to the Hub are separate decisions. All of the preprocessing above can be done locally. If you later want to upload the actual novel text to a Dataset repository, check the source/license/redistribution rights first. Hugging Face supports both public and private dataset repositories and provides Dataset-card metadata for documenting things such as license and provenance. See Sharing a dataset to the Hub https://huggingface.co/docs/datasets/main/upload dataset . That does not affect the technical choice between book/chapter rows and token blocks; it is simply a separate distribution/provenance question. So, if I wanted the least complicated known-good first version , I would probably do this: Keep original TXT files unchanged ↓ Repair only verified artificial hard wraps ↓ Store one book per row or chapter rows + book id if that is more convenient ↓ Choose train/validation source units before token chunking ↓ Tokenize raw prose with the exact Qwen tokenizer ↓ Add <|endoftext| at the chosen independent-document boundary probably each book initially ↓ Pack into consecutive fixed-length blocks 1024 or 2048 is a reasonable first experiment ↓ No overlap initially ↓ labels = input ids for real tokens ↓ Inspect a few decoded blocks, EOD positions, labels, and dropped/padded tail tokens ↓ Train Then, only if the baseline gives you a reason to do so, I would separately experiment with: longer block sizes document-aware packing chapter/scene-preserving packing context-only overlap cross-document attention isolation That way you do not have to solve every long-context or packing question before you can build a good Dataset. The main thing I would preserve carefully is the clean canonical corpus and its book/chapter metadata . Once that exists, generating a different 1024/2048/4096-token training representation later is cheap; reconstructing lost paragraph/chapter/book structure from already-chunked training rows is much harder.