Some parts seem fairly well established, while others seem less settled:
For continued pretraining / causal-LM training on novel text, I would separate how you store the corpus from how you construct the fixed-length sequences that the model actually trains on.
So, for your specific questions, my default answers would be:
| Question | Practical default |
| Paragraph, scene, chapter, or fixed tokens per Dataset row? | Book or chapter for storage; fixed tokens later for training. They do not need to be the same unit. |
| Tokenizer-aware 1024/2048-token chunking? | Yes. Construct training blocks from token IDs produced by the exact model tokenizer. |
| Overlap? | No overlap initially. Consecutive blocks are the simplest baseline. |
| Artificial line breaks? | Repair verified formatting artifacts, but preserve real paragraph/dialogue structure. |
| Is {"text": "..."} enough? | Yes.book_id ,chapter_id , etc. are optional but very useful metadata. |
| Are 20,000+ character rows a problem? | Not by themselves. A long storage row does not mean feeding the whole row to the model at once. |
| Preserve chapter/scene boundaries? | Preserve them in the canonical corpus. Training block boundaries do not necessarily have to match them. |
In other words, I would not make the irreversible dataset representation itself a collection of 2048-token fragments. I would keep a clean, reconstructable corpus first, and derive model-specific training blocks from it.
A reasonable first pipeline would be:
original TXT files
↓
conservative cleanup
↓
canonical book/chapter records + metadata
↓
train/validation split at an appropriate source boundary
↓
Qwen tokenizer
↓
explicit document-boundary token where appropriate
↓
1024/2048/etc. fixed-token blocks
↓
causal-LM labels
That keeps the original book/chapter structure available if you later change tokenizer, block size, model, packing strategy, or evaluation setup.
The Hugging Face text actually supports this distinction directly: it normally samples TXT files line-by-line, but it can also sample by paragraph or by entire document using sample_by. See text data with Datasets.
For example, if one TXT file is one book:
from datasets import load_dataset
from pathlib import Path
files = [str(p) for p in sorted(Path("novels").glob("*.txt"))]
ds = load_dataset(
"text",
data_files=files,
split="train",
sample_by="document",
)
If you want reliable book/chapter IDs, provenance, ordering information, etc., I would probably build the records explicitly instead:
from datasets import Dataset
from pathlib import Path
records = []
for path in sorted(Path("novels").glob("*.txt")):
text = path.read_text(encoding="utf-8")
text = text.replace("\r\n", "\n").replace("\r", "\n")
records.append({
"book_id": path.stem,
"text": text,
})
ds = Dataset.from_list(records)
text is all the language-model objective needs. The extra fields are mainly useful for splitting, inspection, debugging, and rebuilding the tokenized dataset later.
Before doing a long training run, I would check only a few inexpensive things:
- token counts per book/chapter using the exact Qwen tokenizer ;
- a few before/after excerpts from the line-break cleanup;
- whether the intended train/validation source units are actually disjoint;
- a few decoded packed sequences around book boundaries;
- where the document-boundary tokens ended up;
- labels at document-boundary and padding positions;
- how many tokens the final packing procedure drops or pads.
Those checks tend to reveal preprocessing mistakes much more cheaply than discovering them after training.
#
Why “Dataset row”, “document”, “training block”, and “attention boundary” are different
There are really four boundaries here, and they can be chosen independently.
This is mainly an organizational boundary.
For example:
row 0 = Book A
row 1 = Book B
or:
row 0 = Book A / Chapter 1
row 1 = Book A / Chapter 2
row 2 = Book A / Chapter 3
row 3 = Book B / Chapter 1
...
Both are reasonable Dataset representations.
The chapter version can be convenient because you can inspect, filter, reorder, or split chapters easily. The book version is simpler and preserves the original source with less preprocessing.
This means: where should the model be told that one independent document ended and another begins?
That does not have to equal the Dataset row boundary.
For example, you can store chapters separately:
row: Book A, Chapter 1
row: Book A, Chapter 2
row: Book A, Chapter 3
while treating them as one continuous semantic document:
Chapter 1
Chapter 2
Chapter 3
<end of Book A>
So choosing “one chapter per row” does not imply that every chapter needs a document-end token.
For your source format—usually one complete book per TXT—book boundaries are a very natural first choice for independent-document boundaries. Chapter-level document boundaries are also possible if that is deliberately what you want to model, but it is a separate decision.
This is the compute boundary:
token 0 ........ token 2047
token 2048 ..... token 4095
...
A 2048-token block can start or end in the middle of a chapter. That is not automatically a preprocessing error.
The simple Hugging Face causal-LM example does essentially:
tokenize
→ concatenate
→ split into fixed block_size sequences
→ labels = input_ids
See the current run_clm.py.
This is yet another question: if two documents are packed into one physical training sequence, should tokens in the second document be allowed to attend to tokens from the first document?
An end-of-document token provides a semantic boundary signal. Strictly blocking attention across documents requires a different attention-mask/packing mechanism.
I would not make strict cross-document attention isolation a requirement for the first version of this pipeline. It is an advanced packing choice that can be evaluated later if it matters.
This separation is why I would avoid trying to find one universal answer to:
paragraph vs scene vs chapter vs fixed tokens?
Those units solve different problems.
#
Cleaning the artificial line breaks
This is probably the most corpus-specific part of the problem.
Since you already know that some physical line breaks are formatting artifacts, I would correct those before token packing, but only when the rule is reliable enough.
For example, this:
그는 천천히 문을
열었다. 아무도
없었다.
may genuinely be a hard-wrapped representation of:
그는 천천히 문을 열었다. 아무도 없었다.
But this:
그는 문을 열었다.
“누구세요?”
아무 대답도 없었다.
contains useful structural information. Flattening all \n characters would erase paragraph and dialogue formatting that you probably want the model to learn.
So I would avoid a blanket transformation such as:
text = text.replace("\n", " ")
A better approach is:
- keep the original TXT unchanged;
- normalize basic file-level things such as CRLF/LF consistently;
- inspect representative sections from several books;
- identify patterns that reliably indicate hard wrapping;
- repair only those patterns;
- compare cleaned and original excerpts before processing the full corpus.
Things that can help diagnose the pattern include:
- whether the preceding line ends in sentence punctuation;
- whether either line begins/ends with quotation marks;
- blank-line patterns;
- typical physical line lengths;
- chapter headings;
- scene separators.
But I would treat these as corpus diagnostics, not as universal Korean grammar rules.
There is at least a close Korean-novel precedent: the KoCoNovel project says its preprocessing corrected incorrect line breaks in its Korean novel corpus. That supports treating this as a real preprocessing issue, but it does not provide a universal rule that can safely distinguish every artificial line break from intentional literary formatting.
Also, the tokenizer will not make this distinction for you. Literal newlines affect the resulting token sequence, so it is better to decide what the canonical text should look like before creating training blocks.
#
Qwen document boundaries and <|endoftext|>
For Qwen specifically, I would pay attention to document boundaries rather than simply concatenating every book with no separator.
The Qwen documentation describes <|endoftext|> as the end-of-document (EOD) control token inserted between documents in packed pretraining sequences:
Qwen3 Key Concepts — Control Tokens
So a simple book-level stream can look like:
tokens for Book A
<|endoftext|>
tokens for Book B
<|endoftext|>
tokens for Book C
<|endoftext|>
Then that token stream can independently be divided into 2048-token blocks.
One important detail is that ordinary raw-text tokenization should not be assumed to add the pretraining document marker automatically. I would make it explicit:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B-Base")
eod_id = tokenizer.convert_tokens_to_ids("<|endoftext|>")
ids = tokenizer(
text,
add_special_tokens=False,
)["input_ids"]
ids.append(eod_id)
If your Dataset rows are chapters but you want book-level document boundaries, do not blindly append EOD after every row. Instead preserve book_id / chapter order, concatenate the chapters belonging to one book, and append one EOD when that book ends.
For example:
Book A / Chapter 1
Book A / Chapter 2
Book A / Chapter 3
<|endoftext|>
Book B / Chapter 1
...
That is one reason why keeping metadata separate from the model-specific token representation is useful.
Also, EOD does not have to coincide with a 2048-token boundary. A physical block can legitimately contain:
...end of Book A <|endoftext|> beginning of Book B...
That is normal packed-language-model structure.
#
A simple tokenizer-aware packing implementation
For a small/medium corpus, I would start with an intentionally simple implementation before introducing a specialized packing framework.
Assume the train/validation split has already been made at the source level.
from datasets import Dataset
from transformers import AutoTokenizer
MODEL_ID = "Qwen/Qwen3-0.6B-Base"
BLOCK_SIZE = 2048
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
eod_id = tokenizer.convert_tokens_to_ids("<|endoftext|>")
def pack_books(records, block_size=BLOCK_SIZE):
"""
Clear reference implementation, not a streaming-optimized one.
records:
iterable of {"book_id": ..., "text": ...}
"""
stream = []
for record in records:
ids = tokenizer(
record["text"],
add_special_tokens=False,
)["input_ids"]
stream.extend(ids)
stream.append(eod_id)
n_full_blocks = len(stream) // block_size
n_used = n_full_blocks * block_size
blocks = [
stream[i:i + block_size]
for i in range(0, n_used, block_size)
]
dropped_tail_tokens = len(stream) - n_used
lm_dataset = Dataset.from_dict({
"input_ids": blocks,
"labels": [block.copy() for block in blocks],
})
return lm_dataset, dropped_tail_tokens
This gives you:
- exact tokenizer-aware blocks;
- no character-count approximation;
- one EOD after each book;
- no overlapping targets;
- no padding for full blocks;
- explicit accounting of the final incomplete tail.
For large corpora I would implement the same idea incrementally/streamingly rather than constructing one giant Python list, but the data semantics can remain the same.
The final incomplete block has several possible policies:
Simple and often fine if it is a tiny fraction of the corpus.
But count it:
print("Dropped tail tokens:", dropped_tail_tokens)
Useful if you implement batched/streaming preprocessing and do not want to lose a remainder at every processing batch.
Also valid. In that case, create an attention_mask and set labels to -100 only for actual padding positions.
For example, conceptually:
labels = input_ids.copy()
for i, is_real_token in enumerate(attention_mask):
if not is_real_token:
labels[i] = -100
For completely full, equal-length blocks, precomputing labels = input_ids.copy() and using a simple/default collator is particularly easy.
This is also close to the core idea in Hugging Face’s run_clm.py, which concatenates tokenized examples, splits them into block_size chunks, and copies input_ids into labels.
One small difference worth knowing: the stock example drops the incomplete remainder of each batched group_texts() operation. The source currently notes that map(batched=True) normally processes groups of 1,000 texts. This is not necessarily a serious loss, but for a relatively small novel corpus I would simply count the discarded tokens rather than assuming it is negligible.
#
Should the chunks overlap?
For ordinary continued pretraining where every token in every block contributes to the causal-LM loss, I would start with:
block 0: tokens 0..2047
block 1: tokens 2048..4095
block 2: tokens 4096..6143
...
rather than:
block 0: tokens 0..2047
block 1: tokens 1792..3839
...
With the second scheme, the overlap is not merely extra context: if all labels are active, those repeated tokens become training targets multiple times. That changes their effective weighting.
That may occasionally be intentional, but I would not make it the default.
There is another valid design:
[overlapping prefix used only as context][new target tokens]
where labels for the repeated prefix are masked.
Hugging Face’s fixed-length perplexity guide uses exactly this distinction for sliding-window evaluation: overlapping tokens can be supplied as context while the already-accounted-for targets are set to -100.
That does not mean you need sliding-window training here. It just illustrates why:
“using overlap for context”
and
“training twice on every overlapped token”
are different designs.
For a first CPT pipeline, consecutive non-overlapping blocks are much easier to reason about.
#
Should fixed blocks respect chapter or scene boundaries?
I would preserve chapter/scene structure in the canonical corpus, but I would not initially require every training sequence to end exactly at a chapter or scene boundary.
Suppose a chapter is 5,300 tokens and your block size is 2,048.
You could simply get:
block 1: chapter tokens 0..2047
block 2: chapter tokens 2048..4095
block 3: chapter tokens 4096..5299 + following text...
That is a normal fixed-block causal-LM representation.
Trying to force every chapter into its own training sequence can instead produce:
- truncation;
- substantial padding;
- highly variable sequence lengths;
- inefficient packing.
There is legitimate research motivation for preserving document integrity more carefully. For example, Fewer Truncations Improve Language Modeling studies ordinary concatenate-and-split pretraining and proposes Best-fit Packing to reduce unnecessary document truncation while retaining efficient packing.
I would treat that as an advanced comparison, not as evidence that every Korean-novel chapter must be kept intact.
So my order would be:
- preserve all semantic boundaries in your canonical data;
- establish a simple fixed-block baseline;
- only then compare document-aware packing if preserving complete chapters/scenes seems important to your actual evaluation goal.
This also leaves the door open to more specialized long-context training later without making the initial dataset pipeline depend on it.
#
Train/validation splitting: split according to what you want validation to mean
I would avoid randomly generating thousands of adjacent 2048-token blocks first and then assigning those blocks independently to train and validation.
Otherwise neighboring excerpts from the same novel can easily occur on both sides, making validation less independent than it appears.
The correct source boundary depends on what you want to measure.
Hold out complete books first:
books
├─ train books
└─ validation books
then tokenize/pack each split independently
This is conceptually similar to long-form language-model datasets such as PG-19, where complete books are assigned to train, validation, and test partitions.
There is also broader language-model evidence that train/evaluation overlap can distort evaluation; see Deduplicating Training Data Makes Language Models Better. I would not infer from that that your corpus necessarily has a duplication problem—it is simply a good reason not to create avoidable adjacent-text leakage yourself.
Holding out an entire book may remove too much training data.
Then a reasonable compromise can be:
- complete held-out chapters; or
- one or more contiguous held-out spans.
Just interpret that validation correctly: it measures held-out text within the same source/domain, not generalization to a completely unseen book.
A within-book validation split can still be useful for loss monitoring / early stopping. It just answers a narrower question.
So I would choose the split unit from the evaluation goal, rather than treating “book split” as a universal rule.
#
A small Qwen/Transformers padding caveat
There is one implementation detail I would check once if you use generic dynamic padding.
In the current Qwen3-0.6B-Base tokenizer configuration, <|endoftext|> is exposed as both the eos_token and pad_token.
Meanwhile, the current implementation of DataCollatorForLanguageModeling(mlm=False) constructs labels and sets positions whose token ID equals pad_token_id to -100.
That combination matters if <|endoftext|> is also an intentional real EOD inside the sequence: value-based masking cannot distinguish
real <|endoftext|> document boundary
from
<|endoftext|> used as padding
just from the token ID.
This general failure mode has also been discussed in Transformers issue #23530.
So if your pipeline uses that kind of collator, I would inspect one batch and verify:
real EOD position -> label is still the EOD token ID
actual padding position -> label is -100
If you produce completely full fixed-length blocks yourself, the easiest route is to precompute:
labels = input_ids.copy()
and use a simple collator that only stacks the equal-size examples.
If you keep padded tails, mask padding based on the actual attention/padding mask, rather than blindly masking every occurrence of the EOD token ID.
This is a small implementation detail, not a reason to change the overall dataset design.
#
What block size should you actually use?
I do not think there is a generally established answer such as:
Korean novels should use exactly 2048 tokens.
1024, 2048, 4096, etc. are engineering/training choices.
For your goal, I would first measure the corpus with the exact tokenizer:
lengths = [
len(tokenizer(text, add_special_tokens=False)["input_ids"])
for text in ds["text"]
]
print("min:", min(lengths))
print("max:", max(lengths))
print("mean:", sum(lengths) / len(lengths))
Then choose a block length based on:
- available VRAM / training throughput;
- how much local narrative context you want per update;
- total corpus size;
- whether longer sequences materially reduce batch size;
- what you intend to evaluate.
Starting with 1024 or 2048 is perfectly reasonable as a simple baseline; 4096 is also a natural comparison if the training setup handles it comfortably. I would not claim one is intrinsically best without actually comparing the resulting training/evaluation behavior.
Also, do not choose the training length only from one tokenizer metadata number.
At the moment, the HF files for Qwen/Qwen3-0.6B-Base expose:
Those fields are useful configuration information, but neither says that 131,072—or even 32,768—is the sensible sequence length for your particular continued-pretraining run.
“Model can represent a long context” and “this is the most useful/economical training block size for this corpus” are different questions.
#
About the 20,000+ character rows
I would not add an arbitrary character-count limit just to keep Dataset rows short.
A Dataset row can hold an entire book while the training preprocessing later produces many fixed-token examples from it.
This pattern is not unusual in long-form language modeling. For example, PG-19 is explicitly organized around complete books as documents.
So:
Dataset row:
150,000 characters of a book
does not imply:
model forward pass:
all 150,000 characters at once
The latter is controlled by the tokenized training blocks.
For ordinary novels, I would therefore monitor:
number of tokens
number of resulting blocks
remainder/padding
rather than enforcing a fixed maximum number of characters in the canonical Dataset.
Extremely large individual files can eventually become a preprocessing-memory/streaming engineering issue, but that is separate from the model’s sequence-length limit.
#
If by “Hugging Face Dataset” you also mean publishing the text on the Hub
Creating a local datasets.Dataset and publishing the underlying novels to the Hub are separate decisions.
All of the preprocessing above can be done locally.
If you later want to upload the actual novel text to a Dataset repository, check the source/license/redistribution rights first. Hugging Face supports both public and private dataset repositories and provides Dataset-card metadata for documenting things such as license and provenance.
See Sharing a dataset to the Hub.
That does not affect the technical choice between book/chapter rows and token blocks; it is simply a separate distribution/provenance question.
So, if I wanted the least complicated known-good first version, I would probably do this:
Keep original TXT files unchanged
↓
Repair only verified artificial hard wraps
↓
Store one book per row
(or chapter rows + book_id if that is more convenient)
↓
Choose train/validation source units before token chunking
↓
Tokenize raw prose with the exact Qwen tokenizer
↓
Add <|endoftext|> at the chosen independent-document boundary
(probably each book initially)
↓
Pack into consecutive fixed-length blocks
(1024 or 2048 is a reasonable first experiment)
↓
No overlap initially
↓
labels = input_ids for real tokens
↓
Inspect a few decoded blocks, EOD positions, labels, and dropped/padded tail tokens
↓
Train
Then, only if the baseline gives you a reason to do so, I would separately experiment with:
longer block sizes
document-aware packing
chapter/scene-preserving packing
context-only overlap
cross-document attention isolation
That way you do not have to solve every long-context or packing question before you can build a good Dataset.
The main thing I would preserve carefully is the clean canonical corpus and its book/chapter metadata. Once that exists, generating a different 1024/2048/4096-token training representation later is cheap; reconstructing lost paragraph/chapter/book structure from already-chunked training rows is much harder.