What can silently go wrong between an SFT JSONL file and the loss tensor?

An Apache-2.0 diagnostic package called TrainSight (trainsight-preflight) was released to catch silent fine-tuning failures that occur between raw JSONL training records and the computed loss tensor, including truncation that drops assistant completions entirely, DPO rows with identical chosen and rejected completions (implicit reward margin Δr = 0), and collators that fail to mask prompt positions with labels = -100. TrainSight audits JSONL and Parquet datasets against a chosen tokenizer and chat template via `trainsight check`, and audits compiled PyTorch batches through an SFTReconciler and LossContract API that verifies prompt and padding tokens are masked and each active label position (i ≥ 1) can contribute to the shifted causal-loss objective. The package is positioned as a complement to TRL, Hugging Face transformers, and native PyTorch training loops, making loss-contract checks explicit before and during execution.

When an LLM fine-tuning job crashes with a CUDA OOM or produces NaNs, the failure is obvious. The more insidious problems happen when a run completes cleanly with exit code 0, the loss curve descends predictably, but the loss tensor is not optimizing what you intended. Between raw training records on disk and the computed loss tensor on GPU, a training framework executes a multi-stage transformation: Raw records → tokenizer/template → collator → model forward → shifted labels → CrossEntropyLoss Because PyTorch only requires compatible tensor shapes, data and masking defects can pass without exceptions. Here are three silent failure modes across that boundary—and how to audit them before spending GPU hours. When an instruction sample exceeds max seq len , right-side truncation clips the tail. If a verbose prompt consumes the token budget, the assistant completion can be partially or completely removed. When the completion is fully dropped, every label in the sequence becomes -100 . The sample contributes zero supervised loss while still consuming sequence budget in the batch. Depending on the chat template, empty completions can similarly result in delimiter-only supervision, effectively training the model to emit closing tokens immediately. In Direct Preference Optimization DPO , a row with identical chosen and rejected completions chosen == rejected has an implicit reward margin of zero: \Delta r = 0. Under the deterministic DPO objective, this yields no learning signal. Forward and backward passes still execute, but the pair does not express a preference. Stochastic components such as dropout can introduce residual noise, but do not make such pairs useful preference data. In causal SFT, the intended objective is to condition on the prompt and supervise completion tokens. If a custom collator or template integration fails to mask prompt positions with labels = -100 , cross-entropy is calculated over both prompt and answer tokens. The loss will still decrease, but the objective has changed: the model is also trained to reproduce prompt syntax rather than only generate the intended response. I built TrainSight trainsight-preflight , an Apache-2.0 diagnostic package for validating these training contracts. It works at two explicit checkpoints. TrainSight audits JSONL datasets—and Parquet datasets when pyarrow is installed—against the selected tokenizer and chat template. pip install trainsight-preflight Check an SFT dataset for truncation starvation and empty completions trainsight check data/train sft.jsonl \ --tokenizer Qwen/Qwen2.5-0.5B-Instruct \ --max-seq-len 2048 Check a DPO dataset for identical pairs and length bias trainsight check data/train dpo.jsonl \ --task dpo \ --tokenizer Qwen/Qwen2.5-0.5B-Instruct For custom collators or training pipelines, TrainSight audits compiled PyTorch batches. It checks that prompt and padding tokens are masked, and that each active label position i \ge 1 can contribute to the shifted causal-loss objective. python from transformers import AutoTokenizer from trainsight import SFTReconciler, LossContract tokenizer = AutoTokenizer.from pretrained "Qwen/Qwen2.5-0.5B-Instruct" contract = LossContract.from tokenizer tokenizer, max seq len=512, strict=False, reconciler = SFTReconciler tokenizer=tokenizer, contract=contract, prompt lengths must be derived from the same template/collator path. report = reconciler.audit batch batch=batch, prompt lengths=prompt lengths, row ids=row ids, raw samples=raw samples, assert report.passed, report.diagnoses TrainSight complements existing training frameworks—including TRL, Hugging Face transformers , and native PyTorch loops—by making the relevant loss-contract checks explicit before and during execution. I would value input from practitioners here: 1. What silent data or loss-masking edge cases have you encountered in SFT or DPO runs that standard validation missed? 2. How does your team audit collators and template boundaries before launching expensive multi-GPU or multi-node training jobs?