I was fine-tuning a language model on Arabic. The loss was perfect. It spoke Chinese.

A developer built Trainsafe, an open-source tool that monitors language model fine-tuning in real time, after a training run on Arabic unexpectedly produced Chinese outputs. The tool plugs into HuggingFace or TRL pipelines with two lines of code and checks for language drift, output collapse, repetition, echo, and format consistency at every evaluation checkpoint. Trainsafe can stop training and recommend the last healthy checkpoint if the overall health score drops below a threshold.

Repo: github.com/AmmarHassona/trainsafe https://github.com/AmmarHassona/trainsafe I was working on fine-tuning an open-source small language model SLM on Arabic using DPO. I had the data, the pipeline, and everything set up for training. I was fairly confident that this training run would improve the model and align it further to what I wanted. I started the training and let it run until it finished. When I came back to test the checkpoint, it was speaking Chinese. Loss only tells you the model is learning something — not what it's actually learning . By the time training finished, I had wasted my time and my compute with nothing useful to show for it. If only there was something to tell me if training was actually going well before it was too late. This is when I began looking at tools that could help me solve this issue. Nothing existed that did exactly what I needed, so I built it myself. I built trainsafe to plug into any HuggingFace or TRL training pipeline with two lines of code. It runs alongside your training and checks whether the model's outputs are still behaving correctly at every eval checkpoint — catching issues like language drift, output collapse, and repetition loops before the run finishes. Install with pip: pip install trainsafe with language drift detection pip install "trainsafe language " Then add it to your training script with no other changes needed: python from trainsafe import TrainSafeCallback trainer = SFTTrainer model=model, ... callbacks= TrainSafeCallback trainer.train At every eval checkpoint, trainsafe generates a small sample of outputs and runs five checks automatically: Language — detects if the model switches output language mid-training. This is exactly what would have caught my situation. Length — catches output collapse model suddenly generating much shorter text or runaway growth. Compares against a rolling baseline so legitimate learning doesn't trigger false alarms. Repetition — flags n-gram loops inside individual outputs, the classic "the the the the" failure mode. Echo — flags outputs that are mostly a copy of the prompt rather than an actual response. Format — detects if a model trained to output JSON starts producing plain text, or vice versa. All five run with zero configuration. If the overall health score drops below the warning threshold, you get a warning. If it drops below the stop threshold, training stops and trainsafe points you at the last healthy checkpoint. Healthy run TrainSafe @ step 5 ✅ Language consistent en TrainSafe @ step 5 ✅ Output length normal avg 62 words TrainSafe @ step 5 ✅ No repetition detected TrainSafe @ step 5 ✅ No prompt echoing TrainSafe @ step 5 ✅ Format consistent plain TrainSafe @ step 5 Overall health: 1.00 When something goes wrong TrainSafe @ step 600 🚨 Language drift — expected ar, got zh TrainSafe @ step 600 🚨 Output length collapsed avg 3 words vs baseline 87 TrainSafe @ step 600 ⚠️ Repetition detected in 3/5 outputs TrainSafe @ step 600 Overall health: 0.20 TrainSafe stopped training. Recommended checkpoint: step 400. If you have a specific capability you can't afford to lose, you can define fixed prompts and expected behaviors in a YAML file: probes: - prompt: "مرحبا، كيف يمكنني مساعدتك؟" checks: - language: ar - min length: 10 - not contains: "<|im start| ", " " These run at every checkpoint alongside the automatic checks. trainsafe is MIT licensed, early stage, and feedback is very welcome. If you've hit a similar problem during fine-tuning I'd love to hear about it in the comments.