{"slug": "i-was-fine-tuning-a-language-model-on-arabic-the-loss-was-perfect-it-spoke", "title": "I was fine-tuning a language model on Arabic. The loss was perfect. It spoke Chinese.", "summary": "A developer built Trainsafe, an open-source tool that monitors language model fine-tuning in real time, after a training run on Arabic unexpectedly produced Chinese outputs. The tool plugs into HuggingFace or TRL pipelines with two lines of code and checks for language drift, output collapse, repetition, echo, and format consistency at every evaluation checkpoint. Trainsafe can stop training and recommend the last healthy checkpoint if the overall health score drops below a threshold.", "body_md": "Repo: [github.com/AmmarHassona/trainsafe](https://github.com/AmmarHassona/trainsafe)\n\nI was working on fine-tuning an open-source small language model (SLM) on Arabic using DPO. I had the data, the pipeline, and everything set up for training. I was fairly confident that this training run would improve the model and align it further to what I wanted. I started the training and let it run until it finished. When I came back to test the checkpoint, it was speaking Chinese.\n\nLoss only tells you the model is learning *something* — not what it's actually *learning*. By the time training finished, I had wasted my time and my compute with nothing useful to show for it. If only there was something to tell me if training was actually going well before it was too late.\n\nThis is when I began looking at tools that could help me solve this issue. Nothing existed that did exactly what I needed, so I built it myself. I built trainsafe to plug into any HuggingFace or TRL training pipeline with two lines of code. It runs alongside your training and checks whether the model's outputs are still behaving correctly at every eval checkpoint — catching issues like language drift, output collapse, and repetition loops before the run finishes.\n\nInstall with pip:\n\n```\npip install trainsafe\n\n# with language drift detection\npip install \"trainsafe[language]\"\n```\n\nThen add it to your training script with no other changes needed:\n\n``` python\nfrom trainsafe import TrainSafeCallback\n\ntrainer = SFTTrainer(\n    model=model,\n    ...\n    callbacks=[TrainSafeCallback()]\n)\ntrainer.train()\n```\n\nAt every eval checkpoint, trainsafe generates a small sample of outputs and runs five checks automatically:\n\nLanguage — detects if the model switches output language mid-training. This is exactly what would have caught my situation.\n\nLength — catches output collapse (model suddenly generating much shorter text) or runaway growth. Compares against a rolling baseline so legitimate learning doesn't trigger false alarms.\n\nRepetition — flags n-gram loops inside individual outputs, the classic \"the the the the\" failure mode.\n\nEcho — flags outputs that are mostly a copy of the prompt rather than an actual response.\n\nFormat — detects if a model trained to output JSON starts producing plain text, or vice versa.\n\nAll five run with zero configuration. If the overall health score drops below the warning threshold, you get a warning. If it drops below the stop threshold, training stops and trainsafe points you at the last healthy checkpoint.\n\nHealthy run\n\n```\n[TrainSafe @ step 5] ✅ Language consistent (en)\n[TrainSafe @ step 5] ✅ Output length normal (avg 62 words)\n[TrainSafe @ step 5] ✅ No repetition detected\n[TrainSafe @ step 5] ✅ No prompt echoing\n[TrainSafe @ step 5] ✅ Format consistent (plain)\n[TrainSafe @ step 5] Overall health: 1.00\n```\n\nWhen something goes wrong\n\n```\n[TrainSafe @ step 600] 🚨 Language drift — expected ar, got zh\n[TrainSafe @ step 600] 🚨 Output length collapsed (avg 3 words vs baseline 87)\n[TrainSafe @ step 600] ⚠️  Repetition detected in 3/5 outputs\n[TrainSafe @ step 600] Overall health: 0.20\n>>> TrainSafe stopped training. Recommended checkpoint: step 400.\n```\n\nIf you have a specific capability you can't afford to lose, you can define fixed prompts and expected behaviors in a YAML file:\n\n```\nprobes:\n  - prompt: \"مرحبا، كيف يمكنني مساعدتك؟\"\n    checks:\n      - language: ar\n      - min_length: 10\n      - not_contains: [\"<|im_start|>\", \"###\"]\n```\n\nThese run at every checkpoint alongside the automatic checks.\n\ntrainsafe is MIT licensed, early stage, and feedback is very welcome. If you've hit a similar problem during fine-tuning I'd love to hear about it in the comments.", "url": "https://wpnews.pro/news/i-was-fine-tuning-a-language-model-on-arabic-the-loss-was-perfect-it-spoke", "canonical_source": "https://dev.to/ammarhassona/i-was-fine-tuning-a-language-model-on-arabic-the-loss-was-perfect-it-spoke-chinese-5ln", "published_at": "2026-06-14 13:11:48+00:00", "updated_at": "2026-06-14 13:40:42.370322+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "developer-tools", "mlops"], "entities": ["Trainsafe", "HuggingFace", "TRL", "AmmarHassona", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/i-was-fine-tuning-a-language-model-on-arabic-the-loss-was-perfect-it-spoke", "markdown": "https://wpnews.pro/news/i-was-fine-tuning-a-language-model-on-arabic-the-loss-was-perfect-it-spoke.md", "text": "https://wpnews.pro/news/i-was-fine-tuning-a-language-model-on-arabic-the-loss-was-perfect-it-spoke.txt", "jsonld": "https://wpnews.pro/news/i-was-fine-tuning-a-language-model-on-arabic-the-loss-was-perfect-it-spoke.jsonld"}}