{"slug": "need-advice-on-continued-pretraining-cpt-for-diffusiongemma-or-another-text-for", "title": "Need advice on Continued Pretraining (CPT) for DiffusionGemma or another text diffusion model for domain adaptation", "summary": "A developer is seeking advice on adapting DiffusionGemma, a text diffusion model, to a marine domain dataset. The developer reports that supervised fine-tuning (SFT) caused increased hallucination and degraded instruction-following, and notes that DiffusionGemma lacks a base model, preventing the standard continued pretraining (CPT) to SFT workflow.", "body_md": "**Hi everyone,**\n\nI’m trying to adapt **DiffusionGemma** to a **marine-domain** dataset.\n\nI want to stick with **text diffusion models** because their inference speed is excellent for my use case.\n\nI tried **SFT** on my domain dataset, but the model started hallucinating more and its instruction-following quality degraded.\n\nThe problem is that **DiffusionGemma only has an instruction-tuned checkpoint and no base model**, so I can’t follow the usual **Continued Pretraining (CPT) → SFT** workflow.\n\n**Hardware:**\n\nI’m looking for suggestions:", "url": "https://wpnews.pro/news/need-advice-on-continued-pretraining-cpt-for-diffusiongemma-or-another-text-for", "canonical_source": "https://discuss.huggingface.co/t/need-advice-on-continued-pretraining-cpt-for-diffusiongemma-or-another-text-diffusion-model-for-domain-adaptation/177894#post_1", "published_at": "2026-07-16 09:30:49+00:00", "updated_at": "2026-07-16 09:40:48.566413+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools"], "entities": ["DiffusionGemma"], "alternates": {"html": "https://wpnews.pro/news/need-advice-on-continued-pretraining-cpt-for-diffusiongemma-or-another-text-for", "markdown": "https://wpnews.pro/news/need-advice-on-continued-pretraining-cpt-for-diffusiongemma-or-another-text-for.md", "text": "https://wpnews.pro/news/need-advice-on-continued-pretraining-cpt-for-diffusiongemma-or-another-text-for.txt", "jsonld": "https://wpnews.pro/news/need-advice-on-continued-pretraining-cpt-for-diffusiongemma-or-another-text-for.jsonld"}}