# Need advice on Continued Pretraining (CPT) for DiffusionGemma or another text diffusion model for domain adaptation

> Source: <https://discuss.huggingface.co/t/need-advice-on-continued-pretraining-cpt-for-diffusiongemma-or-another-text-diffusion-model-for-domain-adaptation/177894#post_1>
> Published: 2026-07-16 09:30:49+00:00

**Hi everyone,**

I’m trying to adapt **DiffusionGemma** to a **marine-domain** dataset.

I want to stick with **text diffusion models** because their inference speed is excellent for my use case.

I tried **SFT** on my domain dataset, but the model started hallucinating more and its instruction-following quality degraded.

The problem is that **DiffusionGemma only has an instruction-tuned checkpoint and no base model**, so I can’t follow the usual **Continued Pretraining (CPT) → SFT** workflow.

**Hardware:**

I’m looking for suggestions: