Need advice on Continued Pretraining (CPT) for DiffusionGemma or another text diffusion model for domain adaptation

Hi everyone,

I’m trying to adapt DiffusionGemma to a marine-domain dataset.

I want to stick with text diffusion models because their inference speed is excellent for my use case.

I tried SFT on my domain dataset, but the model started hallucinating more and its instruction-following quality degraded.

The problem is that DiffusionGemma only has an instruction-tuned checkpoint and no base model, so I can’t follow the usual Continued Pretraining (CPT) → SFT workflow.

Hardware:

I’m looking for suggestions: