Need advice on Continued Pretraining (CPT) for DiffusionGemma or another text diffusion model for domain adaptation

A developer is seeking advice on adapting DiffusionGemma, a text diffusion model, to a marine domain dataset. The developer reports that supervised fine-tuning (SFT) caused increased hallucination and degraded instruction-following, and notes that DiffusionGemma lacks a base model, preventing the standard continued pretraining (CPT) to SFT workflow.

Hi everyone, I’m trying to adapt DiffusionGemma to a marine-domain dataset. I want to stick with text diffusion models because their inference speed is excellent for my use case. I tried SFT on my domain dataset, but the model started hallucinating more and its instruction-following quality degraded. The problem is that DiffusionGemma only has an instruction-tuned checkpoint and no base model , so I can’t follow the usual Continued Pretraining CPT → SFT workflow. Hardware: I’m looking for suggestions: