Hi everyone,
I am currently running RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic using vLLM through Docker.
My current vLLM command is:
vllm serve --model RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic \
--trust-remote-code \
--attention-backend TRITON_ATTN \
--max-num-seqs 4 \
--max-model-len 8192 \
--gpu-memory-utilization 0.78 \
--generation-config vllm \
--hf-overrides '{"diffusion_sampler":"entropy_bound","diffusion_entropy_bound":0.1}' \
--diffusion-config '{"canvas_length":256}' \
--host 0.0.0.0 \
--port 8085
I am using the same model/server in two different applications:
Interestingly, the model works reasonably well in the RAG chatbot. I don’t see noticeable grammar problems or word-merging issues there.
However, in the marine-domain question-generation application, I sometimes get outputs with:
For example, I can get output similar to:
upcomingbunkering operation
instead of:
upcoming bunkering operation
I also see cases where the generated question structure or grammar is not correct.
The important part is that the same model and vLLM server are being used, but the problem is much more noticeable in the question-generation workflow.
I understand that DiffusionGemma is experimental and that its output quality may not be comparable to standard autoregressive Gemma models. However, I am trying to understand whether the behavior I’m seeing is actually a model limitation or whether there is something different in my question-generation pipeline that I am missing.
Could the difference be caused by things such as:
canvas_length=256
entropy_bound
/ diffusion_entropy_bound
Has anyone experienced similar word-merging or grammar issues with DiffusionGemma specifically during question generation or structured output generation?
Any suggestions on what I should compare or test between the RAG and question-generation pipelines would be appreciated. I would particularly like to know whether this is expected behavior from the diffusion generation approach or whether there is a configuration issue I should investigate.