{"slug": "serving-moss-image-step-count-memory-mode-and-the-traps-that-change-the-words", "title": "Serving MOSS-Image: step count, memory mode and the traps that change the words you ship", "summary": "A developer at InferenceKey served OpenMOSS's Apache 2.0 MOSS-Image-SFT model on a single A40 GPU and found that step count and weight placement dominated output quality and latency: the authors' 50-step recipe rendered all text correctly at 78–81 seconds per ~1MP image, while a community 8-step LoRA turbo add-on cut generation to 8.6–9.2 seconds but misspelled two words, and running the add-on at 4 steps broke lettering entirely. Capping GPU memory to 16 GB with the 17 GB Qwen3.5-9B text encoder offloaded to system RAM peaked at about 14 GB and slowed 50-step generation to 86 seconds, while a 12 GB cap pushed it to 160 seconds.", "body_md": "If your product makes menus or posters through an image API, what matters is that the price on the menu is the price you sent. We served [MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT), an Apache 2.0 model from OpenMOSS at MOSI.AI built for long text in pictures, on one A40 on our cloud. Two settings decided almost everything: the step count, and where the weights (the model's learned numbers) sit while it runs.\n\nWith the authors' 50-step recipe, every word was correct on every text test, at 78 to 81 seconds per picture of about one megapixel. A community turbo add-on from lodestones took 8.6 to 9.2 seconds and got two words wrong. The [full MOSS-Image test on one A40](https://inferencekey.com/research/moss-image-hands-on-test) has all 22 pictures and the deploy steps. This piece covers the choices.\n\nThe model begins with random static and removes some of it per step. Each of the 50 steps calls it twice, with your prompt and with an empty one, and the gap between the answers steers the picture toward your words. That is about 100 runs of a 5-billion-parameter network per picture.\n\nThe [lodestones 8-step add-on](https://huggingface.co/lodestones/MOSS-Image-SFT-8-steps-dmd) is a LoRA: a 1.2 GB set of small corrections laid over the original weights, trained to finish in 8 calls with no empty-prompt call. Its card says the correction is smaller than bf16's rounding step (bf16 is the 16-bit format of the weights), so folding it in would erase it. It stays a separate layer at strength 0.7, as in the hugging-apps demo Space. That strength is shared by every request, so our backend (the server that loads the model and answers requests) runs one picture at a time. One loaded copy serves both modes:\n\n```\n{\"prompt\": \"A clean conference poster on a white wall. Large bold title: OPEN WEIGHTS DAY\",\n \"size\": \"1024x1024\", \"seed\": 42, \"mode\": \"turbo\"}\n```\n\nThe body follows OpenAI's image API, plus `mode` (`sft` or `turbo`). The seed fixes the starting static: keep it the same when you compare modes, or you are comparing luck.\n\nWe tried the 8-step add-on at 4 steps. It took 5.2 seconds and the lettering broke.\n\nThe title read \"OPEN WEIGHT DAYDY\" and times looked like \"1(.C0\". The add-on was trained for 8 steps. Adapters built for 4 exist, and we did not test them. At 8 steps, turbo repeated a word in the poster title, wrote \"PNG bock\" for \"PNG back\" and half-garbled street signs, while the chalkboard menu, the Chinese note and both layouts were right. Use it for drafts and proofread each picture.\n\nMOSS-Image is a pair: a 17 GB reader (the Qwen3.5-9B text encoder from the Qwen team, which turns the prompt into numbers) and a 10 GB painter. The reader is busy only briefly per prompt, so it can live in system RAM and move to the GPU on demand. To mimic smaller cards, we limited PyTorch's share of the A40 with `torch.cuda.set_per_process_memory_fraction`:\n\n| Memory cap | Peak on the card | 50 steps | Turbo | \n|---|---|---|---|\n| None (real A40, 48 GB) | about 30 GB | 80 s | 9 s | \n| 16 GB, reader in main memory | about 14 GB | 86 s | 13 s | \n| 12 GB, reader and part of the painter in main memory | about 11 GB | 160 s | 18 s | \n\nThe generic `enable_model_cpu_offload()` from diffusers fit 24 GB but not 16 GB and ran slower than moving only the reader, so our automatic mode never picks it. A cap copies a card's memory size, not its speed, so trust the fit more than the timings. Neither the fit on a real card nor the automatic choice has been tested yet: we set each mode by hand.\n\nThe [memory ladder and the per-picture measurements](https://inferencekey.com/research/moss-image-hands-on-test) list everything. These checks carry over:\n\n`cmp`). If yours differ, something besides placement changed.` nvidia-smi`, NVIDIA's monitor, counts memory the CUDA runtime keeps for itself, which the PyTorch cap ignores. The 16 GB setup leaves about 2 GiB spare, less with a display attached.\nThe authors' [technical report](https://arxiv.org/abs/2610.23374) claims 96.3% average text coverage on their own benchmark, the best among open-weight models. We read only the abstract. Our menu was right in both modes, the kind of prompt where [Iris-3B, the last image model we tested](https://inferencekey.com/news/iris-3b-pixel-space-images), dropped decimal points.\n\nOpenMOSS at MOSI.AI made the model, pipeline, user guide, layout format and report. The Qwen team made the text encoder, Black Forest Labs the FLUX.2 small VAE that decodes the painter's output into pixels. lodestones trained the 8-step add-on and Kromoss, darask0 and akhaliq the 4-step and 3-step adapters. The ComfyUI team added native support, Comfy-Org made quantized repacks, the hugging-apps Space showed how to apply the LoRA. We spotted the model in a video by AI Search. Our part: the backend, the SDK example and the measurements. Deploy commands and open issues are in the [MOSS-Image hands-on test with every timing and memory figure](https://inferencekey.com/research/moss-image-hands-on-test).", "url": "https://wpnews.pro/news/serving-moss-image-step-count-memory-mode-and-the-traps-that-change-the-words", "canonical_source": "https://dev.to/inferencekey/serving-moss-image-step-count-memory-mode-and-the-traps-that-change-the-words-you-ship-4fln", "published_at": "2026-10-30 06:37:39+00:00", "updated_at": "2026-10-30 06:52:54.597369+00:00", "lang": "en", "topics": ["generative-ai", "ai-infrastructure", "ai-tools", "mlops", "large-language-models"], "entities": ["MOSS-Image-SFT", "OpenMOSS", "MOSI.AI", "InferenceKey", "lodestones", "Qwen3.5-9B", "Qwen", "NVIDIA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/serving-moss-image-step-count-memory-mode-and-the-traps-that-change-the-words", "markdown": "https://wpnews.pro/news/serving-moss-image-step-count-memory-mode-and-the-traps-that-change-the-words.md", "text": "https://wpnews.pro/news/serving-moss-image-step-count-memory-mode-and-the-traps-that-change-the-words.txt", "jsonld": "https://wpnews.pro/news/serving-moss-image-step-count-memory-mode-and-the-traps-that-change-the-words.jsonld"}}