Moondream's Photon inference engine recovers between a few percent and a third of its decode throughput by not making the GPU wait for the CPU between tokens. On a B200 at 32 streams the same workload runs 35% faster than a blocking loop; on a 3090, 12%.
The gap it closes is what Moondream calls the GPU bubble. When an autoregressive model generates text it produces one token at a time, each depending on the one before, so decoding is sequential. Every token is a round trip between CPU and GPU. The GPU runs the model's billions of arithmetic operations to produce the next token. The CPU picks which requests to run next, builds the metadata the GPU needs, selects the actual token out of the model's output, and records it. One token of GPU work is small, and the CPU housekeeping is a fixed cost paid on every trip. If the GPU has to wait for that housekeeping before it can start the next token, it sits idle through part of every loop.
Photon hides the idle with pipelined decoding: start the GPU work for the next token while the CPU is still finishing the last one. This works because the token just sampled does not have to leave the GPU. The next forward reads it straight from GPU memory as its input. A copy on the CPU is still wanted eventually, to detokenize, stream, and decide whether the request is done, but that bookkeeping can run a moment later in the background while the next forward is already going. Not waiting on that copy is the move that removes the bubble. Making the overlap safe takes three pieces.
A decode step needs a working set of buffers: staging for the input token and its position, space for the logits the model writes, a landing spot for the sampled token, and the bookkeeping the attention kernel uses to find each sequence's KV cache. Photon allocates these once and reuses them every step, because runtime GPU allocations can force a device synchronization and reintroduce a bubble, and because fixed addresses let the whole step be captured once as a CUDA graph and replayed to cut launch overhead. It calls the bundle a DecodeSlot.
One slot blocks pipelining, because its buffers stay in use until the step finishes, and the next step would overwrite the first step's results before the CPU read them. So Photon keeps two slots and alternates between them. Both slots put their forwards on the same compute stream, so the GPU still runs one forward at a time; the slots exist only so the CPU can process one slot's results while the GPU runs the other slot's forward. The device-to-host copies go on a separate copy stream, each anchored to an event recorded the instant its step's outputs are written, so a copy waits on exactly that step's work and nothing queued behind it. A slot is released only once the commit that reads it has finished, not when the GPU is done with it, because its pinned host buffer is the landing site for a copy that may still be in flight. Handing the slot to a new step early would overwrite a transfer mid-flight and corrupt it.
The next forward can run ahead because it does not depend on anything the CPU does with the last token. Two things about the next step do depend on the last committed result. One is which sequences are still in the batch. The other is which tokens the next step is allowed to sample, and that comes from constrained decoding. Moondream's spatial skills return structured output instead of free text: point returns a coordinate, detect walks an x, y, size cycle, segment returns an outline. Photon gets those from the same decode loop by forcing the logits of disallowed tokens to negative infinity before sampling. Which tokens are allowed at step t+1 depends on the token sampled at t.
That dependency lives in sampling rather than in the forward. So each scheduler tick splits into launch, commit, and finalize: launch the t+1 forward immediately since it does not need the mask, commit step t to advance the request's decode state, then build the mask and sample t+1 with that state current. The GPU runs the t+1 forward through the commit and the sampling, so the commit drops off the critical path. Plain text has no mask, so forward and sampling both run a step ahead; constrained sequences run the forward ahead but hold sampling until the previous commit, which caps how far ahead they get. One loop handles both with no special-casing.
Batch membership takes more care. The batch for t+1 is decided before t commits, so a sequence can hit its stop token at t while already baked into t+1's forward, and you cannot un-launch GPU work. Photon handles this with two per-sequence fields instead of cancellation logic: finalized, set once the sequence hits EOS or its length cap, and inflight_refs, the number of in-flight steps still referencing it. When t commits and detects EOS, the sequence is marked finalized and its result is emitted, but it is not torn down, because t+1 still references it. At t+1's commit it is already finalized, so the commit is skipped: no token appended, no state mutated. The zombie occupied its slot and wrote some KV that nobody reads. Its KV pages and LoRA slot are released only when inflight_refs reaches zero.
Processing a new request's prompt and image is a one-shot forward over many tokens, and Photon launches it as just another kind of work in the same two slots. A prefill forward can run in one slot while a decode step from the other slot is still being committed. The same commit ordering and the same refcounting keep it correct, so none of the zombie or constrained-decode logic needs a prefill special case. This matters most when outputs are short: a request that emits three tokens spends almost all its life in prefill and admission, not decode, so a workload of many short requests is really a stream of prefills with a little decode sprinkled in. Sharing one pipeline lets that stream overlap its own bookkeeping instead of serializing prefill behind decode and back again.
A decode step is three pieces of work. The forward is the heavy GPU matmuls, memory-bandwidth bound at decode because every token streams the whole weight set through the cores, with a floor near weight_bytes / memory_bandwidth. Sampling turns the scores into a committed token: the mask, the argmax or sample, the spatial decode, and the device-to-host copy. Bookkeeping is the CPU work around them, planning the next batch, launching the graph, and committing the previous step. A blocking loop runs the three in series and idles the GPU through the bookkeeping. Pipelining slides the bookkeeping under the next step's forward and sampling, and measured per step on moondream2 the GPU is busy for essentially the whole period, with under 0.05 ms of idle left.
The speedup is T_block / T_pipe times (1 − z). The first term is the win from tucking the bookkeeping under the forward. The z is the zombie tax: launch t+1 before committing t and a finished sequence costs one wasted forward, about 1% at a generation length of 110 tokens on a single stream. In a batch it nearly vanishes, because the zombie is one more row in a step that is already paying full price to stream the weights. The win grows with GPU speed, since the bookkeeping cost does not change as the forward shrinks, so the bubble becomes a larger share of the step. Faster memory and smaller models have the same effect. The practical read for anyone running these engines is that pipelined decoding earns its keep precisely on the newest accelerators and the smallest models, where the fixed CPU cost would otherwise dominate the step.
Photon's pipelined numbers first came out at blocking speed, traced to an accidental synchronous copy while building the constrained-decode mask. Moving that copy to the copy stream was worth 11% on the 3090 and 34% on the B200. A single stray synchronization point in the hot path erases the entire benefit, which is why the gain only appears once every blocking copy has been moved off the critical path.