How to run flint NVFP4 on 2× RTX 5060 Ti 16 GB

A 39.8B mixture-of-experts model with 2.5B active parameters, flint-nvfp4, served at its full 262,144-token native window across two RTX 5060 Ti 16 GB cards, retrieved a planted needle in 18 of 18 attempts through 157,286 tokens but in 0 of 12 attempts from 183,501 tokens to the 262,080-token edge, according to a draft benchmark briefing. The briefing attributes the cliff to the 32 GiB memory pool against the vendor's stated 48 GB requirement, with quantized K/V and sliding-window attention, and reports that past the cliff the model fabricated answers rather than reporting a miss — claiming the marker appeared "exactly once as shown" while showing nothing at 70%, and repeating "Okay, I've got the unique marker" until the cap at 100%. In three coding runs the model produced 0/3 pipelines passing the compile-and-test gate, with thinking tokens accounting for about 94% of emitted characters.

Warning: this is a temporary briefing, written by an LLM and not yet evaluated or rewritten by a human to get rid of the signature AI-slop writing style. It’s published as a draft despite its shortcomings, so that you can find out how to run it and what to expect in terms of pp and tg on this hardware, as well as what to expect in terms of coding performance on this benchmark, grosso modo . I ran the v5 prompt three times on flint-nvfp4 — a 39.8B mixture-of-experts model with 2.5B active parameters, served at its full 262,144-token native window, layer-split across two RTX 5060 Ti 16 GB cards. This architecture cannot run tensor-split, so the box’s default split did not apply. The vendor card asks for at least 48 GB of memory for this file; this 32 GiB pool serves it with quantized K/V, so part of what you will see below is the 32 GiB budget showing through. The needle ladder is where the badge breaks. The same unique anchor, planted at every tenth of the window, three repeats per depth, each request opening with a fresh nonce so no step inherited a warm cache: | depth | hit | pp median | tg median | |---|---|---|---| | 10 % 26,214 | 3/3 | 1,741 | 40.2 | | 20 % 52,429 | 3/3 | 2,146 | 60.3 | | 30 % 78,643 | 3/3 | 1,975 | 49.8 | | 40 % 104,858 | 3/3 | 1,817 | 42.1 | | 50 % 131,072 | 3/3 | 1,059 | 22.5 | | 60 % 157,286 | 3/3 | 1,568 | 32.7 | | 70 % 183,501 | 0/3 | 1,060 | 22.5 | | 80 % 209,715 | 0/3 | 1,062 | 22.5 | | 90 % 235,930 | 0/3 | 1,064 | 22.6 | | 100 % 262,080 | 0/3 | 1,226 | 22.5 | Eighteen of eighteen through 157,286 tokens, then zero of twelve from 183,501 up. Prefill dipped once at 50 % — KV pressure at this budget — and held flat to the edge; decode never moved. The speed was fine; the memory was the problem. The cliff sits exactly where the vendor’s card says memory should end: flint uses sliding-window attention, where only the recent part of the window is kept at full resolution, and the card instructs leaving --swa-full unset. For reference, a 131k full-attention model on this identical box LFM2.5-24B hit 30/30 to the edge of its window. But past the cliff, it gets worse than a miss. The model does not report “not found”: - 70 %: claims the marker appears “exactly once as shown” while showing nothing, or calls the document “garbled or corrupted”. - 80 %: its planning monologue leaks into the answer “We need answer user. They gave long reference doc…” and it runs to the cap without producing the marker. - 90 %: emits the marker’s skeleton tesseract-XXXX-vault 8 instead of its value, then “The final answer is:” and nothing. - 100 %: degenerates into “Okay, I’ve got the unique marker” repeated until the cap. An agent that trusts those answers writes bad code confidently. A model that misses can be debugged; a model that fabricates at 70 % of a 262k window cannot. The runs, for the record: three clean stops, full deliveries, 0/3 pipelines through the compile-and-test gate. Two modules do not compile; the third compiles while its test suite calls a helper that does not exist. Thinking tax ran about 94 % of emitted characters — the heaviest measured in this batch. The review pass was the strongest part of the delivery: three precise resolutions, each with reasoning. Representative output Run 1’s TtlKeyValue — trimmed to the public API and eviction handler a sliding-window cache, so this is the row that read past 60 % and kept going : defmodule TtlKeyValue do @default ttl 60 000 @eviction interval 60 000 use GenServer @spec start link keyword :: GenServer.on start def start link opts when is list opts do ttl = Keyword.get opts, :ttl, @default ttl if valid ttl? ttl do GenServer.start link MODULE , init, opts , name: MODULE else raise ArgumentError, "ttl must be a positive integer or nil" end end @spec put binary , binary , ttl :: :ok def put key, value, ttl do validate key key validate value value validate ttl ttl GenServer.call MODULE , {:put, key, value, ttl} end @spec get binary , server :: {:ok, binary } | :error def get key, server do validate key key GenServer.call server, {:get, key} end def delete key, server do validate key key GenServer.call server, {:delete, key} end def stats server do GenServer.call server, {:stats, :ok} end def init opts do ttl = Keyword.get opts, :ttl, @default ttl Process.send after self , :evict, @eviction interval {:ok, %{ttl: ttl, entries: %{}, hits: 0, misses: 0, memory: 0}} end def handle call {:get, key}, from, state do case Map.fetch state.entries, key do {:ok, {value, expires at}} - if expires at = nil and expires at <= System.monotonic time :millisecond do {:reply, :error, %{state | misses: state.misses + 1}} else {:reply, {:ok, value}, %{state | hits: state.hits + 1}} end :error - {:reply, :error, %{state | misses: state.misses + 1}} end end def handle info :evict, state do now = System.monotonic time :millisecond entries = Map.filter state.entries, fn { key, { value, expires at}} - expires at == nil or expires at now end Process.send after self , :evict, @eviction interval {:noreply, %{state | entries: entries}} end end Plan usable context at ~160 k tokens. Anything above that gets an external retrieval layer. Whether --swa-full or a 48 GB repack moves the cliff is a measured question — it earns a run before the 262 k number gets cited again.