{"slug": "how-to-run-flint-nvfp4-on-2x-rtx-5060-ti-16-gb", "title": "How to run flint NVFP4 on 2× RTX 5060 Ti 16 GB", "summary": "A 39.8B mixture-of-experts model with 2.5B active parameters, flint-nvfp4, served at its full 262,144-token native window across two RTX 5060 Ti 16 GB cards, retrieved a planted needle in 18 of 18 attempts through 157,286 tokens but in 0 of 12 attempts from 183,501 tokens to the 262,080-token edge, according to a draft benchmark briefing. The briefing attributes the cliff to the 32 GiB memory pool against the vendor's stated 48 GB requirement, with quantized K/V and sliding-window attention, and reports that past the cliff the model fabricated answers rather than reporting a miss — claiming the marker appeared \"exactly once as shown\" while showing nothing at 70%, and repeating \"Okay, I've got the unique marker\" until the cap at 100%. In three coding runs the model produced 0/3 pipelines passing the compile-and-test gate, with thinking tokens accounting for about 94% of emitted characters.", "body_md": "**Warning:** this is a temporary briefing, written by an LLM and not yet evaluated or rewritten by a human to get rid of the signature AI-slop writing style. It’s published as a draft despite its shortcomings, so that you can find out how to run it and what to expect in terms of `pp` and `tg` on this hardware, as well as what to expect in terms of coding performance on this benchmark, *grosso modo*.\n\nI ran the v5 prompt three times on flint-nvfp4 — a 39.8B mixture-of-experts model with 2.5B active parameters, served at its full 262,144-token native window, layer-split across two RTX 5060 Ti 16 GB cards. This architecture cannot run tensor-split, so the box’s default split did not apply. The vendor card asks for at least 48 GB of memory for this file; this 32 GiB pool serves it with quantized K/V, so part of what you will see below is the 32 GiB budget showing through.\n\nThe needle ladder is where the badge breaks. The same unique anchor, planted at every tenth of the window, three repeats per depth, each request opening with a fresh nonce so no step inherited a warm cache:\n\n| depth | hit | pp (median) | tg (median) | \n|---|---|---|---|\n| 10 % (26,214) | 3/3 | 1,741 | 40.2 | \n| 20 % (52,429) | 3/3 | 2,146 | 60.3 | \n| 30 % (78,643) | 3/3 | 1,975 | 49.8 | \n| 40 % (104,858) | 3/3 | 1,817 | 42.1 | \n| 50 % (131,072) | 3/3 | 1,059 | 22.5 | \n| 60 % (157,286) | 3/3 | 1,568 | 32.7 | \n| 70 % (183,501) | 0/3 | 1,060 | 22.5 | \n| 80 % (209,715) | 0/3 | 1,062 | 22.5 | \n| 90 % (235,930) | 0/3 | 1,064 | 22.6 | \n| 100 % (262,080) | 0/3 | 1,226 | 22.5 | \n\nEighteen of eighteen through 157,286 tokens, then zero of twelve from 183,501\nup. Prefill dipped once at 50 % — KV pressure at this budget — and held flat\nto the edge; decode never moved. The speed was fine; the memory was the\nproblem. The cliff sits exactly where the vendor’s card says memory should\nend: flint uses sliding-window attention, where only the recent part of the\nwindow is kept at full resolution, and the card instructs leaving\n`--swa-full` unset. For reference, a 131k full-attention model on this\nidentical box (LFM2.5-24B) hit 30/30 to the edge of its window.\n\nBut past the cliff, it gets worse than a miss. The model does not report “not found”:\n\n- 70 %: claims the marker appears “exactly once as shown” while showing nothing, or calls the document “garbled or corrupted”.\n- 80 %: its planning monologue leaks into the answer (“We need answer user. They gave long reference doc…”) and it runs to the cap without producing the marker.\n- 90 %: emits the marker’s skeleton (`tesseract-XXXX-vault (8)` ) instead of its value, then “The final answer is:” and nothing.\n- 100 %: degenerates into “Okay, I’ve got the unique marker” repeated until the cap.\n\nAn agent that trusts those answers writes bad code confidently. A model that misses can be debugged; a model that fabricates at 70 % of a 262k window cannot.\n\nThe runs, for the record: three clean stops, full deliveries, 0/3 pipelines through the compile-and-test gate. Two modules do not compile; the third compiles while its test suite calls a helper that does not exist. Thinking tax ran about 94 % of emitted characters — the heaviest measured in this batch. The review pass was the strongest part of the delivery: three precise resolutions, each with reasoning.\n\n## Representative output\n\nRun 1’s `TtlKeyValue` — trimmed to the public API and eviction handler (a sliding-window cache, so this is the row that read past 60 % and kept going):\n\n```\ndefmodule TtlKeyValue do\n  @default_ttl 60_000\n  @eviction_interval 60_000\n\n  use GenServer\n\n  @spec start_link(keyword()) :: GenServer.on_start()\n  def start_link(opts) when is_list(opts) do\n    ttl = Keyword.get(opts, :ttl, @default_ttl)\n\n    if valid_ttl?(ttl) do\n      GenServer.start_link(__MODULE__, init, [opts], name: __MODULE__)\n    else\n      raise ArgumentError, \"ttl must be a positive integer or nil\"\n    end\n  end\n\n  @spec put(binary(), binary(), ttl()) :: :ok\n  def put(key, value, ttl) do\n    validate_key!(key)\n    validate_value!(value)\n    validate_ttl!(ttl)\n    GenServer.call(__MODULE__, {:put, key, value, ttl})\n  end\n\n  @spec get(binary(), server()) :: {:ok, binary()} | :error\n  def get(key, server) do\n    validate_key!(key)\n    GenServer.call(server, {:get, key})\n  end\n\n  def delete(key, server) do\n    validate_key!(key)\n    GenServer.call(server, {:delete, key})\n  end\n\n  def stats(server) do\n    GenServer.call(server, {:stats, :ok})\n  end\n\n  def init(opts) do\n    ttl = Keyword.get(opts, :ttl, @default_ttl)\n    Process.send_after(self(), :evict, @eviction_interval)\n    {:ok, %{ttl: ttl, entries: %{}, hits: 0, misses: 0, memory: 0}}\n  end\n\n  def handle_call({:get, key}, _from, state) do\n    case Map.fetch(state.entries, key) do\n      {:ok, {value, expires_at}} ->\n        if expires_at != nil and expires_at <= System.monotonic_time(:millisecond) do\n          {:reply, :error, %{state | misses: state.misses + 1}}\n        else\n          {:reply, {:ok, value}, %{state | hits: state.hits + 1}}\n        end\n\n      :error ->\n        {:reply, :error, %{state | misses: state.misses + 1}}\n    end\n  end\n\n  def handle_info(:evict, state) do\n    now = System.monotonic_time(:millisecond)\n    entries = Map.filter(state.entries, fn {_key, {_value, expires_at}} -> expires_at == nil or expires_at > now end)\n    Process.send_after(self(), :evict, @eviction_interval)\n    {:noreply, %{state | entries: entries}}\n  end\nend\n```\n\nPlan usable context at ~160 k tokens. Anything above that gets an external\nretrieval layer. Whether `--swa-full` or a 48 GB repack moves the cliff is a\nmeasured question — it earns a run before the 262 k number gets cited again.", "url": "https://wpnews.pro/news/how-to-run-flint-nvfp4-on-2x-rtx-5060-ti-16-gb", "canonical_source": "https://overbring.com/ai-benchmarks/2026-10-31-recursive-si-flint-nvfp4-gguf-rtx-5060-ti-16gb-x2-layer-split-llama-server-cuda/", "published_at": "2026-10-31 04:11:25+00:00", "updated_at": "2026-11-09 01:58:36.423419+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research"], "entities": ["flint-nvfp4", "RTX 5060 Ti 16 GB", "LFM2.5-24B", "TtlKeyValue"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-run-flint-nvfp4-on-2x-rtx-5060-ti-16-gb", "markdown": "https://wpnews.pro/news/how-to-run-flint-nvfp4-on-2x-rtx-5060-ti-16-gb.md", "text": "https://wpnews.pro/news/how-to-run-flint-nvfp4-on-2x-rtx-5060-ti-16-gb.txt", "jsonld": "https://wpnews.pro/news/how-to-run-flint-nvfp4-on-2x-rtx-5060-ti-16-gb.jsonld"}}