How to run flint NVFP4 on 2× RTX 5060 Ti 16 GB

Warning: this is a temporary briefing, written by an LLM and not yet evaluated or rewritten by a human to get rid of the signature AI-slop writing style. It’s published as a draft despite its shortcomings, so that you can find out how to run it and what to expect in terms of pp and tg on this hardware, as well as what to expect in terms of coding performance on this benchmark, grosso modo.

I ran the v5 prompt three times on flint-nvfp4 — a 39.8B mixture-of-experts model with 2.5B active parameters, served at its full 262,144-token native window, layer-split across two RTX 5060 Ti 16 GB cards. This architecture cannot run tensor-split, so the box’s default split did not apply. The vendor card asks for at least 48 GB of memory for this file; this 32 GiB pool serves it with quantized K/V, so part of what you will see below is the 32 GiB budget showing through.

The needle ladder is where the badge breaks. The same unique anchor, planted at every tenth of the window, three repeats per depth, each request opening with a fresh nonce so no step inherited a warm cache:

Eighteen of eighteen through 157,286 tokens, then zero of twelve from 183,501 up. Prefill dipped once at 50 % — KV pressure at this budget — and held flat to the edge; decode never moved. The speed was fine; the memory was the problem. The cliff sits exactly where the vendor’s card says memory should end: flint uses sliding-window attention, where only the recent part of the window is kept at full resolution, and the card instructs leaving --swa-full unset. For reference, a 131k full-attention model on this identical box (LFM2.5-24B) hit 30/30 to the edge of its window.

But past the cliff, it gets worse than a miss. The model does not report “not found”:

• 70 %: claims the marker appears “exactly once as shown” while showing nothing, or calls the document “garbled or corrupted”.
• 80 %: its planning monologue leaks into the answer (“We need answer user. They gave long reference doc…”) and it runs to the cap without producing the marker.
• 90 %: emits the marker’s skeleton (tesseract-XXXX-vault (8) ) instead of its value, then “The final answer is:” and nothing.
• 100 %: degenerates into “Okay, I’ve got the unique marker” repeated until the cap.

An agent that trusts those answers writes bad code confidently. A model that misses can be debugged; a model that fabricates at 70 % of a 262k window cannot.

The runs, for the record: three clean stops, full deliveries, 0/3 pipelines through the compile-and-test gate. Two modules do not compile; the third compiles while its test suite calls a helper that does not exist. Thinking tax ran about 94 % of emitted characters — the heaviest measured in this batch. The review pass was the strongest part of the delivery: three precise resolutions, each with reasoning.

Representative output #

Run 1’s TtlKeyValue — trimmed to the public API and eviction handler (a sliding-window cache, so this is the row that read past 60 % and kept going):

defmodule TtlKeyValue do
  @default_ttl 60_000
  @eviction_interval 60_000

  use GenServer

  @spec start_link(keyword()) :: GenServer.on_start()
  def start_link(opts) when is_list(opts) do
    ttl = Keyword.get(opts, :ttl, @default_ttl)

    if valid_ttl?(ttl) do
      GenServer.start_link(__MODULE__, init, [opts], name: __MODULE__)
    else
      raise ArgumentError, "ttl must be a positive integer or nil"
    end
  end

  @spec put(binary(), binary(), ttl()) :: :ok
  def put(key, value, ttl) do
    validate_key!(key)
    validate_value!(value)
    validate_ttl!(ttl)
    GenServer.call(__MODULE__, {:put, key, value, ttl})
  end

  @spec get(binary(), server()) :: {:ok, binary()} | :error
  def get(key, server) do
    validate_key!(key)
    GenServer.call(server, {:get, key})
  end

  def delete(key, server) do
    validate_key!(key)
    GenServer.call(server, {:delete, key})
  end

  def stats(server) do
    GenServer.call(server, {:stats, :ok})
  end

  def init(opts) do
    ttl = Keyword.get(opts, :ttl, @default_ttl)
    Process.send_after(self(), :evict, @eviction_interval)
    {:ok, %{ttl: ttl, entries: %{}, hits: 0, misses: 0, memory: 0}}
  end

  def handle_call({:get, key}, _from, state) do
    case Map.fetch(state.entries, key) do
      {:ok, {value, expires_at}} ->
        if expires_at != nil and expires_at <= System.monotonic_time(:millisecond) do
          {:reply, :error, %{state | misses: state.misses + 1}}
        else
          {:reply, {:ok, value}, %{state | hits: state.hits + 1}}
        end

      :error ->
        {:reply, :error, %{state | misses: state.misses + 1}}
    end
  end

  def handle_info(:evict, state) do
    now = System.monotonic_time(:millisecond)
    entries = Map.filter(state.entries, fn {_key, {_value, expires_at}} -> expires_at == nil or expires_at > now end)
    Process.send_after(self(), :evict, @eviction_interval)
    {:noreply, %{state | entries: entries}}
  end
end

Plan usable context at ~160 k tokens. Anything above that gets an external retrieval layer. Whether --swa-full or a 48 GB repack moves the cliff is a measured question — it earns a run before the 262 k number gets cited again.