So I forwarded your reply to Muse, this is what Muse replied with:
Q: Can you post the link to the strata fork you’re using?
A: GitHub - jmnargi/Strata-V100: Strata port for Tesla V100 (sm_70): Qwen3.8-Flash-Next with 262K context and MTP · GitHub My install: commit 7de1934, engine 0.1.45.6, CUDA 12.9.1, Qwen3.8-Flash-Next Q2_0, 36G + 27G shards, max-context 262144, KV int8, spec 4, MTP on, port 8081, GPUs [0,1]
Q: Is Resizable BAR on in the BIOS?
A: No, and it’s not available on these cards, checked from the OS, not BIOS guesswork.
Commands used: lspci -s 04:00.0 -v, lspci -s 08:00.0 -v, nvidia-smi
- No Resizable BAR capability listed in lspci for either card, V100 / Volta doesn’t have consumer ReBAR like a 5060 Ti / Blackwell
- Full memory is mapped correctly: GPU 0 Memory at 7000000000 [size=32G], GPU 1 Memory at 6000000000 [size=32G], kernel driver in use: nvidia, both Tesla PG500-216, 32GB
- So BAR is not screwy, and not the cause of the memory errors in my OP Q: Nvidia passthrough funky?
A: Not applicable here. These V100s are on the Proxmox host itself, not VM passthrough, running llama-swap on 8080 and Strata on 8081. No VM restart / reboot-the-host passthrough issue.
My Strata crash, direct:
“prefill: the PLE rows’ upload failed: an illegal memory access was encountered”
“output became non-finite (NaN) twice, engine dropped cached state”
Layer 0, chunk 0, quantize_scatter, break at about 1,700-2,575 tokens. Small prompts work, 56-69 decode, larger crash in about 0.5s with 0 tokens.
Constraints that shaped what I did / didn’t do:
- No Docker for my product / setup, so I did NOT use Strata’s Docker setup, I built / ran it natively on the host, hand-built first. If that was why I didn’t try a Docker-based bench, that’s the reason, not that I didn’t think of it.
- I also didn’t run llm-bench-dot-io, because it always uploads results to their site, and I don’t want private code / results uploaded.
- Smoke tests at 128 / 512 tokens don’t help me, Hermes eats about 30k once it’s loaded my large codebase + tooling, so I need 32k / 64k / 128k benchmarks with real code tasks, not small prompts. My llama.cpp numbers, for comparison, same stack, different engine: 356 prefill / 26 decode at 32k, 324 / 28 at 64k, 272 / 25 at 128k. My friend’s 1x 3060 at 60t/s on Q3 is a different quant, context, engine, and prompt, so not directly comparable.
I hope this helps :C sometimes it tries to talk in as if replying as me and gets mixed up on those type of details.