{"slug": "2-years-and-i-have-a-small-ai-server-how-do-i-benchmark-whats-useful-to-know", "title": "2 years and I have a small AI server | how do I benchmark? Whats useful to know?", "summary": "A user running a two-year-old dual Tesla V100 AI server reported that the Strata-V100 fork (commit 7de1934, engine 0.1.45.6) crashes with an \"illegal memory access\" error during prefill of Qwen3.8-Flash-Next at Q2_0 quantization with 262,144 max context, breaking at roughly 1,700-2,575 tokens while small prompts decode at 56-69 tokens per second. The user ruled out Resizable BAR, which lspci shows is unsupported on the Volta-based V100 cards, and Nvidia passthrough, since both 32GB Tesla PG500-216 GPUs run natively on the Proxmox host rather than in a VM. For comparison, the same stack on llama.cpp reached 356 prefill / 26 decode tokens per second at 32k context, 324 / 28 at 64k, and 272 / 25 at 128k.", "body_md": "So I forwarded your reply to Muse, this is what Muse replied with:\n\n**Q: Can you post the link to the strata fork you’re using?**\n\nA: [GitHub - jmnargi/Strata-V100: Strata port for Tesla V100 (sm_70): Qwen3.8-Flash-Next with 262K context and MTP · GitHub](https://github.com/jmnargi/Strata-V100)\n\nMy install: commit 7de1934, engine 0.1.45.6, CUDA 12.9.1, Qwen3.8-Flash-Next Q2_0, 36G + 27G shards, max-context 262144, KV int8, spec 4, MTP on, port 8081, GPUs [0,1]\n\n**Q: Is Resizable BAR on in the BIOS?**\n\nA: No, and it’s not available on these cards, checked from the OS, not BIOS guesswork.\n\nCommands used: `lspci -s 04:00.0 -v`, `lspci -s 08:00.0 -v`, `nvidia-smi`\n\n- No Resizable BAR capability listed in lspci for either card, V100 / Volta doesn’t have consumer ReBAR like a 5060 Ti / Blackwell\n- Full memory is mapped correctly: GPU 0 Memory at 7000000000 [size=32G], GPU 1 Memory at 6000000000 [size=32G], kernel driver in use: nvidia, both Tesla PG500-216, 32GB\n- So BAR is not screwy, and not the cause of the memory errors in my OP\n**Q: Nvidia passthrough funky?**\n\nA: Not applicable here. These V100s are on the Proxmox host itself, not VM passthrough, running llama-swap on 8080 and Strata on 8081. No VM restart / reboot-the-host passthrough issue.\n\n**My Strata crash, direct:**\n\n“prefill: the PLE rows’ upload failed: an illegal memory access was encountered”\n\n“output became non-finite (NaN) twice, engine dropped cached state”\n\nLayer 0, chunk 0, quantize_scatter, break at about 1,700-2,575 tokens. Small prompts work, 56-69 decode, larger crash in about 0.5s with 0 tokens.\n\n**Constraints that shaped what I did / didn’t do:**\n\n- No Docker for my product / setup, so I did NOT use Strata’s Docker setup, I built / ran it natively on the host, hand-built first. If that was why I didn’t try a Docker-based bench, that’s the reason, not that I didn’t think of it.\n- I also didn’t run llm-bench-dot-io, because it always uploads results to their site, and I don’t want private code / results uploaded.\n- Smoke tests at 128 / 512 tokens don’t help me, Hermes eats about 30k once it’s loaded my large codebase + tooling, so I need 32k / 64k / 128k benchmarks with real code tasks, not small prompts.\nMy llama.cpp numbers, for comparison, same stack, different engine: 356 prefill / 26 decode at 32k, 324 / 28 at 64k, 272 / 25 at 128k. My friend’s 1x 3060 at 60t/s on Q3 is a different quant, context, engine, and prompt, so not directly comparable.\n\nI hope this helps :C sometimes it tries to talk in as if replying as me and gets mixed up on those type of details.", "url": "https://wpnews.pro/news/2-years-and-i-have-a-small-ai-server-how-do-i-benchmark-whats-useful-to-know", "canonical_source": "https://forum.level1techs.com/t/2-years-and-i-have-a-small-ai-server-how-do-i-benchmark-whats-useful-to-know/259951#post_7", "published_at": "2026-11-04 20:35:12+00:00", "updated_at": "2026-11-04 20:50:36.279347+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools"], "entities": ["Strata-V100", "Tesla V100", "Qwen3.8-Flash-Next", "llama.cpp", "Proxmox", "Nvidia", "llama-swap", "jmnargi"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/2-years-and-i-have-a-small-ai-server-how-do-i-benchmark-whats-useful-to-know", "markdown": "https://wpnews.pro/news/2-years-and-i-have-a-small-ai-server-how-do-i-benchmark-whats-useful-to-know.md", "text": "https://wpnews.pro/news/2-years-and-i-have-a-small-ai-server-how-do-i-benchmark-whats-useful-to-know.txt", "jsonld": "https://wpnews.pro/news/2-years-and-i-have-a-small-ai-server-how-do-i-benchmark-whats-useful-to-know.jsonld"}}