# Poor(er) man's local AI

> Source: <https://forum.level1techs.com/t/poor-er-mans-local-ai/252071#post_8>
> Published: 2026-10-25 18:18:19+00:00

Rinse and repeat

I’ve been testing “smaller” local models to see how they stack up as coding assistants on small to mid size projects.

When switching from Qwen Coder Next to DeepSeek v4 Flash (alpha) I was genuinely impressed. Decided to throw away my first prototype and start over from scratch (with lessons learned). A couple of weeks in, there were definitely some cracks that started to show, started thinking about a Kimi K3 subscription to set everything straight after every couple of features.

With the Flash 0731 revision, however, it seemed to be able to handle the role I was considering Kimi for by itself. In the end, I only turned to Kimi K3 for the alpha release reviews.

I created a separate thread for the [OpenFromSource](https://forum.level1techs.com/t/openfromsource/259444) project.

When I first tried to serve the [native model](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) with vLLM on Ampere generation, I got the following error:

```
ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")
```

Makes sense, since most of the model weights are in an unsupported format on Ampere. There are a couple of ways around this, the one that had struck my *attention* was the [GGUF model](https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF) from the Unsloth brothers.

Ref [DeepSeek-V4: How to Run Locally | Unsloth Documentation](https://unsloth.ai/docs/models/deepseek-v4#quantization-analysis)

Our UD-Q8_K_XL quant is fully lossless. DeepSeek-V4-Flash is quantization-aware-trained: the official checkpoint stores its routed experts (96% of the model) natively in MXFP4 and everything else in FP8 or BF16. GGUF’s MXFP4 is exactly that format,

so we repack the experts bit-for-bit, and FP8 dequantizes into BF16 with no rounding. We checked every tensor against the official DeepSeek weights: all 1,328 are bit-identical, and it stays lossless at inference (KL-divergence ~0, 100% top-token agreement).

Of course, it was only officially supported on Llama.cpp (and Unsloth studio). Meant stepping away from my trusty vLLM setup. Getting 30 t/s out of the box was quite a step back, and with [dspark](https://unsloth.ai/docs/models/deepseek-v4#dspark-speculative-decoding) I could barely hold everything in VRAM, but hey, I’ll take it.

The GPUs are heavily underutilized.

We’re missing Tensor Parallelism from vLLM. There’s [an experimental feature on Llama.cpp](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md#4-tensor-parallelism-experimental) that backs the same concept, but it wasn’t supported for Deepseek v4 (yet).

```
Loading model... \0.01.719.375 E llama_model_load: error loading model: LLAMA_SPLIT_MODE_TENSOR not implemented for architecture 'deepseek4'
```

The occasional full prompt/context re-processing.

Which could easily take 5-10min.

The correct term is KV cache eviction. The server drops the cached key-value tensors (VRAM pressure, slot rotation, or a changed prompt prefix), so the next request misses the cache and must re-prefill (recompute) the entire context.

To be honest, this was the first time I could see myself adding feature after feature and actually make it to the finish line. The quality was there, together with what for me counts as long(er)-horizon capabilities.

Kept a close eye on [the vLLM SM8x issue tracker](https://github.com/vllm-project/vllm/issues/50576). Once the work matured,

decided to give the following fork a shot:

Apart from doubling the token generation speed, which was welcome, the behavior was once again reliable (no occasional re-processing). Overall it felt good moving back to vLLM after spending approximately a month on Llama.cpp.

Before committing to anything, try to get some actual work done with your model of choice. Don’t just select 4 models, run a benchmark (and/or one-shot) and call it a day. Whether you use it for coding, drafting legal documents or interior remodeling, try sticking with a model for an extended project so you can get that iterative cycle, where technical debt typically builds up. With last gen open source models, the first week(s) won’t uncover much. It’s how they hold up in the long run that matters.