{"slug": "jan-v0-8-6-hands-on-running-local-llms-on-windows", "title": "Jan v0.8.6 Hands-On: Running Local LLMs on Windows", "summary": "StorageReview published a hands-on review of Jan v0.8.6, an open-source desktop app that runs local large language models through llama.cpp, tested on an HP Z4 Rack G5 workstation with Windows 11 Pro for Workstations, an Intel Xeon w5-2465X, 128GB of RAM, and an NVIDIA RTX 5000 Ada Generation GPU with 32GB of VRAM. Jan detected the hardware automatically and enabled GPU acceleration without additional CUDA setup, and the reviewer reran all performance tests after v0.8.5 and v0.8.6 arrived shortly before publication, bundling llama.cpp into Jan, changing GPU offload configuration, adjusting parallel-sequence defaults, and adding conversation prompt caching. The app required no account for local use, offered a 2.8GB Jan V3.5 recommended model download, and its Hub lists models from Hugging Face with download size, quantization, capability tags, and a hardware-fit indicator, including a 2.5GB Q4_K_M Jan V3.5 4B and a 4.7GB Jan V2 VL High with Multimodal and Tools support.", "body_md": "Running large language models locally has become much easier over the past couple of years, but a gap remains between downloading a model and having a useful desktop environment for it. Jan describes itself as a ChatGPT alternative, offering an open-source desktop app with a familiar chat-style interface that keeps most of the setup inside the application. It can run downloaded models locally through llama.cpp while also connecting to hosted AI services, so you can use local and cloud-based models from the same interface.\n\nThis hands-on is also the first guide tied to our [Best Local LLM Tools leaderboard](https://www.storagereview.com/best/local-llm-tools), where we are adding repeatable performance results from our own test hardware as we work through the platforms on the list.\n\nBesides basic local chat, Jan includes a built-in model Hub, web search and page fetching, image input, MCP integrations, and an OpenAI-compatible local API.\n\nFor this hands-on look, we installed Jan from scratch on an [HP Z4 Rack G5](https://www.storagereview.com/review/hp-z4-rack-g5-review) workstation with Windows 11 Pro for Workstations, equipped with an Intel Xeon w5-2465X, 128GB of RAM, and an NVIDIA RTX 5000 Ada Generation GPU with 32GB of VRAM, to see how the setup process works, how Jan manages local models, whether GPU acceleration works automatically, and how its main features behave once everything is running.\n\nJan changed between our first pass and publication, with v0.8.5 and v0.8.6 arriving shortly before this review went live. The newer releases address several areas from the earlier build, including bundling llama.cpp directly into Jan, changing how GPU offload is configured, adjusting parallel-sequence defaults, and adding conversation prompt caching. Because those changes affect local inference, we updated to v0.8.6 and reran all of our performance tests before publishing.\n\n## Test System\n\n| Component | Configuration | \n|---|---|\n| **Operating System** | Windows 11 Pro for Workstations | \n| **CPU** | Intel Xeon w5-2465X, 16 cores | \n| **Memory** | 128GB | \n| **GPU** | NVIDIA RTX 5000 Ada Generation | \n| **VRAM** | 32GB | \n| **NVIDIA Driver** | 597.06 | \n| **CUDA Compatibility** | 13.2 (nvidia-smi) | \n| **Jan** | v0.8.6 | \n| **llama.cpp** | 0.5.0, build b11146 | \n\nJan detected the system hardware correctly through its Hardware settings page, including the CPU, system memory, and RTX 5000 Ada. On this system, which already had NVIDIA drivers installed, Jan enabled GPU acceleration without requiring any additional CUDA setup inside the application before we loaded a model.\n\n## Installing Jan and Getting Started\n\nThe Windows installation uses a standard .exe installer, and the first launch goes directly into Jan’s main chat interface. Jan did not require an account to use a local model. On first launch, it asked whether we wanted to share anonymous usage data and then offered to download its recommended local model, Jan V3.5.\n\nThe initial Jan V3.5 download was listed at 2.8GB. Clicking Download started the process inside the app, and Jan displayed a success notification when it completed. Once installed, the model became available from the selector at the top of the chat window.\n\n## Browsing and Downloading Models\n\nJan’s Hub keeps model discovery and installation inside the application, with direct access to models from Hugging Face and useful information available before a download begins. Each entry includes the download size, quantization, capability tags, and a hardware-fit indicator, which helps narrow down models that are appropriate for the available memory and GPU.\n\nOn our RTX 5000 Ada system, several models were marked with a green Fits label. The Hub showed a 2.5GB Q4_K_M option for Jan V3.5 4B, while Jan V2 VL High was shown at 4.7GB and included Multimodal and Tools support. Alternate variants were available from the same model card.\n\nJan can also use hosted models from providers such as OpenAI, Anthropic, Gemini, Groq, and Mistral alongside models running locally through llama.cpp. This gives users one interface for downloaded and cloud-based models, while the Hub removes much of the manual work involved in finding GGUF files and checking memory requirements before installing them.\n\nWe downloaded Jan V2 VL High as our second model because it added image support while still fitting easily inside the RTX 5000 Ada’s 32GB frame buffer.\n\n## Model Settings and GPU Use\n\nOne of the bigger changes introduced during the v0.8.5 and v0.8.6 update cycle is that llama.cpp is now bundled in the application, whereas earlier builds managed it separately. Our installation showed llama.cpp 0.5.0, build b11146, with Parallel Sequences set to 0 so llama.cpp can use its default behavior, while Remember Conversations Between Sessions was enabled to keep each conversation’s prompt cache on disk.\n\nWith Jan V3.5 4B installed, the gear icon beside the active model provides access to its sampling and runtime settings. In our installation, Jan showed a temperature of 0.8, Top P of 0.95, Top K of 40, Min P of 0.05, and a Repeat Penalty of 1.\n\nAlthough the recent runtime changes altered the default GPU-offload behavior, our upgraded Jan V3.5 4B and Jan V2 VL High profiles still displayed 100 GPU layers in v0.8.6, so we left those saved values unchanged for the rerun. Additional controls included an 8,192-token context size, CPU settings for Mixture of Experts weights, KV-cache offload, and a batch size of 2,048. These options let you adjust memory use and inference behavior directly in Jan without editing command lines or separate configuration files.\n\n## Local Generation Performance\n\nWith both models running on the RTX 5000 Ada, we repeated our text-generation test on Jan v0.8.6 to measure decode speed under the updated runtime. We disabled Web Search, kept the context size at 8,192 tokens, kept GPU layers at the 100 value retained by our upgraded model profiles, and kept the batch size at 2,048.\n\nWe used the same text-only prompt for every run: “Explain RAID 5 and RAID 6 to someone building their first NAS. Include the main advantages and disadvantages of each, in about 400 words.” The first response after loading each model was discarded as a warm-up, followed by three measured runs in three fresh threads.\n\nJan V3.5 4B Q4_K_XL recorded a median decode rate of 123 tokens/sec across the three measured runs, with a median of 503 Jan-reported output tokens and a median peak GPU memory use of 4,839 MiB. Jan V2 VL High Q4_K_M reached a median of 82 tokens/sec, with 1,728 Jan-reported output tokens and a median peak GPU memory use of 8,197 MiB.\n\nThe llama.cpp timing lines also tracked Jan’s on-screen decode figures closely, with median eval rates of 122.55 tokens/sec for Jan V3.5 4B and 82.30 tokens/sec for Jan V2 VL High. We did not report a prefill median because conversation prompt caching was enabled in v0.8.6, and several repeated runs showed only one prompt token being evaluated. Those cached values would not provide a useful comparison of full-prompt processing performance.\n\n| **Model** | **Quantization** | **Output Tokens** | **Decode Speed** | **GPU Memory** | \n|---|---|---|---|---|\n| Jan V3.5 4B | Q4_K_XL | 503 | 123 tok/s | 4,839 MiB | \n| Jan V2 VL High | Q4_K_M | 1,728 | 82 tok/s | 8,197 MiB | \n\n**Note:** Output-token counts, decode rates, and GPU memory figures are medians of three measured runs after one discarded warm-up run. GPU memory is the median of the peak memory use observed during each measured run. Jan V2 VL High’s reported output count appears to include reasoning tokens, since the visible answer was around 400 words while Jan reported a much larger token total. Its output-token count should therefore not be compared directly with Jan V3.5 4B.\n\n## Using Jan for Local Chat\n\nJan’s main interface is simple to navigate once a model is installed. Recent conversations appear in the left sidebar, the active model stays visible at the top, and attachments and tools sit directly below the prompt field.\n\nSwitching between installed models takes only a few clicks. Jan also displays information at the end of a response, including token count and generation rate. Reasoning models can display their thinking activity while generating, though the level of detail depends on the model used.\n\nOur first local test asked Jan to explain the differences between RAID 5 and RAID 6 for someone building a NAS. Both downloaded models completed the request, with Jan V2 VL High generating a longer answer and using 8,197 MiB of GPU memory, compared with 4,839 MiB for the 4B model.\n\n## Web Search\n\nJan includes web search and page fetching directly in the chat interface, allowing supported models to bring current online information into a conversation when those tools are enabled. We repeated our earlier Web Search test after updating to Jan v0.8.6 by enabling the feature and asking Jan V2 VL High the same RAID 5 versus RAID 6 question used during our local testing.\n\nJan again decided on its own that current information could help and attempted to invoke Web Search even though the prompt did not ask it to search online. The reasoning view showed the model deciding to verify current NAS recommendations before submitting a search query. In this run, the search itself could not complete because the configured Exa service had reached its free MCP rate limit, but the test confirmed that Jan V2 VL High still automatically calls Web Search when it decides outside information may be useful.\n\nJan also supports MCP, which allows compatible external tools and services to be connected to the application, including options such as browser interaction and other integrations. We did not test MCP during this session, but its inclusion gives users another way to extend what locally running models can access and perform beyond the standard chat interface.\n\n## Multimodal Input\n\nTo retest image input after updating to Jan v0.8.6, we attached a fresh nvidia-smi screenshot to Jan V2 VL High and asked it to identify the GPU model, NVIDIA driver version, total VRAM, GPU utilization, GPU temperature, and power usage shown in the image. The model correctly identified all six values: NVIDIA RTX 5000 Ada Generation, driver 597.06, 32,760 MiB of VRAM, 1% GPU utilization, 35C temperature, and 4W / 250W power usage.\n\nThat’s better than our initial multimodal test on the earlier Jan build, which spent much of its response working through the terminal layout. The model still spent 21 seconds reasoning before answering, and Jan reported 1,979 output tokens, but the visible response was concise and answered the request directly.\n\n## Local API Server\n\nJan also includes a local OpenAI-compatible API server, accessible from its Settings menu. We selected Jan V3.5 4B as the default local API model and started the server directly from the interface.\n\nJan reported the server running on:\n\nhttp://127.0.0.1:1337/v1\n\nThe built-in server log confirmed a successful launch, while Jan also provided access to interactive API documentation.\n\nWe used PowerShell to make sure the local API worked as expected, starting with a query to /v1/models, which returned the installed models, including Jan V2 VL High. From there, we sent a chat-completions request to Jan-v3.5-4B-Q4_K_XL asking for a one-sentence explanation of RAID 6, and Jan returned the response directly in PowerShell:\n\n“RAID 6 is a disk array configuration that provides data redundancy by using two independent parity blocks per data block, allowing the array to survive the failure of up to two disks without data loss.”\n\nThe response contained one factual error: RAID 6 uses two parity blocks per stripe, not two parity blocks per data block.\n\nWith both requests working, we confirmed that software supporting an OpenAI-compatible endpoint can communicate directly with Jan’s downloaded models, without sending every prompt through the desktop chat interface.\n\n## What Went Wrong\n\nThe main issue from our initial testing came from the multimodal test, where Jan V2 VL High understood the nvidia-smi screenshot but spent far too long working through a simple extraction request. The v0.8.6 retest described above returned all six values in a concise response, though the 21 seconds of reasoning and 1,979 reported output tokens show the model still does a lot of work before it answers.\n\nThe Web Search retest on v0.8.6 hit a limit outside Jan: the configured Exa service had reached its free MCP rate limit, so the automatic search the model attempted during the RAID question couldn’t complete. Anyone leaning on Web Search should expect to configure a search provider with enough headroom for regular use.\n\nOutside those points, the v0.8.6 testing went smoothly. All six repeated local generation runs completed without crashes, model-load failures, or GPU detection problems.\n\n## Conclusion\n\nFor anyone curious about running local LLMs on Windows, Jan v0.8.6 removes much of the setup work while still leaving plenty of room to adjust how models run. On our test system, it automatically detected the RTX 5000 Ada; both downloaded models used GPU acceleration, and the Hub showed model size and hardware-fit information before we committed to a download. It provides enough guidance for someone getting started, while still offering settings such as context size, GPU layers, and memory options when more control is needed.\n\nJan goes well beyond running local models and providing a chat window, adding many features to the same interface. We tested Web Search directly from a conversation, tested image input with Jan V2 VL High, saw remote AI providers available alongside local llama.cpp models, and sent requests to the OpenAI-compatible API from PowerShell.\n\nOn our RTX 5000 Ada system, Jan V3.5 4B reached a median of 123 tokens/sec in our updated text-generation test, while Jan V2 VL High reached 82 tokens/sec. Jan V2 VL High also produced a much more concise response in our repeated multimodal test after the v0.8.6 update, correctly identifying all six requested values from the nvidia-smi screenshot.\n\nJan ran reliably throughout the session, and going from a fresh installation to running local models on the GPU took one installer, one model download, and a few clicks.\n\nWe’ll use the same testing method as we work through additional entries in our [Best Local LLM Tools leaderboard](https://www.storagereview.com/best/local-llm-tools), giving readers performance results from the same hardware alongside the differences in features and day-to-day use.", "url": "https://wpnews.pro/news/jan-v0-8-6-hands-on-running-local-llms-on-windows", "canonical_source": "https://www.storagereview.com/review/jan-v0-8-6-hands-on-running-local-llms-on-windows", "published_at": "2026-10-13 19:11:05+00:00", "updated_at": "2026-10-15 05:15:13.585779+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-products", "mlops"], "entities": ["Jan", "StorageReview", "llama.cpp", "HP Z4 Rack G5", "Windows 11 Pro for Workstations", "Intel Xeon w5-2465X", "NVIDIA RTX 5000 Ada Generation", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/jan-v0-8-6-hands-on-running-local-llms-on-windows", "markdown": "https://wpnews.pro/news/jan-v0-8-6-hands-on-running-local-llms-on-windows.md", "text": "https://wpnews.pro/news/jan-v0-8-6-hands-on-running-local-llms-on-windows.txt", "jsonld": "https://wpnews.pro/news/jan-v0-8-6-hands-on-running-local-llms-on-windows.jsonld"}}