cd /news/ai-infrastructure/your-model-fits-in-vram-your-context… · home › topics › ai-infrastructure › article
[ARTICLE · art-172694] src=dev.to ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Your Model Fits in VRAM. Your Context Window Might Not

A developer outlines a method for verifying that Ollama's actual allocated context window matches the requested size, since model weights fitting in VRAM does not guarantee the key/value cache and runtime buffers will. The approach uses `ollama ps` to record allocated context and GPU/CPU processor split, then repeats runs with baseline, typical, and stress prompts while capturing prompt_eval_count, eval_count, and peak device memory. The guidance recommends shrinking context or using retrieval when offload or memory pressure appears, and comparing quantization or model changes only after recording a baseline.

recommendation

Before trusting a model's advertised context length, run `ollama ps` after a request to confirm allocated context and GPU/CPU split, and test with your own typical and large prompts.

more in playbook: Inference and self-hosting
by read3 min views4 publishedNov 19, 2026

“This model fits in my GPU memory” answers only one part of the question. The model weights are the obvious allocation, but inference also needs working memory. A longer context can increase the key/value cache, while runtime buffers and other GPU processes compete for the same device memory. The outcome can be partial CPU offload, slower responses, or a request that cannot run at the context size you expected.

Check the context Ollama actually allocated before relying on the model’s advertised maximum. Then measure it with the prompts your application sends.

Prerequisites: a working Ollama installation, one model already available, and a representative prompt. No new model download is needed. First record the model name and machine state, then load it with the context size you actually want to test. In Ollama, context length can be configured in the app or through OLLAMA_CONTEXT_LENGTH when starting the server. Changing the value after a server is already running may require restarting it.

After sending one request, run:

ollama ps

Record the CONTEXT and PROCESSOR columns. The latter tells you whether the model is fully on GPU or split between GPU and CPU. Also note other GPU workloads, because the same inference process may behave differently when another application is using memory.

Now repeat with three prompt sizes: a short baseline, a typical real task, and a deliberately large but plausible task. Keep the model, output limit, and machine state as consistent as you can. Capture the API’s prompt_eval_count, eval_count, and timing fields for each request. A request with a 30,000-token prompt is a different memory and latency workload from a 3,000-token one, even when both use the same model.

Ollama documents default context lengths selected by VRAM bands and notes that increasing context increases memory requirements. GPU placement still depends on the model and available memory. Its context guide recommends checking ollama ps for allocated context and offload status. See Ollama context length and API usage metrics.

Make a small table for each run:

Run Prompt tokens Requested context Allocated context Processor split Peak device memory Task result
Baseline record it record it from ollama ps from ollama ps from OS/GPU monitor useful / incomplete
Typical record it record it from ollama ps from ollama ps from OS/GPU monitor useful / incomplete
Stress record it record it from ollama ps from ollama ps from OS/GPU monitor useful / incomplete

Use the operating system or vendor’s GPU monitor for peak memory; Ollama’s usage fields report token counts and durations. Record the exact model tag, quantization, Ollama version, context setting, and other active GPU work alongside the table so you can compare runs later.

A short prompt after setting a large maximum context tells you little about GPU placement when the context fills. Conversely, allocating an enormous context for consistently short prompts can reserve memory you do not need. Include the typical and unusually large prompts from your own workload.

If the context is allocated as requested and the task is useful, keep the configuration and monitor it under realistic concurrent load. If Ollama reports a CPU/GPU split or memory use is near the device limit, first ask whether your workflow truly needs the full context. Retrieval that supplies a few relevant documents can be more practical than passing an entire repository. A smaller context may also leave room for another GPU workload.

If the task genuinely needs long context, compare a different quantization or model only after recording the current baseline. Each choice trades memory against output quality, throughput, or compatibility; there is no context setting that makes those costs disappear. Also distinguish “the request completed” from “the model used the context effectively.” Evaluate the answer against a task where the needed fact is far from the end of the prompt.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @ollama 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/your-model-fits-in-v…] indexed:0 read:3min 2026-11-19 · —