“This model fits in my GPU memory” answers only one part of the question. The model weights are the obvious allocation, but inference also needs working memory. A longer context can increase the key/value cache, while runtime buffers and other GPU processes compete for the same device memory. The outcome can be partial CPU offload, slower responses, or a request that cannot run at the context size you expected.
Check the context Ollama actually allocated before relying on the model’s advertised maximum. Then measure it with the prompts your application sends.
Prerequisites: a working Ollama installation, one model already available, and a representative prompt. No new model download is needed. First record the model name and machine state, then load it with the context size you actually want to test. In Ollama, context length can be configured in the app or through OLLAMA_CONTEXT_LENGTH when starting the server. Changing the value after a server is already running may require restarting it.
After sending one request, run:
ollama ps
Record the CONTEXT and PROCESSOR columns. The latter tells you whether the model is fully on GPU or split between GPU and CPU. Also note other GPU workloads, because the same inference process may behave differently when another application is using memory.
Now repeat with three prompt sizes: a short baseline, a typical real task, and a deliberately large but plausible task. Keep the model, output limit, and machine state as consistent as you can. Capture the API’s prompt_eval_count, eval_count, and timing fields for each request. A request with a 30,000-token prompt is a different memory and latency workload from a 3,000-token one, even when both use the same model.
Ollama documents default context lengths selected by VRAM bands and notes that increasing context increases memory requirements. GPU placement still depends on the model and available memory. Its context guide recommends checking ollama ps for allocated context and offload status. See Ollama context length and API usage metrics.
Make a small table for each run:
| Run | Prompt tokens | Requested context | Allocated context | Processor split | Peak device memory | Task result |
|---|---|---|---|---|---|---|
| Baseline | record it | record it | from ollama ps |
from ollama ps |
from OS/GPU monitor | useful / incomplete |
| Typical | record it | record it | from ollama ps |
from ollama ps |
from OS/GPU monitor | useful / incomplete |
| Stress | record it | record it | from ollama ps |
from ollama ps |
from OS/GPU monitor | useful / incomplete |
Use the operating system or vendor’s GPU monitor for peak memory; Ollama’s usage fields report token counts and durations. Record the exact model tag, quantization, Ollama version, context setting, and other active GPU work alongside the table so you can compare runs later.
A short prompt after setting a large maximum context tells you little about GPU placement when the context fills. Conversely, allocating an enormous context for consistently short prompts can reserve memory you do not need. Include the typical and unusually large prompts from your own workload.
If the context is allocated as requested and the task is useful, keep the configuration and monitor it under realistic concurrent load. If Ollama reports a CPU/GPU split or memory use is near the device limit, first ask whether your workflow truly needs the full context. Retrieval that supplies a few relevant documents can be more practical than passing an entire repository. A smaller context may also leave room for another GPU workload.
If the task genuinely needs long context, compare a different quantization or model only after recording the current baseline. Each choice trades memory against output quality, throughput, or compatibility; there is no context setting that makes those costs disappear. Also distinguish “the request completed” from “the model used the context effectively.” Evaluate the answer against a task where the needed fact is far from the end of the prompt.