← All notes

Summary

A machine that is already on can serve a local model without sending every prompt to a vendor. The limit is memory, not a special "idle mode." Ollama and the llama.cpp HTTP server both load weights into GPU memory or system RAM, hold them while they are in use, and can drop them when they sit unused or when another model needs the room.

See where the weights went

Ollama's default context is 4096 tokens. Override it with an environment variable, a session parameter, or the context option on the generate API. Parallel requests grow the context by the number of requests, so memory scales with that.

The process list shows a full GPU load, a full CPU load, or a split. Concurrent GPU loads require each new model to fit entirely in video memory. Otherwise requests wait until an idle model is unloaded. A model that fits one GPU stays there; otherwise it is spread across GPUs. Flash Attention, when the device supports it, cuts memory as context grows, and the key-value cache can then use a smaller type than the default.

Hold a model, or give the memory back

Ollama keeps a model loaded for 5 minutes, then unloads it. Stop a model from the command line, or set keep-alive on the generate and chat APIs: a duration, a number of seconds, a negative value to leave it loaded, or zero to unload after the reply. A server environment variable sets the default; the API value overrides it. An empty generate or chat request preloads a model.

Several models load only if they fit. The stated default cap is three times the GPU count, or three for CPU. Parallel requests per model are capped too, and memory scales with parallelism times context length. A full queue returns an overloaded error. On Windows with Radeon GPUs the FAQ's current default is one model.

The server binds the loopback address until you change the host variable. A local run does not send prompts to the vendor. Cloud-hosted models are processed to provide the service; the FAQ says that content is not stored, logged, or used for training. Disable cloud features in the server config or with the no-cloud environment variable, then restart. You lose cloud models and web search.

llama.cpp on the same machine

The llama.cpp server runs full-precision and quantized models on GPU or CPU. A quick start loads one model and a context size and listens on the loopback address at the server's default port. How many layers go to video memory defaults to automatic, and a fit option (on by default) adjusts unset arguments so they fit in device memory. Mixture-of-experts weights can stay on the CPU.

With no model argument, the process is a router that loads and unloads models and forwards each request. A cap limits how many models are loaded at once. A memory margin, disabled at zero, unloads least-recently used models until a new one fits and some device memory stays free.

Sleep after idle is off unless you set a number of seconds. Then the model and its key-value cache leave RAM, and the next real task loads them again. Health, properties, the model list, and metrics do not count as tasks and do not reset the timer. Use sleep when the machine should not hold weights between jobs. Use a negative keep-alive, or leave sleep off, when you would rather spend memory to keep the next reply fast.

Claims

• id: C1 claim: Ollama's default context length is 4096 tokens unless you override it. source: S1 evidence: "By default, Ollama uses a context window size of 4096 tokens."
• id: C2 claim: Models stay loaded for five minutes by default, then they are unloaded. source: S1 evidence: "By default models are kept in memory for 5 minutes before being unloaded."
• id: C3 claim: When memory is tight, idle models are unloaded so a new model can load, and queued work runs in order. source: S1 evidence: "As prior models become idle, one or more will be unloaded to make room for the new model."
• id: C4 claim: Concurrent GPU loads require each new model to fit entirely in video memory. source: S1 evidence: "When using GPU inference new models must be able to completely fit in VRAM to allow concurrent model loads."
• id: C5 claim: A local Ollama run does not show your prompts to the vendor. source: S1 evidence: "Ollama runs locally. We don't see your prompts or data when you run locally."
• id: C6 claim: The llama.cpp server can run full-precision and quantized models on GPU or CPU. source: S2 evidence: "LLM inference of F16 and quantized models on GPU and CPU"
• id: C7 claim: After the configured idle time, sleep mode unloads the model and its key-value cache from RAM. source: S2 evidence: "the model and its associated memory (including the KV cache) are unloaded from RAM to conserve resources."
• id: C8 claim: In router mode, a memory margin unloads least-recently used models until a new model fits and free device memory remains. source: S2 evidence: "unload LRU models until a new one fits"

@misc{iil_note_kws1vjzm,
  author = {{iinventedlight}},
  title = {{Running local models on idle hardware}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-03},
  url = {https://iinventedlight.com/library/notes/ao-idlehw/},
  note = {Permanent ID iil:note:kws1vjzm. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-idlehw.md.