Ollama
Ollama makes running open-weight language models on your own machine about as easy as running a container. One command downloads a model; another starts a chat. Behind that simplicity it manages model files, picks GPU or CPU execution, loads and unloads models on demand, and exposes a local REST API — including OpenAI-compatible endpoints — so applications and tools can use local models with little or no code change.
Teams use Ollama for private experimentation with sensitive data, offline development, prototyping retrieval and agent workflows, embedding generation, and small internal tools. It is not designed as a hardened multi-tenant serving platform; knowing where its boundaries are is part of using it well.
TL;DR
ollama pulldownloads a model;ollama runchats;ollama psshows loaded models.- Models are identified by name and tag, which encodes size and quantization (for example
llama3.2:3b). - A Modelfile builds custom models with a system prompt, parameters such as temperature and context length, and adapters.
- The REST API on
localhost:11434offers chat, generate, and embeddings;/v1endpoints are OpenAI-compatible. - Supports structured outputs via JSON schema, tool calling, and vision for capable models.
- There is no built-in authentication: keep it bound to localhost or behind an authenticated proxy.
Quick Example
Pull and try a small model:
Create a customized assistant with a Modelfile:
Call it from existing OpenAI client code by changing the base URL:
Request structured output with a JSON schema through the native API:
Core Concepts
Models, Tags, and Quantization
Ollama's library hosts popular open-weight models in GGUF format. A tag like qwen2.5:7b-instruct-q4_K_M names the family, parameter count, variant, and quantization. Quantization stores weights in fewer bits (for example 4-bit), shrinking memory needs dramatically with modest quality loss; higher-bit variants trade memory for quality. Models from Hugging Face in GGUF format can also be pulled or imported. Check each model's license before using it commercially. See Vector Quantization for the related idea in vector search, and Small Language Models.
Modelfiles
The API
Responses stream by default; set "stream": false for a single JSON response.
Hardware, Memory, and Context
Memory needed is roughly the model's quantized size plus the KV cache for the context window, which grows with num_ctx and the number of parallel requests. Ollama uses NVIDIA (CUDA), AMD (ROCm), and Apple Silicon (Metal) GPUs, splitting layers between GPU and CPU when VRAM is insufficient — which works but is much slower. ollama ps shows the split. Larger context windows enable longer documents but consume memory and slow generation. See GPU Computing and CUDA.
Server Settings
Environment variables configure the server: OLLAMA_HOST (bind address, default 127.0.0.1:11434), OLLAMA_KEEP_ALIVE (how long models stay loaded), OLLAMA_NUM_PARALLEL (concurrent requests per model), OLLAMA_MAX_LOADED_MODELS, and OLLAMA_ORIGINS (allowed browser origins).
Security Boundaries
- No authentication: anyone who can reach the port can run models, pull new ones, and delete existing ones. Keep the default localhost binding, or put Ollama behind a reverse proxy with authentication and TLS.
- Never expose it directly to the internet; scans regularly find thousands of open instances.
- Browser access: restrict
OLLAMA_ORIGINSso arbitrary websites cannot call your local API. - Model provenance: pull from trusted sources and review licenses; model files are large binaries you are choosing to trust.
- Data privacy: prompts stay on the machine, which helps with sensitive data, but logs, shell history, and application traces can still capture them.
Ollama Versus Production Inference Servers
Develop against Ollama's OpenAI-compatible API, and switch the base URL to a production server or hosted API when needs grow. See LLM Inference.
Best Practices
Pin Model Tags
Use explicit tags (size and quantization) rather than latest, so results are reproducible across machines.
Size the Context Deliberately
Set num_ctx to what your use case needs; defaults may truncate long prompts, and very large contexts waste memory.
Keep the Server Local or Protected
Bind to localhost by default, and add authentication, TLS, and network restrictions before sharing an instance.
Evaluate Before Relying on a Model
Small local models vary widely in quality. Test them on your own tasks with an evaluation set before building on them. See LLM Evaluation.
Common Mistakes
Binding to All Interfaces
Silent Context Truncation
Treating Local Models as Equivalent to Frontier Models
FAQ
What is Ollama?
A tool for downloading and running open-weight language models locally, with a command-line interface, model management, GPU acceleration, and a local REST API that includes OpenAI-compatible endpoints.
Does Ollama need a GPU?
No, but a GPU makes generation much faster. Ollama runs models on CPU when necessary and splits layers between GPU and CPU when the model does not fit entirely in GPU memory.
Can I use Ollama with the OpenAI SDK?
Yes. Point the client's base URL at http://localhost:11434/v1 and use any placeholder API key; most chat and embedding calls work unchanged.
Is Ollama safe to expose on a network?
Not directly. It has no built-in authentication, so anyone who can reach it can use and manage models. Keep it on localhost or behind an authenticated, encrypted reverse proxy.
What do quantization tags like q4_K_M mean?
They describe how model weights are compressed. Lower-bit quantizations use less memory and run faster with some quality loss; higher-bit versions are closer to the original model but need more memory.
Related Topics
- LLM Inference — Serving and optimizing model inference
- Small Language Models — Compact models that run locally
- Large Language Models — How LLMs work
- GPU Computing and CUDA — Hardware acceleration
- Hugging Face — Model hub and tooling
- Vector Quantization — Compression in vector search