Run local models

Local models are the reason Osmium is free. The app talks to a model server on your own machine over localhost, so there is no key, no account for the model, and nothing metered.

Ollama, the easy path

1. 01Install Ollama from ollama.com. Osmium can open the download for you if it is missing.
2. 02Pull a model sized for your GPU. If you are unsure, start with the 7B coder.
3. 03In Osmium, open Settings → Local models. Ollama is detected on 127.0.0.1:11434 automatically.
4. 04Select the model. It becomes available in every chat mode.

Not sure what fits? The model catalog on this site rates every model against common GPUs, and the app does the same check against your real hardware including system RAM and free disk.

LM Studio

Start LM Studio's local server and Osmium will find it on 127.0.0.1:1234. Anything LM Studio can serve is usable, and the OpenAI-compatible endpoint means no extra configuration.

Local GGUF files

Osmium can also start a llama.cpp server itself and serve a GGUF file from disk on port 39291. Use this when you want an exact quantisation that neither Ollama nor LM Studio packages.

Picking a size

4 GB VRAM

Qwen2.5 Coder 1.5B or 3B

8 GB VRAM

Qwen2.5 Coder 7B — the best default

12 GB VRAM

Qwen2.5 Coder 14B

24 GB VRAM

Qwen2.5 Coder 32B

Apple Silicon

Unified memory counts as VRAM; leave headroom for the OS

What local models cannot do

• They do not generate images. Vision models read images you attach; none of them produce pixels.
• Smaller models are weaker at multi-file refactors. Use the largest one that fits comfortably.
• A few models have no tool-calling support in their template, which limits agent work. Check the tool-calling row on any model page.