selfhostkit.dev · guide · self-hosted AI

How to run an LLM locally

You don't need OpenAI to run a capable model. I run open-source LLMs on my own hardware and on a VPS, for agents and private data, with no API bill and nothing leaving the box. Here's the short path that actually works in 2026.

Why self-host a model at all

Your data stays yours. Prompts and documents never leave your machine. For anything sensitive, that's the whole point.
No per-token bill. Once it runs, it runs. Great for agents that loop, batch jobs, or just tinkering without watching a meter.
It doesn't disappear. An open-weights model you downloaded is yours. No deprecation email, no surprise price change.

Honest take: a local 8B-14B model won't match the best hosted frontier models on hard reasoning. But for summarizing, drafting, classification, RAG over your own docs, and powering agents, the current open models are genuinely good, and they run on modest hardware.

Step 1: Know what your hardware can run

The one number that matters is memory. A model's weights have to fit in VRAM (on a GPU) or RAM (CPU-only), plus some room for context. Rough guide:

8 GB GPU or 16 GB RAM → 7-8B models at Q4. Fast, and plenty for most tasks.
16-24 GB → 13-14B comfortably, or a 32B at tight quant.
48 GB+ (or 2 GPUs) → 70B territory, the sweet spot for quality on open models.

Don't guess: drop the model size, quant and context into the LLM VRAM calculator and it tells you exactly what fits.

Step 2: Install Ollama

Ollama is the fastest way in. It downloads models, handles quantization, and exposes a local API, in one command. On Linux:

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1:8b

That pulls a Q4 build and drops you into a chat. Swap the tag for another model: ollama run qwen2.5:14b, ollama run mistral, ollama run gemma2:9b. It also serves an OpenAI-compatible API on localhost:11434, so most existing code and agent frameworks just point at it.

Step 3: Add a web chat (optional)

For a ChatGPT-style interface, put Open WebUI in front of Ollama. It's one container:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Open localhost:3000, and you have multi-user chat, history, and document upload over your local models. Prefer a compose file? Paste that command into the docker run → compose tool.

Step 4: Run it on a VPS (no GPU needed)

No graphics card? Small models run fine on CPU, just slower. A VPS with enough RAM will happily serve a 7-8B model for personal use or light automation. I cover which one to rent, by real price and specs, in the best VPS guide. Put it behind a reverse proxy with HTTPS and you have a private AI endpoint on your own domain.

For real GPU inference on a server, you want a rented GPU instance, not a shared CPU VPS. But for getting started, learning the stack, and running small models, CPU-only is the cheapest on-ramp, and everything you learn transfers.

Where this goes

Once Ollama is serving a model, it becomes a building block: local RAG over your notes, agents that don't phone home, a private assistant wired into your own tools. That's the direction I'm building toward, and more self-hosted AI tools and guides are landing here.

← back to the tools · VRAM calculator →