Private AI at Home: Ollama, Open WebUI, GPU Passthrough, Model Sizing, Document Chat and Local Voice for Home Assistant on Your Own Hardware

Private AI at Home is a practical guide to running your own AI server on a Proxmox homelab: a Debian 13 VM with a passed-through GPU, Ollama as a properly configured service, Open WebUI with real accounts, chat with your own documents, and a local voice pipeline for Home Assistant. It starts with the question that decides everything, whether a model fits in your card's memory, and shows the arithmetic: weights plus KV cache plus overhead, worked through for one model at four quantizations and three context lengths. Then it builds the server one decision at a time: GPU passthrough on Proxmox VE 9, the NVIDIA driver from Debian's own repositories, Ollama with a systemd override you can read in one screen, Open WebUI pinned to a version behind Caddy, retrieval that you understand well enough to write yourself, and Whisper, Piper and an Ollama conversation agent wired into Home Assistant with only safe devices exposed. Because Ollama's API has no authentication, a full chapter covers who can reach which port, remote access without port forwards, API keys and prompt injection. The last chapters cover updates, monitoring, choosing models by testing them on your own prompts, and a complete reference build. Every version, flag, environment variable and default was checked against the official Ollama, Open WebUI, Proxmox VE, Debian, Home Assistant and Wyoming documentation in October 2026. No prices, no speed claims.

What you get

• Before We Start: what local models do well and badly, privacy as the reason, and the hardware decision (CPU only, integrated GPU, NVIDIA, AMD with ROCm, Apple silicon) as one table
• Models and sizing: parameters, GGUF quantization names (Q4_K_M, Q5_K_M, Q8_0), the KV cache computed from a model's own config, a worked memory estimate per card size, context length and its cost, model licenses, the Ollama library and Hugging Face GGUF
• GPU passthrough on Proxmox VE 9: the IOMMU, vfio modules, reading IOMMU groups, a q35 + OVMF VM with All Functions and PCI-Express, the Debian 13 NVIDIA driver from non-free, nvidia-smi, the LXC alternative and AMD notes
• Ollama as a proper service: the install script and manual install, a systemd override (OLLAMA_HOST, OLLAMA_MODELS, OLLAMA_KEEP_ALIVE, OLLAMA_CONTEXT_LENGTH, OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS, OLLAMA_KV_CACHE_TYPE), the CLI, Modelfiles, the REST API with curl and Python streaming, embeddings, the OpenAI-compatible /v1 endpoint and structured outputs
• Open WebUI: a pinned Docker Compose stack behind Caddy with HTTPS, the first admin account, sign-ups and roles, the ConfigVar trap, the Ollama connection, model presets, where the data lives, a backup script and safe updates
• Chat with your documents: how retrieval works, embedding models in Ollama, Knowledge collections, chunk size, overlap and top K, why answers still need checking, and a small retrieval script of your own in Python with SQLite
• Speech and Home Assistant: Wyoming Whisper, Piper and openWakeWord in Docker, a fully local Assist pipeline, exposing only safe entities, and the Ollama conversation agent with Control Home Assistant, set up as separate chat and control agents
• Security and access: Ollama's API has no authentication, so three layers (bind address, OPNsense rules, nftables on the VM), remote access by VPN or Tailscale Serve, Caddy basic_auth, Open WebUI API keys, and prompt injection
• Day 2: the monthly update order, the silent CPU fallback after a kernel update, GPU metrics with nvidia_gpu_exporter and Prometheus, uptime and disk checks, measuring thermals and idle power yourself, and backups that skip the model store
• Choosing models for jobs and putting it together: a job-to-model starting table, an evaluation script for your own prompts, when a hosted model is the better tool, a complete reference build as commands, an AI server checklist and a cheat sheet

75 pages, 24 diagrams plus a chapter-opener graphic for every chapter, an AI server checklist and a cheat sheet. PDF and EPUB. Homelab the Right Way, Book 14.

Not sure yet? Read the free sample first: the cover, the full table of contents and all of Chapter 3, "Ollama" (installing Ollama with the script or by hand, a systemd override for OLLAMA_HOST, OLLAMA_MODELS, OLLAMA_KEEP_ALIVE, OLLAMA_CONTEXT_LENGTH and the parallelism and KV cache settings, the CLI and ollama ps, Modelfiles, the REST API with curl and streaming Python, embeddings, the OpenAI-compatible /v1 endpoint and structured outputs), at https://payhip.com/b/NfDLe

Also in a bundle: Everything Bundle: every ebook in the store, $89 · Homelab Bundle, $59.