Local LLMs on One RTX 3090

You have an RTX 3090, or you're about to buy one used, and you want to know which engine, model, quant and coding agent to run on it before you spend a weekend finding out. This book is our lab notebook from October 2026, cleaned up for other people to read. Every number in it was measured on our own test box. If we couldn't trace a number back to a run log, we cut it.

What's inside

• Hardware and power: Our card's core sat in the 50s C while its GDDR6X memory went past 100 C. We show the fan and power-limit settings that held it at 98 C, and what they cost: a 250 W limit took 8.7% off single-stream dense decode.
• llama.cpp vs vLLM (and SGLang): For one user, llama.cpp. For many, vLLM or SGLang with AWQ: serving an 8B model to 16 users, vLLM gave 2.2x llama.cpp's best total throughput. Includes the GGUF trap, prefix caching and what each engine costs in VRAM.
• Models and quants for 24 GB: Thirteen models on the same agent bench at 64K context. Our default, Qwen3.6-35B-A3B, passed 12 of 12 at about 149 tok/s on an empty context and 126 at 42K, in 21.9 GiB. Also covered: Gemma 4 26B-A4B as a backup, a 74B MoE at 2-bit, dense 27B and what abliterated ("uncensored") models lose on tool calls.
• Coding agents: goose, opencode and aider on the same local server, graded by hidden checks. goose and opencode passed 24 of 24 runs; aider passed 15 of 24. Both good agents sometimes "passed" the repo's own tests with a wrong fix, and the chapter shows how.
• A bilingual EN↔ES intake test: Text translation with a short system prompt, Hunyuan-MT-7B as a dedicated model, and faster-whisper next to Qwen3.6 on the same card, with a translation ready about 1.5 seconds after the audio ends. All test sentences were written for the test and the audio is synthetic.
• Small training runs with nanochat: A 110M-parameter model pretrained in 205 minutes at 220 W, chat fine-tuning, the dataset licence traps we hit, and why a GPT-2-class run doesn't fit one card's schedule.
• Commands appendix: The power, fan, server and benchmark commands we ran, so you can repeat the main tests on your own box.
• Benchmark appendix and CSV: All 91 benchmark rows as a table in the book and as benchmarks.csv, so you can sort, filter and compare against your own card.

How it fits with the Local AI Stack Kit

The Local AI Stack Kit ($29, sold separately) is the quick start: the scripts and configs to run Qwen3.6-35B-A3B behind a coding agent tonight. This handbook doesn't include those scripts. It covers what the kit leaves out: the measurements behind the kit's choices, thermals, engines, the full agent bench, translation and training. The numbers in both come from the same runs, so they agree.

Before you buy

• Test hardware: one used RTX 3090 Founders Edition (24 GB), Ubuntu 24.04, NVIDIA driver 595.99.02, CUDA 13.2, llama.cpp b11753, vLLM 0.32.0, SGLang 0.5.22. Power limit 300 W for most runs; 250 W and 220 W where noted.
• These are results from one card, one driver, one build of each engine and usually one quant per model. Your results will vary with your card, cooling, driver, engine build and model files. Nothing here is a performance guarantee.
• You should be comfortable on a Linux command line. Power, fan and training settings push hardware hard; you apply them at your own risk.
• The book doesn't include any models, engines or agents. You download those from their publishers.

How it was made

Written with AI help (Claude) and reviewed by AmagiLabs before release. The measurements came from real runs on our hardware, and every number was checked against the log it came from.

Refunds

All sales are final. You get this digital download right away, so we don't give cash refunds. If a file won't open or the product doesn't match this page, email us within 7 days of purchase. We'll fix it or send a replacement. If we can't, you'll get store credit for the full price.

EU and UK buyers: you'd normally have 14 days to withdraw from this purchase. You lose that right once your download or access starts, because you ask for immediate access at checkout. If the product is faulty or not as described, the 7-day limit doesn't apply to you. If we can't fix it, you can choose a refund instead of store credit.

Nothing in this policy limits your rights under the law where you live. You can ask us for a written copy of this policy at any time.

Format: one zip with the PDF (69 pages, 6x9 in), benchmarks.csv and the EPUB edition for e-readers.