Written in April 2026. The setup has grown since, and the current map is at duncansmith.tech/homelab.
A $300 gaming GPU runs my language model, speech-to-text and embeddings. Document search runs on the NAS beside it. Sensitive material stays on hardware I own. A cloud model is the fallback for work a small model can’t do.
I work with contracts, financial records and personal notes. Sending those to a hosted API puts them on servers I don’t control, under terms that can change. I wanted chat with my documents, transcription and semantic search without that trade.
The hardware
Two machines, under $1,000 together:
A Windows PC with an NVIDIA RTX 4060 (8GB VRAM) and 32GB of RAM. It runs LM Studio and WhisperX for inference, embeddings and speech-to-text.
A TrueNAS box with an Intel N95 and 16GB of RAM. It runs Qdrant, AnythingLLM and n8n. I wrote about it in building a 24-service homelab.
The stack
LM Studio serves models through an OpenAI-compatible API, so n8n, AnythingLLM and my own scripts all connect the same way. I run Qwen3.5-9B for general work. It fits in 8GB with int8 quantization and answers in 1 to 3 seconds. Embeddings come from nomic-embed-text v2, a 768-dimensional model.
A model this size handles summarization, classification, question answering and structured extraction well enough for daily use.
WhisperX runs in Docker with CUDA acceleration. A 30-minute recording transcribes in about 2 minutes on the “base” model. The “large” model is slower and better with accents and jargon. SolScribe, my transcription app, started on top of it.
Qdrant stores the embeddings and searches by meaning. “Docker networking issues” finds a note titled “Container can’t reach host network”. It holds three collections: Paperless documents, Obsidian notes and wiki pages.
AnythingLLM connects LM Studio to Qdrant. I upload documents into a workspace and ask questions, and the answers cite their sources.
n8n ties it together through three shared sub-workflows: a local LLM call, a variant that tries a larger cloud model first and an embedder. The detail is in 80 workflows later.
What it does day to day
Document questions. I upload a 50-page PDF and ask “What are the payment terms?” The answer arrives in about 3 seconds with a citation to the page and paragraph.
Voice memos. I record on my phone. SolScribe transcribes it, n8n summarizes it with the local model, creates an Obsidian note and embeds it. It takes under a minute.
Search across everything. One query returns results from documents, notes and wiki, ranked by meaning.
Document intake. A file dropped into Paperless-ngx is OCR’d, tagged by the model and embedded. A notification reaches my phone when it’s done.
What it costs
GPU: about $300, once
Electricity: $8 to $12 a month
API and subscription fees for the local path: none
The cloud services it replaced would run $100 to $200 a month, so it broke even in about three months.
Where it falls short
Model quality. A 9B model is not GPT-4. For complex reasoning, creative writing or nuanced analysis, cloud models are still better, and I use OpenRouter or Claude for those.
Context. Local models handle 4K to 8K of context well. More needs more VRAM or quantization tricks.
One user. Serving a team would take a bigger GPU or several inference servers.
Maintenance. Models and containers need updating, an estimated 2 to 3 hours a week.
Starting small
LM Studio or Ollama on any machine with an NVIDIA GPU, with a small model such as Phi-3 Mini or Qwen 2.5 3B, is enough to try the chat interface. Qdrant starts with a single Docker command.