Local AI

Local AI refers to running language models directly on your own hardware — a laptop, desktop, or home server — rather than sending requests to a hosted service like Claude, ChatGPT, or Gemini over the internet. Tools like Ollama and LM Studio make this practical even on consumer hardware, using quantized versions of open-weight models that trade some raw capability for a dramatically smaller memory and compute footprint.

Local vs cloud: strengths and trade-offs

Local models excel at:

• Privacy — nothing leaves the machine; sensitive notes, drafts, or personal data never touch a third-party server
• Cost — no per-token API fees, no subscription; the only cost is the electricity and the hardware you already own
• Offline availability — works with no internet connection, useful for travel or unreliable connectivity
• Latency for small tasks — for simple, well-scoped jobs, a local model can respond faster than a network round trip to a cloud API
• Control — full choice over which model version runs, no risk of a provider changing or deprecating a model out from under a workflow

Cloud models excel at:

• Raw capability — the largest cloud models substantially outperform anything that fits on consumer hardware, especially for complex reasoning, long-context understanding, and nuanced writing
• Consistency — no local hardware constraints, thermal throttling, or memory limits to work around
• Maintenance-free — no model downloads, driver issues, or local resource management
• Multimodal and specialised capability — larger vision, coding, or agentic capabilities are often cloud-first before trickling down to smaller local-friendly models

In practice, the strongest setups don’t pick one over the other — they route tasks to whichever tier fits, using local models for fast, private, low-stakes work and reserving cloud models for anything demanding deeper reasoning or higher accuracy.

Role in the my homelab

This is the exact split used across the Fastigiata homelab. On the primary M1 Pro Mac, a local model ladder handles everyday tasks — small 3B-class models (like ministral-3b and llama3.2:3b) for speed, a vision-capable model for image-related tasks, and a larger local reasoning model as a ceiling — before falling back to cloud models like Claude Haiku or Gemini Flash for anything that benefits from more horsepower.

Within Obsidian, this local-first approach powers a retrieval-augmented generation (RAG) pipeline: plugins like Smart Connections or Obsidian Copilot embed and index notes locally, then use a local or cloud model to answer questions grounded in the vault’s own content, rather than a model’s general training knowledge. VS Code plays a supporting role alongside this — not running inference itself, but serving as the environment where the surrounding tooling (scripts, configs, git operations) that keeps the RAG pipeline and this published vault’s Quartz pipeline running is edited and managed.

For this published vault specifically, the plan is a lighter version of the same idea on the Linux Mint machine — a modest local model (ministral-3B or Llama 3B class) handling retrieval over this smaller, curated set of notes, with Claude Haiku available via API as a cloud fallback for anything the local model isn’t strong enough to handle well.