How I built a 6-agent AI system with hybrid RAG, guardrails, and measurable quality — self-hosted on a $200 mini PC, with zero cloud bills.
Anyone can chain OpenAI calls. I wanted to answer a harder question: can a genuinely useful multi-agent system run on hardware I own, with full data privacy, at zero marginal cost? That constraint forces the engineering that actually matters in production — model selection under memory pressure, retrieval quality you have to measure instead of assume, and reliability without a cloud provider's safety net.
The result is Groot OS: a supervisor-routed agent system that handles my research, coding help, tutoring, and job-search workflows — reachable from Slack and Telegram, running around the clock on a used Lenovo M720Q (i5-8500T, 16GB RAM, Ubuntu 22.04) on my desk in Toronto.
Every request enters through Groot, a supervisor agent built on LangGraph that classifies intent and routes to a specialist. Each agent runs a model chosen for its task — and for its memory footprint, since all seven models share 16GB of RAM through Ollama on CPU only.
Routing through a supervisor instead of a flat toolbelt keeps each specialist's context small and its prompt focused — which matters enormously when your largest model is a 14B running on CPU.
Pure vector search fails on exact identifiers — error codes, function names, acronyms. Pure keyword search fails on paraphrase. Groot fuses both:
Knowledge lives in 6 ChromaDB collections split by domain, with RBAC enforced per agent — the tutoring agent cannot read job-search data, and vice versa. Splitting by domain keeps retrieval precise; access control keeps a misrouted query from leaking context across workflows.
The seven models total 27GB on disk — they can't all be resident at once. Groot treats model residency as a scheduling problem: keep the supervisor's router model warm for latency, load specialists on demand, and evict by least-recently-used when memory pressure rises.
A RAG pipeline you haven't measured is a RAG pipeline you're guessing about. Groot's retrieval is scored with Ragas, using qwen2.5:7b as the judge model and nomic-embed-text embeddings — the same stack that serves production, so the eval measures what users actually get.
| Metric | Score | What it means |
|---|---|---|
| Faithfulness | 1.000 | Zero hallucinated claims in answers |
| Answer Relevancy | 0.676 | Honest number — improving via query rewriting |
| Overall | 0.838 | Composite pipeline score |
I publish the 0.676 on purpose. Perfect scores across the board usually mean a soft eval set; a real system has a number it's still working on. Output quality is enforced at runtime by guardrails-ai validators before anything reaches Slack or Telegram.