I Built a Personal AI That Actually Knows My Projects (RAG + Ollama, Zero Cloud)
"Here's the architecture. Here's the README. Here's what I tried last time." Every. Single. Time.
So I built a local RAG (Retrieval-Augmented Generation) system that knows my projects, my notes, and my docs — permanently. No cloud. No API costs. No context window resets.
Here's exactly how it works and what I learned running it for the last few months.
LLMs don't remember. You paste the same 200 lines of context every session, hit the token limit, and start over. It's fine for one-off questions. It's exhausting for ongoing projects.
The standard solution is RAG: instead of stuffing everything into the prompt, you store docs in a vector database and retrieve only the relevant chunks when you ask a question. The model sees 3-5 paragraphs of targeted context instead of your entire repo.
Everything that would normally eat my context window: Project READMEs and architecture docs My personal notes (Obsidian vault) Code snippets and past solutions API documentation I use regularly Stack Overflow answers I bookmarked (because I always forget them again) Config files and deployment notes
That's the entire stack. No Docker required (though Chroma has a Docker option if you want a persistent server).
Real answers from your own documentation. No hallucinations about your specific setup.
Run this as a cron job every hour. Your knowledge base stays current automatically.
Before RAG: "Explain the background service memory limit in my Garmin project" → paste 200 lines → wait → answer Every new chat session: context reset, start explaining again
After RAG: ask("Garmin background service memory limit") -> "64KB sandbox, pass data via Background.exit(dictionary)" — in 1.8 seconds
My LLM now answers questions about projects I haven't touched in 6 months. No context management. No pasting. Just ask.
| Setup | RAM | Embedding Speed | Query Speed | |-------|-----|----------------|-------------| | Mac Mini M4 8GB | 8GB | ~500 docs/min | ~2s | | RTX 3060 12GB | 12GB VRAM | ~3000 docs/min | ~0.5s | | Old laptop 8GB | 8GB | ~100 docs/min | ~5-8s |
The embedding step (indexing) is the slow part — run it once, then it's instant.
Tips from Running This for 3 Months Chunk size matters — 400 tokens works well for prose and docs. For code, try 200 with more overlap. Metadata is your friend — store filename and section in chunk metadata. When the AI says "see the deployment notes," you know exactly where to look. Re-rank when accuracy matters — if top-5 chunks aren't enough, add a re-ranker step (Cohere has a free API, or use a local cross-encoder). Watch your embed model — nomic-embed-text beats most larger models for RAG. Don't use your chat LLM for embeddings. Hybrid search — combine vector search with BM25 keyword search for better results on technical queries with specific names/functions.
Indexing is slow on CPU. My first run on the Mac Mini took 45 minutes for ~4,800 chunks. The GPU machine does it in 8 minutes. Not a dealbreaker, but plan a coffee break.
Code chunks are tricky. RAG works great for prose. For code, you often need the full function context, not just a 400-token snippet. I ended up indexing both small chunks (for search) and full files (for context) — dual storage, but worth it.