The first time my RAG system gave a confidently wrong answer, I did what everyone does: I blamed the model. I swapped in a bigger one. I tuned the prompt. I added "only answer from the context provided" in bold. The answer got no better.
The problem was never the model. The model was faithfully summarizing the context it was handed — the context was just wrong. It had retrieved the wrong chunks, so it answered the wrong question, fluently.
This turns out to be the norm, not the exception. Industry analysis in 2026 keeps landing on the same number: when RAG fails, the failure is in retrieval roughly 73% of the time, not generation. The LLM gets blamed for a mistake that happened several steps upstream, before it ever saw a token.
So here's the checklist I wish someone had handed me before I shipped — organized as a walk through the whole pipeline, because retrieval isn't one step, it's a chain, and it can break at any link. Naive RAG ("chunk, embed, cosine similarity, stuff into prompt") was always a prototype. This is the gap between that and production.
The mistake underneath a lot of RAG pain is treating RAG as a single flow. It's actually two separate paths that most people accidentally couple together.
The indexing path (offline). Runs when documents are added or changed: parse the source → clean the text → chunk → (optionally) enrich each chunk with context → embed → write to the vector store and a keyword index. This can take minutes per document and runs in the background.
The query path (online). Runs on every user request, in real time, under a latency budget (aim for under ~3 seconds end to end): take the query → optionally rewrite it → retrieve candidates → rerank → assemble the prompt with citations → generate → log the trace.
The most common architectural mistake is coupling these. If re-indexing forces the query path offline, you can't iterate on chunking or swap embedding models without downtime — so you stop iterating, and a frozen pipeline is a stale pipeline. Keep them independent from day one.
Check: Can you re-chunk and re-embed your whole corpus without taking live search down? If not, decouple the paths before anything else.
Chunking is where pipelines silently fail, because bad chunks don't throw errors — they just quietly return technically-relevant, practically-useless context.
The naive default — "split every 1,000 characters with 100 overlap" — is a fast start and a slow ceiling. Fixed-size splitting cuts sentences mid-thought, tables mid-row, and code mid-function. The retrieved chunk looks relevant and is missing the half that mattered.
Better options, roughly in order of effort: Structure-aware splitting — split on the document's own boundaries: ## headings for docs, per-function or per-class for code, per-row for tables. Low effort, big payoff, respects how the content is actually organized. Semantic chunking — compute similarity sentence-to-sentence and start a new chunk where the meaning shifts, so each chunk is one complete thought. More compute, but a published comparison reported it lifting accuracy meaningfully over fixed-size on the same dataset.
The rule to hold onto: each chunk should be able to answer a question on its own. If a chunk only makes sense next to its neighbor, your splitting is too aggressive. Also mind chunk size — too small and you fragment ideas; too large and you dilute the signal, forcing the model to average across a wall of mostly-irrelevant text.
Check: Pull ten random chunks and read them cold. Does each stand on its own, or are half of them sentence fragments and orphaned table rows?
A subtle, high-impact one. If you embed only the raw body text of a chunk, you strip away the context that told a human what it meant — which section it's under, which product it's about, what came before it.
Two fixes, both cheap relative to their payoff: Embed context, not just body. Prepend the heading, a short document summary, or a one-line description of what the chunk is about before embedding. This aligns the chunk's vector with how people actually phrase questions. (This is the core idea behind "contextual retrieval" — giving each chunk a little situating context before it's indexed measurably improves recall.) Keep metadata attached. Every document arrives with structure — author, date, source, section, product version, document type, access level. Store it alongside the chunk. You'll use it in the next step.
Check: Does an isolated chunk in your index carry any signal about where it came from, or is it a naked paragraph with no situating context?
This is the single most common retrieval mistake, and it hides in plain sight because vector search usually works.