For a while, my retrieval setting was topk = 10. It felt responsible. If I only pulled back 3 chunks and the right one wasn't among them, the answer would be wrong for a reason that had nothing to do with the model. Pulling back 10 felt like insurance — cast a wide net, let the LLM sort out what actually mattered.
I remember one question in particular — something specific, with a clear correct chunk sitting somewhere in the middle of those 10. The LLM's answer wandered. It grabbed a detail from a chunk that was only loosely related, ignored the one that actually answered the question, and produced something that read confidently but missed the point. The right information was in the prompt. It just wasn't the information the model reached for.
That's when I went back and actually read what I'd been sending. Ten chunks, most of them tangential, all competing for the model's attention in one long prompt. The correct answer wasn't easy to find in there — even for me, reading it slowly, on purpose. I'd been treating "send more context" as a safety net. It was actually working against the model, and against my bill, on every single request.
So I tried the opposite: retrieve more candidates than I needed, but add a reranking step that scores them for actual relevance, and only send the LLM the top 3 after that. Fewer tokens per prompt. And, somewhat counter to what I expected, better answers — not despite sending less, but because of it.
That flipped something I'd assumed without ever testing it: more retrieved context isn't more safety. Past a point, it's noise the model has to wade through, paid for at input-token prices, on every question.
Retrieval quality determines how much you end up overpaying the LLM to compensate for it. Sending more chunks isn't a safety margin — it's a tax that also makes the model's job harder.
Every earlier episode in this series was about getting clean material into the vector database. This one is about the last decision before that material reaches the model: how much of it, and how well-chosen, actually makes it into the prompt. Get that wrong, and no amount of extraction quality, chunking care, or metadata precision from Episodes 2 through 5 will save you — the right answer can be sitting in the prompt and still get missed.
The most direct cost is the one from the story: every extra chunk in topk is extra input tokens, paid on every single query.
This part is intuitive once you say it out loud — more chunks costs more tokens. What's easy to miss is that this cost buys you very little if the extra chunks aren't actually useful. You're not paying for more safety. You're paying for more text the model has to read past.
This is the cost that surprised me in the story, and it doesn't show up as a token count — it shows up as answer quality.
Language models don't weigh every part of a long prompt equally. Information buried in the middle of a long, noisy context is measurably easier for a model to underweight or miss than information near the start or end of a tightly-focused prompt. This is often described as the "lost in the middle" effect, and it means a longer prompt isn't just more expensive — it can be actively worse at surfacing the right answer, even when the right chunk is technically present.
This is the cost that turns "safety margin" logic on its head. More context isn't neutral-at-worst. Past a point, it can make the model's job harder while also making you pay more for the privilege.
If sending fewer, better chunks is the goal, you need a way to know which chunks are actually the best ones — vector similarity alone isn't always precise enough to trust for the final cut.
A reranker sits between retrieval and generation: retrieve a wider set of candidates cheaply, score them for relevance with a more precise (and more expensive per-item) model, then send only the top few to the LLM.
This is the same shape as every quality-vs-cost decision earlier in this series — you're moving cost to an earlier, cheaper stage to avoid paying for it at a later, more expensive one. Reranking isn't free. But it's usually cheap relative to the LLM tokens it saves, because scoring candidate relevance is a fundamentally lighter task than generating a full answer.
This is the cost I think is easiest to fall into without noticing, because it looks like progress instead of a symptom.
When retrieval quality is poor, one common instinct is to reach for a bigger, more capable model — reasoning that a smarter model can compensate for messier context. Sometimes it can, partially. But it's an expensive way to patch a cheaper problem.
A bigger model paying attention to bad context is still paying attention to bad context — it's just more expensive while doing it. The fix that actually addresses the problem (better retrieval, reranking, tighter topk) usually costs far less than the fix that just papers over it (a bigger model). This is the retrieval-episode version of Episode 2's lesson: fixing quality upstream is almost always cheaper than compensating for it downstream.