Why Your RAG System Isn't Accurate — and How to Fix Retrieval Quality

The single most useful diagnostic step is also the one teams skip: for a set of questions the system gets wrong, look at what was actually retrieved before the LLM saw it. This immediately splits every RAG failure into one of two categories, and they have completely different fixes.

If the correct information was not in the retrieved chunks, you have a retrieval problem — the model never had a chance, and improving prompts or switching to a bigger model will not help. If the correct information was retrieved but the model still answered wrong, you have a generation problem — the context was there but the model ignored it, misread it, or the prompt didn't instruct it to rely on it.

In practice, the large majority of 'inaccurate RAG' cases are retrieval problems, which is why chasing prompt and model changes so often fails to move the needle. Logging retrieved context alongside every answer (the same observability habit from the LLM integration guide) is what makes this diagnosis possible at all.

Naive fixed-size chunking (split every document into equal character blocks) is the default in most tutorials and the root cause of a large share of retrieval failures. It cuts sentences, tables, and logical sections in half, so the chunk that gets retrieved contains a fragment of the answer without the context needed to use it — or the answer spans two chunks and only one gets retrieved.

Better chunking respects the structure of the document: split on headings, sections, or semantic boundaries rather than arbitrary character counts, keep tables and lists intact, and use chunk overlap so a fact near a boundary isn't orphaned. For structured or technical documents, chunking strategy matters more to final accuracy than almost any other single choice.

This is exactly the kind of decision a proof of concept is meant to surface — testing chunking against real, representative documents before building on top of it, as covered in the RAG POC guide, rather than discovering the chunking is wrong after the system is in production.

Pure vector (semantic) search retrieves chunks that are broadly similar in meaning, but the top result by embedding similarity isn't always the most relevant to the specific question. A reranking step — running the retrieved candidates through a cross-encoder or reranker model that scores each against the query — reorders them so the genuinely most relevant chunks land at the top, where they belong in the prompt. Adding reranking is one of the highest-leverage accuracy improvements for an already-working RAG system.

Hybrid search combines semantic search with keyword (exact-match) search, which matters when users search for specific terms the embedding might gloss over — product codes, names, error messages, precise numbers. If your failures cluster around exact-match queries that semantic search fuzzes over, adding a keyword component (built into Weaviate, addable to pgvector and Pinecone setups — see the vector database comparison) directly addresses that failure mode.

A related, cheap lever is tuning how many chunks you retrieve: too few and the answer isn't present, too many and the relevant chunk gets buried in noise the model has to wade through (which also inflates cost — see reducing LLM API costs). The right number is found by testing, not guessed.

The user's raw question is often a poor search query. Short, ambiguous, or conversational questions ('what about the second one?') retrieve poorly because they lack the terms that would match the right chunks. Query transformation — rewriting or expanding the question before retrieval — closes this gap.

Common techniques: query rewriting (using an LLM to turn a conversational question into a standalone, well-formed search query), query expansion (adding synonyms or related terms), and for follow-up questions in a conversation, rewriting the query to include the context of what was discussed before so 'the second one' becomes an explicit, searchable phrase.

These techniques add a small amount of latency and cost per query, so they're worth adding when your diagnosis shows retrieval is missing relevant chunks specifically because the incoming queries are underspecified — not as a blanket default before you know that's the problem.

When your diagnosis shows the right context was retrieved but the answer was still wrong, the fixes are on the generation side. The most common is a prompt that doesn't firmly instruct the model to answer only from the provided context and to say when the answer isn't there — without that instruction, models blend retrieved context with their training knowledge and produce confident, subtly wrong answers.

Explicitly instructing the model to ground answers in the retrieved passages, to cite which passage each claim comes from, and to respond with 'I don't have that information' when the context doesn't cover the question, sharply reduces hallucination. Asking for citations also makes wrong answers easy to catch, because you can check the claim against the cited source.

If the model is genuinely misreading correctly-retrieved context on hard questions, that's the narrow case where a more capable model earns its cost — but reach for it only after confirming, through the diagnostic in the first section, that retrieval is actually delivering the right context and the failure is truly in generation.

Every change above should be validated against a fixed evaluation set — a list of representative questions with known correct answers or expected retrieved sources — run before and after each change. Without this, it's impossible to know whether a tweak helped, hurt, or did nothing, and RAG tuning degenerates into changing things and hoping.

Measure retrieval and generation separately: for retrieval, whether the correct chunk appears in the top results; for generation, whether the final answer is correct given the retrieved context. Separating the two tells you exactly which stage a regression came from and prevents a retrieval fix from being masked by a generation issue or vice versa.

This is the same evaluation discipline used to validate a RAG proof of concept, run continuously — it's what turns RAG accuracy from a subjective 'seems better' into a number you can actually move and defend, and it's the foundation any serious RAG or LLM engagement should be built on.