Retrieval augmented generation in production, an architecture that survives contact with real data

Retrieval augmented generation is the idea that, instead of trying to bake all of your knowledge into a model's weights, you keep the knowledge in systems that are good at search, retrieve the most relevant pieces at the moment of a question, and let the model compose an answer from them. The technique was introduced by Patrick Lewis and colleagues in their 2020 paper, Retrieval Augmented Generation for Knowledge Intensive Natural Language Processing Tasks, and it has since become the default way to ground a language model in private or current data.

The reason it has become the default is practical. Fine tuning a model on your corpus is expensive, slow to update, and hard to audit. Retrieval keeps the knowledge outside the model, where you can govern it, version it, and change it without retraining anything. The catch, and the reason so many projects stall after the demo, is that the naive version fails at the retrieval step a great deal of the time. Industry write ups in 2026 routinely report that simple pipelines return the wrong supporting passages for a large share of real questions, and when that happens the model still produces a confident, well structured answer. It is just grounded in the wrong material. In production the retrieval step, not the generation step, is the bottleneck.

The architecture, layer by layer

Here is the shape of a pipeline that holds up under real traffic. Each stage exists to fix a specific way the naive version breaks.

Ingestion and indexing

Everything downstream depends on how you cut your documents into passages. Chunks that are too large dilute the signal, and chunks that are too small lose the context that makes them meaningful. You attach metadata to every passage, such as its source, its date, and its access level, because that metadata is what later lets you filter, cite, and govern. The passages are embedded into vectors and also indexed for keyword search, so that you can later combine both kinds of matching.

Retrieval, the most common point of failure

The single most common failure is the semantic gap. A user asks how to cancel a subscription, and the relevant document is titled account termination policy, so a pure vector search that relies on meaning can still miss it, while a pure keyword search misses paraphrases. The fix that the industry has converged on is hybrid retrieval, which runs both semantic and keyword search and combines the results. You retrieve far more candidates than you intend to use at this stage, because it is cheaper to over retrieve and then filter than to miss the one passage that matters.

Reranking and compression

Casting a wide net means most of what you retrieve is noise, and a context window full of loosely relevant passages produces a mediocre answer because the model averages across all of it. So a second pass uses a cross encoder, a model that reads the question and a candidate passage together and scores how well they actually match, to push the best handful to the top and discard the rest. Compression then strips the irrelevant sentences out of the passages that remain. These steps add latency, and the accuracy they buy is almost always worth it.

Generation and citation

Only now does the language model see anything. It receives the compressed, highly relevant passages and is asked to compose an answer and to cite which passage each claim came from. The citation step is not decoration. It is how a reviewer, or an auditor, or the user can check that the answer is actually grounded rather than confidently invented.

When a flat index is not enough

For questions that require connecting facts that live in different documents, a flat list of passages cannot help, because it has no notion of how passages relate to one another. This is the case for graph based retrieval, where the knowledge is organized as entities and the relationships between them. Microsoft's research on this approach, published as a graph based method for query focused summarization, showed meaningful gains on complex analytical questions that require reasoning across many connected facts. The honest tradeoff is that building and maintaining a graph is more work, and for a straightforward question and answer use case it can add noise rather than value. The guiding principle is to start with the simplest pipeline that could work and add complexity only when your evaluation numbers tell you to.

Evaluation, and why it cannot be skipped

The reason retrieval failures are dangerous is that they are invisible from the outside. The answer reads well. The only way to know whether your system is reliably correct rather than reliably fluent is to measure it, continuously, against a held out set of questions with known good answers. Open frameworks such as RAGAS score the things that matter, including whether the retrieved passages were actually relevant and whether the final answer is grounded in them rather than in the model's imagination. Treating evaluation as a first class part of the system, built in from the very beginning rather than bolted on at the end, is the single habit that most separates teams who ship reliable retrieval systems from teams who ship demos.

Summary

Retrieval augmented generation is only as good as the context it can see. The model is downstream of the retriever, the retriever is downstream of the index, and the index is downstream of how well you governed and prepared your knowledge in the first place. If the source is stale, ungoverned, or poorly chunked, no amount of model cleverness will save the output. Spend your engineering effort where the quality actually lives, which is in retrieval, in reranking, and in the evaluation that tells you the truth about both.