What is RAG? Retrieval-augmented generation explained

A large language model is confident, fluent, and completely unaware of your business. Ask it about your pricing, your internal runbooks, or last quarter’s numbers and it will answer just as smoothly as it does about anything else; it just won’t be right. The knowledge it has is frozen at training time and drawn from the public internet, not from your systems. Retrieval-augmented generation, or RAG, is the pattern that closes that gap: it lets the model look things up in your data before it answers.

RAG has quietly become the default architecture for enterprise LLM applications, and for good reason. It is the cheapest, fastest, and most auditable way to put a model to work on knowledge it never saw during training. This piece explains what RAG actually is, why it beats the alternatives, how each stage of the pipeline works, and where the real engineering effort goes once you move past the demo.

What retrieval-augmented generation actually is

RAG is a simple idea with a lot of engineering behind it. Instead of relying only on what a model learned during training, you give it a way to fetch relevant information from an authoritative source at question time, then answer using that retrieved context. The model still does the reasoning and the writing; the facts come from you.

Mechanically, you sit a retrieval system in front of the model. When a question arrives, you search your own content for the passages most likely to answer it, paste those passages into the prompt, and instruct the model to answer from them. The model’s parametric knowledge (what it memorised in training) is now backed by non-parametric knowledge (what you retrieved at runtime). That separation is the whole point, and it fixes the failure modes that make raw LLMs risky in production:

• Hallucination: grounded in retrieved passages, the model has far less room to invent. It is answering an open-book exam rather than reciting from memory.
• Stale knowledge: a model’s training has a cutoff date, and everything after it is invisible. Update the underlying documents and the answers update with them, with no retraining and no redeploy.
• No provenance: because you know exactly which passages were retrieved, you can return citations and let users verify claims against the source. For regulated, legal, or medical work, that traceability is often non-negotiable.
• Access control: knowledge lives in a store you own, so you can filter retrieval by who is asking. A model that memorised the data cannot forget it for one user; a retrieval layer can simply not return it.

Why not just retrain or fine-tune the model?

This is the question every team asks first, so it is worth answering properly, because the three approaches solve different problems.

Retraining or continued pre-training a foundation model on your data is enormously expensive and slow, needs a serious ML team and GPU budget, and has to be repeated every time the data changes. For a knowledge base that shifts weekly, that is a non-starter.

Fine-tuning is far cheaper, but it teaches the model form far better than facts: tone, format, house style, how to follow a particular kind of instruction. It is poor at injecting specific, changing knowledge, and it bakes whatever it learns into the weights, so a correction means another training run. Worse, fine-tuning on a narrow corpus can cause the model to forget general capability.

RAG keeps the knowledge outside the model, where it belongs. Your data stays in a system you control, you can change it in seconds, and you can enforce who is allowed to see what. Adding a document is an insert, not a training job. For most “answer questions over our own content” problems, that is the right architecture.

These are not mutually exclusive. A common production pattern is to fine-tune a smaller model for behaviour (answer in this format, always cite, refuse politely when unsure) and use RAG for knowledge. Fine-tuning shapes how the model talks; RAG decides what it knows.

How RAG works, step by step

Under the hood, a RAG system is a retrieval pipeline bolted to a generation step. There is an offline half that prepares the knowledge and an online half that answers questions. Four stages matter.

1. Prepare and index the knowledge base

Take your source material; documents, wiki pages, tickets, database rows, PDFs, and split it into chunks. Chunking is not a formality: chunk too large and you dilute the relevant sentence with noise and blow your token budget; chunk too small and you strip away the context needed to understand it. Split on semantic boundaries such as headings, sections, and paragraphs rather than a fixed character count, and keep a little overlap between chunks so a sentence that straddles a boundary is not orphaned. Carry metadata with each chunk (source, title, author, date, permissions); you will use it later for filtering and citations.

Each chunk is then passed through an embedding model, which converts its meaning into a vector, a list of numbers positioned so that texts with similar meaning sit close together in space. Those vectors go into a vector database (or a vector index inside your existing store) that can answer “which chunks are nearest to this point?” quickly, using an approximate nearest-neighbour algorithm so it stays fast over millions of vectors.

2. Retrieve the relevant context

When a user asks something, embed the query with the same model and search the index for the nearest chunks; the passages most similar in meaning, not merely in wording. This is where naive systems quietly fail, so production retrieval usually goes further:

• Hybrid search combines dense vector similarity with a keyword (BM25) method. Vectors handle paraphrase and synonyms; keyword search nails exact terms, product codes, and rare names that embeddings blur together.
• Metadata filtering restricts the search up front (this customer, documents newer than a date, only content this user may see), which is both a relevance win and your access-control boundary.
• Re-ranking retrieves a broad candidate set, then uses a cross-encoder to score each candidate against the query and keep only the best few. It is more accurate than the first-pass similarity, and cheap because it runs on a shortlist rather than the whole corpus.

3. Augment the prompt

Assemble the retrieved passages into the prompt alongside the user’s question, with a system instruction to answer only from the supplied context and to say when the context does not contain the answer. Ordering and budget matter: models attend more reliably to the start and end of a long context than the middle, so put the strongest passages where they will be read, and keep the total within the context window with headroom for the answer. Give each passage an identifier so the model can cite it. A minimal template:

Answer the question using only the context below.
Cite the source id in square brackets after each claim.
If the context does not contain the answer, say you don't know.

Context:
[doc-12] {retrieved_passage_1}
[doc-07] {retrieved_passage_2}

Question: {user_question}

4. Keep the knowledge base fresh

A RAG system is only as current as its last index update. Documents change, get added, and get deleted, so re-embed and re-index on a schedule or, better, on write, and make sure deletions propagate so the model cannot cite a document that no longer exists. Watch for drift as your embedding model, chunking strategy, or content mix evolves; a reindex is the moment to re-run your evaluation set and confirm quality has not slipped.

RAG vs semantic search

People sometimes conflate the two, but they sit at different layers. Semantic search is the retrieval half; the technology that scans your knowledge base and returns passages by meaning rather than exact keywords, ranked by relevance. It answers “which passages relate to this query?” and hands you a list.

RAG uses semantic search (usually alongside keyword search) and then adds the generation step on top, turning that list of passages into a single written, cited answer in natural language. Put simply: semantic search finds; RAG finds and then explains. Good retrieval is what makes RAG work, so a strong semantic search layer is a prerequisite, not a competitor. If the wrong passages come back, the most capable model in the world has nothing useful to ground on, and a confident wrong answer is the result.

Where RAG earns its keep, and where it bites

RAG shines wherever the value is in your own content: internal knowledge assistants, customer support over your documentation, search across contracts or research, question answering that must stay current and cite its sources. Anywhere the honest answer is “it depends what our documents say,” RAG is the natural fit.

The honest caveat is that RAG moves the hard problem rather than removing it. The quality of the answer is capped by the quality of retrieval, and retrieval quality depends on unglamorous work:

• Retrieval failures are the top cause of bad answers; the right chunk was never fetched, so no prompt wizardry can save the response. Measure retrieval on its own with a labelled set of questions and the passages that should answer them, and track recall before you touch the model.
• Evaluation has to be systematic. Build a test set of real questions with expected behaviour and run it on every change, scoring both retrieval (did we fetch the right passages?) and generation (was the answer grounded and correct?). Without this you are tuning blind.
• Observability means logging the query, the retrieved chunks, and the final answer, so that when a user reports a bad response you can see whether retrieval or generation was at fault.
• Governance is a genuine advantage of RAG but only if you use it: enforce permissions at retrieval time, keep an audit trail via citations, and prevent sensitive chunks from ever entering a prompt.
• Cost and latency add up; every question now involves an embedding call, a vector search, often a re-rank, and a larger prompt. Caching, sensible top-k limits, and right-sizing the model keep it economical at scale.

A demo takes an afternoon; a system you can trust in front of customers takes real engineering. The teams that succeed treat retrieval quality as a metric they monitor, not a box they tick once.

Takeaway

RAG is the most reliable way to make a language model useful on data it was never trained on, without the cost of retraining and with the provenance that regulated and high-stakes work demands. Treat it as a data problem first and a model problem second: get the chunking and retrieval right, keep the knowledge base fresh, enforce access at the retrieval layer, and measure both halves of the pipeline. Do that and the model becomes the easy part. If you are weighing RAG for your own data, our AI and machine learning and data engineering work is exactly this, and we are happy to talk it through.