RAG Retrieval Augmented Generation Explained: Complete 2026 Guide

Large language models only know what they learned during training, so they cannot answer questions about your internal documents, recent events, or proprietary data. Retrieval Augmented Generation (RAG) fixes this by fetching the right passages from your own knowledge base and handing them to the model before it writes an answer. This guide explains how RAG works, what it costs, when it is the wrong tool, and how to run it reliably in production.

What is RAG (Retrieval Augmented Generation)?

RAG is a technique that enhances LLM responses by retrieving relevant information from external knowledge bases before generating an answer. Instead of relying solely on the model's training data, a RAG system:

Retrieves relevant document chunks from a knowledge base using semantic or hybrid search
Augments the LLM's prompt with the retrieved context
Generates a response grounded in the provided information, ideally with citations

This reduces hallucinations, gives the model access to current information, and lets it reason over proprietary or domain-specific data that was never in its training set.

How RAG Works: The Technical Pipeline

Step 1: Document ingestion and embedding

Your knowledge base (documents, web pages, database records) is prepared offline:

Chunking — Split documents into passages, usually no more than a few hundred tokens each, often with some overlap so ideas that span a boundary are not lost.
Embedding — Convert each chunk into a vector using an embedding model such as OpenAI's text-embedding-3-small or text-embedding-3-large.
Storage — Store the vectors in a vector database: Pinecone, Weaviate, Qdrant, or pgvector if you already run PostgreSQL.

Step 2: Query processing

When a user asks a question:

Embed the query with the same embedding model used for the documents.
Similarity search finds the most semantically similar chunks in the vector store.
Ranking and filtering orders results, optionally with metadata filters (date, document type, department) or a re-ranker model for better precision.

Step 3: Context augmentation

The retrieved chunks are formatted into the prompt alongside the user's question and instructions to answer only from the provided context:

Context:
[Chunk 1: Company policy on remote work...]
[Chunk 2: Recent update to vacation policy...]

User question: What is our company's remote work policy?

Answer using only the context above. If the answer is not there, say so.

Step 4: Grounded generation

The LLM writes the answer using the retrieved passages, and the system can return the source documents alongside the response so users can verify it.

What RAG Actually Costs: A Worked Example

Embedding is cheap, and it is a one-time cost per document (plus re-indexing when content changes). OpenAI's embeddings pricing gives concrete numbers, using their own assumption of roughly 800 tokens per page:

text-embedding-3-small costs $0.02 per million tokens and produces 1536-dimension vectors (MTEB benchmark score 62.3%).
text-embedding-3-large costs $0.13 per million tokens and produces 3072-dimension vectors (MTEB 64.6%), with the option to shorten vectors via the dimensions parameter to save storage.

For a 10,000-page knowledge base (about 8 million tokens):

Initial indexing with text-embedding-3-large: roughly $1.04. With text-embedding-3-small: roughly $0.16.
Query-time embedding is one short query per question, effectively free.
The real running costs are the vector database hosting and the LLM tokens for generation, since every answer includes several retrieved chunks in the prompt.

The lesson: do not optimise embedding model choice for cost. Optimise it for retrieval quality, because retrieval failures are what make RAG systems answer badly.

RAG vs Fine-Tuning: When to Use Each

These two techniques solve different problems, and the right answer is often both.

Frequently changing knowledge — RAG wins. Update the knowledge base and new documents are retrievable immediately. Fine-tuning means retraining for every update.
Factual accuracy with citations — RAG wins. The answer is grounded in a specific document you can show the user. A fine-tuned model can still hallucinate and cannot cite its sources.
Domain-specific tone, format, or behaviour — Fine-tuning wins. RAG retrieves facts; it does not teach the model how to speak or structure output.
Cost to start — RAG is cheaper. No training runs, no GPU time, and the knowledge base doubles as documentation.
Latency — Fine-tuning is faster per request. RAG adds a retrieval round trip and a longer prompt.

One constraint worth knowing: Anthropic's Claude models are not available for customer fine-tuning. When a project needs fine-tuning, that means open-weight models such as Llama, Qwen, Mistral, or Gemma. Many production systems combine the two: a fine-tuned open-weight model for style and task behaviour, with RAG supplying the facts.

Implementing RAG: Working Code

The older RetrievalQA class is deprecated in current LangChain. The maintained pattern uses create_retrieval_chain with a document-combining chain (in LangChain 1.x these live under the langchain_classic package):

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma(persist_directory="./kb", embedding_function=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer using only the context below. If the answer "
               "is not in the context, say you do not know.\n\n{context}"),
    ("human", "{input}"),
])

docs_chain = create_stuff_documents_chain(ChatOpenAI(temperature=0), prompt)
rag_chain = create_retrieval_chain(retriever, docs_chain)

result = rag_chain.invoke({"input": "What is our return policy?"})
print(result["answer"])

Swap in your preferred chat model and vector store; the pattern stays the same. For a fuller walkthrough, see our RAG developer guide.

Advanced Techniques That Move the Needle

Hybrid search. Embeddings capture meaning but miss exact strings like product codes, error numbers, or regulation names. BM25 keyword search catches those. Combining both covers the failure modes of each; Anthropic's Contextual Retrieval research uses the example of querying "Error code TS-999", which pure semantic search often misses.

Contextual chunk enrichment. A chunk pulled out of a document often loses the context that makes it findable ("the second quarter" of which year, for which product?). Anthropic's Contextual Retrieval prepends a short model-generated explanation to each chunk before embedding it. In their experiments, contextual embeddings plus contextual BM25 reduced failed retrievals by 49%, and by 67% when combined with reranking.

Re-ranking. A cross-encoder or reranker model (Cohere Rerank, or an open model) re-scores the top retrieval results before they enter the prompt. Cheap, and usually the single biggest precision gain after hybrid search.

Query expansion. Generate several phrasings of the user's question and retrieve for all of them, merging the results. Helps when users ask vaguely.

Contextual compression. Strip irrelevant sentences from retrieved chunks before they reach the LLM, reducing noise and token cost.

Where RAG Gets Used

Customer support

RAG answers questions from product documentation, past tickets, and internal knowledge bases, and it can point customers to the exact source article. Pairing retrieval with the practices in our guide to handling AI agent hallucinations in production keeps wrong answers from reaching customers.

Legal and compliance

Law firms and in-house teams use RAG to search case law, contracts, regulations, and internal policy, where source citation is not optional. This is a core part of our Claude for legal implementations.

Financial services

Fund operations, research, and KYC teams use RAG over filings, fund documents, and policy libraries, often inside a private deployment for confidentiality. We build these as part of our Claude for financial services work.

Internal knowledge management

Enterprise RAG indexes wikis, chat history, meeting transcripts, and code repositories so employees get cited answers instead of searching ten systems.

When RAG Is the Wrong Choice

RAG is not the default answer to every knowledge problem.

Small knowledge bases. If your corpus is under about 200,000 tokens (roughly 500 pages), Anthropic recommends skipping retrieval entirely and putting the whole knowledge base in the prompt. Prompt caching makes this fast and cheap: they report latency improvements of more than 2x and cost savings of up to 90% on cached context.
You need style, not facts. If the problem is tone, output format, or task behaviour, fine-tuning or better prompting solves it. RAG only adds knowledge.
Live transactional data. Account balances, stock levels, and order statuses belong behind tool calls to your APIs, not in a vector index that goes stale between syncs.
Relationship-heavy questions. "Which suppliers feed into which products that ship to Kenya?" is a graph query. Knowledge graphs or plain SQL answer it better than chunk similarity.

Common Implementation Challenges

Chunking strategy. Arbitrary splits break ideas in half. Use structure-aware splitting (headings, sections) or overlapping windows, and keep chunks to a few hundred tokens.

Retrieval quality. The right document exists but never gets retrieved. Test embedding models on your own queries, add hybrid search, and use metadata filters before reaching for exotic fixes.

Context window limits. Too many chunks crowd the prompt. Re-rank aggressively, retrieve broadly then zoom in hierarchically, or use a long-context model.

Stale information. The index drifts from reality. Automate re-indexing on document change, version your content, and score documents by freshness where it matters.

Measuring RAG Performance

Evaluate the two halves separately.

Retrieval metrics:

Context precision — are the retrieved chunks actually relevant?
Context recall — did retrieval find all the relevant material?

Generation metrics:

Faithfulness — does the answer stick to the retrieved context?
Response relevancy — does it answer the question that was asked?

The Ragas framework implements exactly these metrics, plus noise sensitivity and agentic evaluations; DeepEval is another common choice. Build a test set of real questions with known correct sources before you tune anything, or you will tune blind.

Best Practices for Production RAG

Log retrieval failures. Track queries where no good chunk was found; they point directly at gaps in your knowledge base.
Close the feedback loop. Let users flag bad answers and feed those cases back into your test set.
Version your embeddings. Changing embedding models changes the vector space. Re-index the whole corpus at once or keep the old index alive until you do.
Use metadata filters. Date, author, document type, and department filters often fix "bad retrieval" faster than model changes.
Test the empty case. Decide what the system says when no relevant document exists, and when two sources contradict each other. Silence is better than a confident guess.

Where RAG Is Heading

Retrieval is becoming a decision an agent makes rather than a fixed pipeline stage. In what the LangChain documentation calls agentic RAG patterns, an agent decides when to search, how to phrase the query, whether the evidence it found is sufficient, and whether to delegate deep reading to subagents with grading rubrics checking that answers stay grounded. Alongside this, multimodal RAG extends retrieval to images, audio, and video, and GraphRAG-style approaches add explicit relationship structure on top of vector search. The fundamentals in this guide (chunking, hybrid retrieval, reranking, evaluation) remain the foundation all of these build on.

Frequently asked questions

What is the difference between RAG and fine-tuning? RAG gives a model access to external knowledge at query time by retrieving relevant documents into the prompt. Fine-tuning changes the model itself through additional training. Use RAG for facts that change or must be cited; use fine-tuning (of open-weight models) for tone, format, and specialised behaviour.

How much does a RAG system cost to run? Indexing is cheap: embedding a 10,000-page knowledge base with OpenAI's text-embedding-3-large costs about $1. The ongoing costs are vector database hosting and LLM generation tokens, which dominate because every answer includes retrieved context. Exact figures depend on query volume and model choice.

Which vector database should I choose? If you already run PostgreSQL, pgvector keeps your stack simple. Qdrant and Weaviate are strong self-hosted options. Pinecone is the common managed choice when you do not want to operate infrastructure. For most teams the choice matters less than chunking and retrieval tuning.

Does RAG eliminate hallucinations? No, it reduces them. The model can still misread retrieved context, and retrieval can return the wrong passages. Grounding instructions, citations, faithfulness evaluation, and human review for high-stakes answers are still necessary.

When can I skip RAG entirely? When your knowledge base fits comfortably in the model's context window, under roughly 200,000 tokens or about 500 pages, putting the whole corpus in the prompt with caching is simpler and often more accurate. Add RAG when the corpus outgrows that.

Where to go from here

If you are weighing RAG against fine-tuning or a private model deployment for a bank, insurer, or regulated enterprise, the architecture decisions (chunking, hybrid search, evaluation, governance) matter more than the tooling. Our enterprise AI service covers exactly this: RAG systems, open-weight fine-tuning, private deployment, and the evaluation and governance documentation that risk teams need.

RAG Retrieval Augmented Generation Explained: The Complete Guide for 2026