What Is RAG in AI? Retrieval-Augmented Generation Explained, and When Not to Use It

The term comes from a 2020 paper by Lewis and colleagues at Facebook AI Research, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which paired a pre-trained generator with a dense vector index of Wikipedia. Six years on, the question is whether retrieval fits your problem, and whether yours is good enough to trust.

What is RAG in AI, in one paragraph?

A large language model (LLM) answers only from what it saw in training, so it cannot see your contracts, tickets or manuals. RAG splits a corpus into chunks, embeds each chunk as a numeric vector, and indexes them. At query time it retrieves the chunks closest to the question and puts them in the prompt. The weights never change; knowledge lives outside the model, refreshed by re-indexing.

• RAG was introduced by Lewis et al. in 2020 as a pre-trained generator paired with a dense retrieval index, and it set state-of-the-art results on open-domain question answering.
• A RAG pipeline has two halves: offline indexing that chunks and embeds documents, and online retrieval that answers from the nearest chunks.
• Practitioners treat RAG as a search problem first: Hamel Husain scores retrieval with recall and precision at k, and Jason Liu singles out recall, because a chunk that never surfaces cannot be recovered later.
• RAG reduces hallucination but does not remove it, so retrieval quality and answer faithfulness must be measured separately.
• RAG is not always the right tool: Anthropic’s engineering guidance puts the threshold at about 200,000 tokens, roughly 500 pages, below which a knowledge base can go in the prompt with no retrieval at all.
• Fine-tuning changes how a model behaves; RAG changes what it knows at answer time.

How does RAG work, step by step?

Every production RAG system is a variation on these eight steps.

1. Collect the corpus, recording ownership and permissions per source.
2. Chunk the documents into passages, since boundaries decide what can be retrieved as a unit.
3. Embed each chunk, producing one vector per chunk.
4. Store the vectors, chunk text and metadata such as source, date and access rights.
5. Embed the user question with the same embedding model.
6. Retrieve the top-k chunks by similarity, usually merged with keyword search and reranked.
7. Augment the prompt with the chunks, the question and instructions to cite sources and refuse when context is thin.
8. Generate the answer, with citations back to the retrieved chunks.

How should you chunk documents?

Pinecone’s chunking guide lists the strategies: fixed-size windows, sentence or paragraph splitting, structure-aware splitting on headings and markup, semantic chunking that breaks where sentence embeddings shift topic, and contextual chunking that prepends a generated summary. The tradeoff that matters: chunks must be big enough to carry meaning and small enough to keep retrieval precise. LlamaIndex’s production guide adds the move most teams miss: decouple the chunk you retrieve from the chunk you send to the model, because the best unit for matching a query is rarely the best unit for answering it.

What are embeddings and vector databases?

An embedding is a vector, a list of floating point numbers, that represents meaning so related texts sit close together. OpenAI’s embeddings documentation describes text-embedding-3-small at 1,536 dimensions and text-embedding-3-large at 3,072, and notes that because the vectors are normalized, cosine similarity is a dot product. Chunks and queries must use the same model, so switching means re-indexing. A vector database indexes those vectors for fast nearest-neighbor search. pgvector, the Postgres extension, offers HNSW, fast to query and slow to build, and IVFFlat, the reverse, and is honest that an approximate index trades recall for speed. Keeping vectors, metadata and permissions in one database simplifies filtering by tenant before search.

Dense retrieval, keyword retrieval, or both?

The retriever in the original paper was Dense Passage Retrieval (DPR), described by Karpukhin et al. in 2020, which trains separate question and passage encoders and beat the keyword baseline BM25 by 9 to 19 percentage points in top-20 accuracy. Dense retrieval handles paraphrase well; Eugene Yan’s patterns for LLM systems notes it struggles with names, acronyms and identifiers, and that hybrid retrieval beats either alone. Merge both ranked lists with rank fusion or a weighted score, the alpha parameter in Weaviate’s advanced RAG guide. Anthropic’s contextual retrieval experiments show how much detail matters: prepending generated context to each chunk before embedding cut the retrieval failure rate from 5.7% to 3.7%, contextual BM25 to 2.9%, and reranking to 1.9%, a 67% reduction.

Why add a reranker?

An embedding compresses a chunk’s every possible meaning into one vector before it has seen any query, so information is lost. A cross-encoder reranker reads query and chunk together, which is more accurate but far slower: Pinecone’s reranker guide estimates that a small BERT reranker scoring 40 million records would need over 50 hours per query on a V100 GPU, against under 100 milliseconds for vector search. Retrieve a few dozen candidates fast, then rerank those. Hybrid search plus a reranker is the baseline to beat, not an optimization to postpone. Late interaction models such as ColBERT keep token-level vectors and sit between the two.

Why do practitioners call RAG a search problem?

The people who run RAG in production write about retrieval, not prompts. Hamel Husain judges retrieval with information-retrieval metrics: recall at k, how many relevant documents landed in the top k; precision at k, how many of those k were relevant; and mean reciprocal rank, how high the first one sat. Ben Clavié’s talk adds that what became obsolete is naive single-vector search, not RAG: BM25, ColBERT and rerankers are all retrieval.

Jason Liu argues that RAG is more than embedding search. A question embedding and the right chunk’s embedding are not always close, one query string cannot express a date range or an owner, and most companies have several backends. The answer is query understanding. LangChain’s write-up on query transformations names the standard moves: rewrite-retrieve-read for questions never phrased for search, step-back prompting for a broader question alongside the specific one, multi-query retrieval that fans one question into several and merges the results with reciprocal rank fusion, and conversational rewriting for follow-ups.

Filters and routing carry the rest. In Systematically Improving Your RAG, Liu names the biggest mistake as tuning synthesis before checking whether the right data is retrieved at all, and prescribes extracting metadata such as dates, versions and owners so filters can do what similarity cannot. LlamaIndex calls this structured retrieval. Routing applies the idea across indices: treat each index as a tool, classify which one a query needs, then measure precision and recall per tool rather than for the system as a whole.

What does a production RAG architecture look like?

The Gao et al. survey maps RAG architecture into three paradigms: naive RAG, the eight-step pipeline with no refinements; advanced RAG, which adds query rewriting, metadata filtering, reranking and compression; and modular RAG, which treats retrieval, memory, routing and generation as swappable parts. A production architecture has these parts:

• An ingestion service that re-embeds only what changed, so the index never drifts from the documents.
• A query layer that rewrites the question, infers metadata filters and routes it to the right index.
• A hybrid retriever and a reranker, with permission filters applied before ranking, not after.
• A prompt that separates instructions, context and question, and permits refusal when context is insufficient.
• A citation layer mapping each claim back to a chunk identifier.
• Tracing on every request. OpenTelemetry’s generative AI semantic conventions define standard attributes for the model, token counts, the retrieval query and the documents returned, so one trace shows which chunks produced an answer and what it cost.

RAG vs fine-tuning: which should you use?

Fine-tuning continues training a model on your own examples so the weights change. RAG leaves the weights alone and changes what the model sees at answer time.

• Choose RAG for facts that change, are private or must be cited. Re-indexing is cheaper than retraining, and a retrieved passage gives the user a source.
• Choose fine-tuning to change behavior: a strict output format, a house style, or a domain vocabulary the base model mangles.
• Do not fine-tune to inject facts. A fine-tuned model cannot cite a fact or update it without retraining.
• Neither fits when the question is really a database query: the balance on account 4471 belongs to the accounts service.

Does RAG stop hallucination?

No. Lewis et al. measured RAG output as more factual than a parametric-only baseline, but RAG has more places to fail than a plain model, and each produces a confident wrong answer:

• Retrieval miss: the right passage exists but missed the top-k, because the question was phrased differently, a boundary split the answer, or the index was stale.
• Wrong passage, right topic: last year’s policy outranks this year’s. Filters on date and version are the fix, not a bigger model.
• Context ignored: Lost in the Middle found accuracy highest when the relevant passage sits at the start or end of the context and lower in the middle, and Chroma’s context rot study of 18 models found each added distractor hurts further. Fewer, better-ranked chunks beat a long list.
• Conflicting sources: two chunks disagree and the model picks one silently. Surface the conflict.
• Poisoned corpus: any document users can edit can carry instructions. Retrieved text is data, never instructions.

How do you evaluate a RAG system?

RAG evaluation measures retrieval and generation separately, because a bad answer can come from either. RAGAS, by Es et al. scores context relevance, faithfulness and answer relevance with a judge model and no reference answers, so it runs on every code change. Jason Liu’s There Are Only 6 RAG Evals generalizes this: given a question, a context and an answer, there are exactly six relationships to check, with retrieval precision and recall as the foundation tier. A practical plan has four layers:

1. Retrieval metrics on a synthetic question set. Ask a model for questions answerable by each chunk, then measure how often that chunk comes back. LlamaIndex’s RetrieverEvaluator scores hit rate, mean reciprocal rank, precision, recall and NDCG, with no labels needed.
2. Faithfulness and answer relevance by a judge model, with the judge itself validated. Eugene Yan’s product evals recipe is to label 200 or more samples containing at least 50 to 100 failures, align an evaluator per criterion on three quarters of them, and require a Cohen’s kappa of 0.4 to 0.6 against human labels.
3. Refusal quality on questions the corpus cannot answer. A system that never refuses is hallucinating on that set.
4. Error analysis on logged failures, covered next.

Two cautions. Clavié notes that public benchmarks such as BEIR and MTEB are now in training data, so leaderboard scores are a weak signal for your corpus. And the TREC 2025 RAG track, with over 150 submissions, scored attribution and completeness alongside relevance, the bar a cited assistant should meet. Sigi’s guide to building an AI chatbot makes the same point: measure faithfulness against the corpus, not thumbs-up alone.

How do you decide what to fix next?

Scores tell you the system is wrong. Error analysis tells you what to change, and Hamel Husain calls it the single most valuable activity in AI development. The method is bottom-up: read real traces, note any undesired behavior, have a model group those notes into a taxonomy of failure modes, then count how often each fires. In one product he documents, three issues caused over 60% of all problems, and fixing the worst took date handling from 33% to 95% correct. Eugene Yan argues the same against reaching for another judge model: look at the data, annotate wins and failures, inspect the retrieved documents and reasoning traces, then test a hypothesis against a baseline.

Then group the failures by topic. Liu’s systematic approach to RAG clusters real queries with k-means or model-based topic labeling and asks, for each weak cluster, whether the problem is inventory or capability: inventory means the answer is not in the corpus, so ingest it; capability means it is there but retrieval cannot surface it, so fix search, filters or metadata. Those are different projects, and an aggregate score hides which one you have. His improvement flywheel adds two habits: keep a catch-all category in the classifier and watch its share as an early signal of drift, and track retrieval experiments per week as a leading indicator, not only a lagging quality score.

When should you not use RAG?

RAG adds an index, a retrieval system, an evaluation harness and latency. Each is worth it only when the alternative is worse.

• The corpus is small. Anthropic’s guidance is that a knowledge base under roughly 200,000 tokens fits in a cached prompt. No index, no retrieval misses.
• Accuracy matters more than cost and the documents fit the context window. Li et al. in 2024 found long-context models outperformed RAG on average when resourced sufficiently, RAG’s much lower cost being its advantage, and their Self-Route method lets the model choose per query. A 2025 evaluation agreed for Wikipedia-style questions while finding RAG stronger on dialogue.
• The problem is behavior, not knowledge: format and tone are prompt or fine-tuning problems.
• The answer is a computation. Totals, balances and eligibility belong in a tool call against the system of record.
• Nobody maintains the corpus. RAG over stale documents gives confident answers from stale text.

What is agentic RAG?

Agentic RAG is a RAG system in which an agent controls the retrieval loop instead of running one fixed retrieve-then-generate pass. The 2025 survey by Singh et al. describes agents applying reflection, planning, tool use and collaboration: the agent reads what came back, decides whether it answers the question, rewrites or splits the query if not, and may switch source. LangChain’s retrieval docs draw the line by workload: two-step RAG has predictable latency and suits FAQ bots, agentic RAG suits research assistants with several tools.

It helps on multi-hop questions, costs more per query, and is harder to evaluate because the path varies. AgenticRAGTracer, a 2026 benchmark of 1,305 multi-hop items, publishes the intermediate hop-level questions so you can see which step failed. It reports failures from reasoning chains that collapse early or over-extend, with GPT-5 at 22.6% exact-match accuracy on its hardest slice. A 2025 survey of reinforcement-learning search agents shows where research is heading. Make single-pass RAG measurably good first, then add agentic control only for the query clusters that need it.

What is GraphRAG and when do you need it?

GraphRAG is retrieval over a knowledge graph built from the corpus rather than over flat chunks. The Microsoft paper From Local to Global targets global questions such as the main themes in a dataset, which chunk retrieval cannot answer: a model extracts an entity graph, communities of entities get pre-generated summaries, and a query combines partial answers from those summaries. On million-token corpora it reported substantial gains over conventional RAG in comprehensiveness and diversity, and a 2025 survey frames it as an answer to knowledge spread across sources. The cost is an LLM-driven indexing step that must be re-run as documents change, and a 2026 benchmark under agentic search found that agentic search substantially improves dense RAG and narrows the gap, while GraphRAG keeps an advantage on complex multi-hop reasoning once its offline cost is amortized. Reserve it for whole-corpus sensemaking, not lookup.

What does a RAG assistant typically cost to build?

Scope drives cost: how many sources are indexed, whether permissions differ per user, and how strict the faithfulness bar is. Sigi does not publish client invoices and has no RAG case study to attach a figure to. The tiers below are typical-scope planning estimates aligned with the bands in Sigi’s guide to how much it costs to build a mobile app.

Running cost is dominated by tokens per query, since retrieved chunks are input tokens on every call. Fewer chunks, prompt caching and the smallest model that passes the evaluation set are the levers Sigi describes on its LLM optimization page. Eugene Yan is skeptical of caching answers on semantic similarity alone, which pays only when queries follow a power law, and a 2026 study of grounded cache routing found a naive cache served unsafe answers 15 to 35% of the time on a multi-hop benchmark, and 51.5% once source documents drifted, unless reuse was gated on evidence overlap and source version.

Related reading

For the product side of the same system, read the guide to how to build an AI chatbot for your business. Sigi Technologies describes grounded question answering over private documents on its RAG assistants page, cost and reliability work on LLM optimization, task-completing systems on AI agent development, and the wider practice on AI development services. To discuss a corpus and whether retrieval fits, contact Sigi.