How RAG changed generative AI's enterprise problems

Retrieval-augmented generation (RAG) is an architecture that retrieves relevant external information at the moment a question is asked and gives it to a large language model as context, instead of relying only on what the model learned during training.

RAG addresses five problems with one mechanism: the knowledge cutoff, private enterprise data, hallucination, the cost of large document collections, and source attribution. RAG reduces hallucination without removing it, and a RAG answer is only as good as the data pipeline behind it.

Topics covered: retrieval, grounding, vector search, and fine-tuning.

Retrieval-augmented generation (RAG) is an architecture that retrieves relevant external information at the moment a question is asked and gives it to a large language model as context, instead of relying only on what the model learned during training.

A large language model can write code, summarize documents, and reason over instructions. It does not know a company's latest sales numbers, internal policies, customer records, or private documents, because none of that was in its training data. RAG closes that gap by moving the knowledge out of the model and into a source the model can read from at query time.

The table below lists the problems that one change addresses. Each row is covered in its own section.

Retrieval separates a model's frozen training knowledge from an organization's live knowledge

The knowledge cutoff problem is the limitation that a large language model only knows what existed in its training data up to a fixed point in time. Anything created or changed after that point does not exist for the model unless it is supplied in the request.

Enterprise data changes far faster than models are retrained. Take a product price that moves from $100 on Monday to $120 on Friday. A model trained or fine-tuned on Monday's price still answers $100 on Friday.

How does RAG keep answers current?

RAG retrieves current information from an external knowledge base at query time instead of expecting the model to have memorized it. A question about the company's leave policy is answered in three steps: the system searches company documents, retrieves the latest HR policy, and sends the policy and the question to the model together. When the policy changes, the team updates the knowledge base. The model stays as it is.

Does RAG require retraining the model when data changes?

No. RAG keeps a model's learned ability separate from an organization's current knowledge. Updating a document in the knowledge base changes what the retriever returns, with no retraining of the underlying model.

A retriever connects private enterprise data to a model that was never trained on it

A retriever is the component of a RAG system that searches a knowledge base and returns the passages most relevant to a given question. It is what makes a company's own data reachable by a general-purpose model.

Foundation models are trained on broad, publicly available datasets. A business runs on private data:

• Customer records and support tickets

• Financial reports and legal contracts

• Product specifications and engineering documentation

• Employee policies and internal wikis

None of that is in a foundation model's training set. In a RAG system, the enterprise data is organized into a knowledge base, the retriever finds the passages relevant to a question, and only those passages are passed to the model as context. The model never has to be trained on the company's contracts or support tickets. It only needs the retriever to hand it the right ones.

Why can't a foundation model answer questions about a company's private documents?

A foundation model is trained on broad, publicly available data and has never seen an individual company's contracts, policies, or support tickets. RAG supplies that private information at query time, so the information does not need to be part of the model's training data.

Grounding reduces hallucination and does not remove it

Grounding is the practice of supplying a model with retrieved evidence before it generates an answer, so the answer is drawn from a document and not from the model's unsupported internal knowledge.

A large language model generates tokens from learned patterns, which is why it can state something confidently that is not true. Asked about a leave policy with no supporting context, a model might answer "employees receive 30 days of annual leave" when the actual policy says 24. Both answers would read equally well.

With grounding, the system hands the model the relevant passage and asks it to answer from that. Given a retrieved passage stating that employees are entitled to 24 days of annual leave, the model's output follows the evidence.

Why does a grounded model still hallucinate?

Grounding only helps when the evidence is right. If the retriever returns an irrelevant or wrong document, the model can still produce an incorrect answer with the same confidence, now based on the wrong evidence. A correct answer needs both good retrieval and good generation.

Does RAG eliminate hallucination?

No. RAG reduces hallucination by grounding an answer in retrieved evidence. If the retriever surfaces the wrong document, the model can still generate a confident, incorrect answer from it, so retrieval quality and generation quality both decide the result.

Retrieval makes large document collections usable without filling the context window

A context window is the maximum amount of text a large language model can read in a single request. Everything the model uses to answer has to fit inside it.

A company with 10,000 PDFs, 50,000 support tickets, an internal wiki, and separate engineering, HR, and legal documentation cannot send all of that with every request. Even where a larger context window could hold more of it, most of what was sent would have nothing to do with the question.

What is vector search?

Vector search is a retrieval method that converts a question and the stored documents into numerical embeddings and returns the passages whose embeddings sit closest to the question's. It matches on meaning, so a question about "time off" can find a passage about "annual leave".

The retriever uses vector search, keyword search, or both to find the top matching chunks in the full collection. Only those chunks go to the model.

Does a bigger context window remove the need for retrieval?

No. A very large context window still cannot hold an entire enterprise archive for every question. Retrieval narrows a large collection down to the passages relevant to one question, and that step is needed however much context a model can accept.

Sending less context lowers inference cost, and retrieval adds costs of its own

Inference cost is what a team pays each time a model processes a request, and it rises with the number of tokens sent in and generated. A system that pushes a large document set into every request pays for all of those tokens every time, whether or not they were relevant to the question.

RAG changes where the money goes. Documents are embedded and indexed once. For each question, the retriever returns a small number of relevant chunks, and only those are sent to the model. The per-request token bill drops because the model reads a few passages and not the full collection.

The savings are not free. A RAG system adds costs that a plain model call does not have:

• Embedding the collection, and re-embedding documents when they change

• Hosting and querying the vector or search index

• Reranking and other retrieval steps that run before the model call

A team should compare both sides for its own request volume and collection size before calling RAG the cheaper design.

How does RAG reduce LLM inference cost?

RAG reduces inference cost by sending only the chunks relevant to a specific question into the model's context, instead of an entire document set on every request. RAG also adds indexing and retrieval costs, so the net saving depends on request volume and collection size.

Source attribution makes an answer traceable

Source attribution is the practice of returning, alongside a generated answer, the document or passage the answer was drawn from. A model on its own gives the user no way to check where an answer came from.

A RAG system can keep the identity of the retrieved passage with the answer. A response such as "employees receive 24 days of annual leave" can be shown with its source: "Employee Handbook, Leave Policy, Section 4.2." This matters most where a reader has to verify an answer before acting on it, as in finance, healthcare, legal, and compliance work.

How did retrieval change enterprise search?

Traditional enterprise search matched keywords to documents and returned a list of results for the user to read through. RAG-based search retrieves the relevant passages by meaning and has the model combine them into one natural-language answer. The user checks one answer against its cited source and no longer opens each document in turn.

Why does source attribution matter in a RAG system?

Source attribution lets a reader verify a generated answer against the document it came from. In finance, healthcare, legal, and compliance work, a RAG system that returns the retrieved source with the answer gives the reader something to check before acting.

Fine-tuning and RAG solve different problems

Fine-tuning is further training of a model on additional data to change its behavior, style, or task performance. RAG leaves the model unchanged and supplies external knowledge at the moment of inference.

A system meant to answer questions from the latest employee handbook fits RAG better than repeated fine-tuning. When the handbook changes, the knowledge base is updated and the retriever returns the new version. Retraining a model every time a policy document changes is rarely practical.

When should a team choose fine-tuning over RAG?

Fine-tuning fits when the goal is to change how a model behaves, such as its tone, format, or task performance. RAG fits when the goal is to answer questions from information that changes over time, such as a policy document or a product catalog.

Data engineering decides the quality of a RAG answer

Chunking is the process of splitting a document into smaller passages before embedding, so a retriever can return one relevant section and not a whole document. It is one stage of a data pipeline that runs before any retrieval or generation happens.

What does a production RAG pipeline include?

A production RAG system runs these stages in order:

• Ingestion

• Cleaning

• Parsing

• Chunking

• Metadata

• Embeddings

• A vector or search index

Every stage can change the final answer. Poor parsing produces poor chunks. Poor chunks produce poor retrieval. Poor retrieval gives the model poor context, and the model answers from it.

The simplest RAG design is a single path: documents, embeddings, a vector database, similarity search, and the model. A production system can add hybrid search, metadata filtering, reranking, query rewriting, multi-step retrieval, graph-based retrieval, structured database queries, and tool calling. Each addition is another pipeline component that has to be built, tested, and monitored.

Why is data engineering central to RAG quality?

A RAG answer is only as good as the weakest stage in its data pipeline. Poor parsing produces poor chunks, poor chunks produce poor retrieval, and poor retrieval produces a poor answer, however capable the underlying model is.

RAG fails in five identifiable ways

RAG evaluation is the practice of measuring retrieval quality and answer quality separately, so a team can tell which stage produced a wrong answer. It is needed because RAG failures have specific causes:

• Poor retrieval: the correct document exists in the knowledge base, and the retriever does not surface it.

• Poor chunking: important information is split across two chunks, so neither one contains the full answer.

• Outdated indexes: the knowledge base holds a stale version of a document that has since changed.

• Incorrect source data: the retriever returns the right document, and the document itself is wrong.

• Context overload: too many retrieved chunks bury the relevant passage in noise the model has to sort through.

A production RAG system depends on all of the following together:

• Good source data

• Good retrieval and good context

• Good prompting and a capable model

• Ongoing evaluation and observability

Removing any one of them leaves a gap the others cannot fully cover.

What are the common ways a RAG system fails?

A RAG system commonly fails in five ways: poor retrieval, poor chunking, outdated indexes, incorrect source data, and context overload. Four of the five sit in the data and retrieval pipeline, before the model generates anything.

The shift is from what a model knows to what it can access

Outdated knowledge, private data access, hallucination, cost, and source attribution used to be treated as five separate problems. Retrieval-augmented generation addresses all five with one idea, which is to retrieve the information a model needs at the moment it needs it. The model stops being the place where knowledge is stored. The quality of an answer then depends on the data pipeline behind the retriever as much as on the model, which puts data engineering, metadata, and governance at the center of the generative AI stack.

Quick reference glossary

Retrieval-augmented generation (RAG): An architecture that retrieves relevant external information at the moment a question is asked and gives it to a large language model as context, instead of relying only on what the model learned during training.

Knowledge cutoff: The fixed point in time after which a large language model has no training data.

Retriever: The component of a RAG system that searches a knowledge base and returns the passages most relevant to a given question.

Grounding: The practice of supplying a model with retrieved evidence before it generates an answer, so the answer is drawn from a document and not from the model's unsupported internal knowledge.

Context window: The maximum amount of text a large language model can read in a single request.

Embedding: A numerical representation of a piece of text that places passages with similar meaning close together.

Vector search: A retrieval method that converts a question and the stored documents into numerical embeddings and returns the passages whose embeddings sit closest to the question's.

Hybrid search: A retrieval method that combines vector search with keyword search to capture both similar meaning and exact term matches.

Chunking: The process of splitting a document into smaller passages before embedding, so a retriever can return one relevant section and not a whole document.

Source attribution: The practice of returning, alongside a generated answer, the document or passage the answer was drawn from.

Fine-tuning: Further training of a model on additional data to change its behavior, style, or task performance.