We can’t expect a general-purpose model to know everything, but we can build a system around it that knows where to find that information and surfaces it as needed.
In this series, we’ve already talked quite a bit about the different ways we can extend the capabilities of generative AI models, including prompt engineering, MCP servers, agent skills and more. But there’s an important method that deserves its own deep dive: Retrieval Augmented Generation (or RAG).
LLMs are trained on enormous amounts of data, but that doesn’t mean they know everything. Not only does the training data have a cutoff date (meaning that information created or shared recently won’t be included), but it also, by nature, won’t include things like proprietary data or data specific to your application/knowledge base.
Even if the information is somewhere in the model’s training data, we can’t necessarily expect the model to accurately produce a specific fact from everything it learned during training. As we discussed in the earlier article about hallucinations, if a model doesn’t have enough information to confidently answer a question, it may simply generate a plausible response instead—which isn’t particularly helpful if we’re building an application where accuracy matters (i.e., pretty much every application).
One possible solution to this issue is to fine-tune the model on our additional data—but that’s a fairly time and resource intensive process and it may not actually be the best option. What if the information in question changes regularly, like documentation that updates with each new quarterly release? What if we have thousands of documents that we want the model to reference, but don’t actually want to use that content as training data? What if the information is private and specific to a particular customer or user?
RAG gives us another option: instead of teaching the model, we can simply allow it to retrieve the relevant information as needed.
What Is RAG?
Actually, RAG does a pretty good job of being exactly what it says on the tin.
- Retrieval: finding relevant information from an external source
- Augmentation: adding that information to the model’s context
- Generation: using that additional context to generate a response
Let’s say that we’re building an AI assistant for a company’s internal documentation. An employee might ask the chatbot: “How many days of parental leave do we offer?” Your average, off-the-rack LLM probably knows about the general concept of parental leave, but it has no way of knowing what our particular company offers. That kind of specific HR information wouldn’t have been included in its training data.
Without access to a source that has the actual answer to that question, it’s likely to either give a generic answer about parental leave (unhelpful and annoying) or … just make something up (worse).
These models are designed to generate a likely response based on the patterns it’s learned. If we need it to answer questions based on a specific set of documents, it’s better to just give it access to that document’s content rather than expect it to guess correctly.
That’s where the retrieval part of RAG comes in. Rather than asking the model to answer a question using everything it learned during training, we first search a separate collection of information for content that is relevant to the question. We then give that content to the model as part of its context, allowing it to generate a response based on information that we have specifically selected.
With RAG, we can provide the company’s employee handbook to the agent and have it reference that for information related to parental leave. We haven’t changed the model or taught it anything new; we’ve simply given it additional information to work with at the time it generates its response.
How Does AI Find the Right Information?
Now that we have a high-level understanding of how the process works, let’s take a deeper look at how our chatbot will figure out which information is relevant in the first place.
Chunks
Imagine we have a 100+ page employee handbook that includes information about healthcare, vacation, parental leave, expense reimbursement, workplace policies and dozens of other topics. We don’t want to send the entire document to the model each time someone asks a question; not only would that be inefficient, but giving the model that much (mostly irrelevant) information won’t necessarily make it better at answering our user’s specific question.
Instead, we break the document into smaller pieces, called chunks. A chunk could be a paragraph, several paragraphs, a section of a document or some other logical unit of information. We then index those chunks so that our AI bot can search through them.
Chunking is important for a couple reasons. First, because it allows us to limit the amount of information we input into the (limited) context window. And, second, because having smaller, logically organized chunks of information makes it easier for the retrieval system to find the relevant content.
Embeddings
Now imagine that our user asks, “How much time do I get off after having a baby?” There may not be a single place in our documentation that uses those exact words. The relevant section might instead be titled “Parental Leave,” and it might contain information about paid leave, eligibility requirements and how far in advance an employee needs to notify their manager. A traditional keyword search looking for the words “have” and “baby” probably isn’t going to be particularly useful here, because the most relevant document might not contain those specific words.
This is where embeddings come into play. An embedding is a numerical representation of a piece of information—basically, a list of numbers that align with characteristics of the text. We generate an embedding for each of our document chunks, which get saved to a vector database (or other similar system). Then, when a user asks a question, we generate an embedding for that as well. By comparing those embeddings, our RAG system can find pieces of text that are mathematically similar to the meaning of the question, even when they don’t use exactly the same words.
Semantic Searching
This process is referred to as semantic searching, and it’s one of the most common approaches used in RAG systems. Instead of asking, “Which documents contain the same words as my question?” we’re asking something closer to, “Which pieces of information are most similar in meaning to my question?”
Of course, many RAG systems will use this approach in combination with traditional/keyword search, relational databases, graph databases or APIs in order to offer the most thorough and accurate responses.
RAG Is a System, Not a Feature
One of the reasons that RAG has become such a popular approach is that it gives developers a relatively straightforward way to connect generative AI models to information outside of the model, itself. However, it’s worth remembering that RAG isn’t a single technology, but rather a collection of decisions about how we store, organize, retrieve and provide information to a model.
We have to consider where our source data comes from, how we process it, how we divide it into chunks, how we search it, how much information we retrieve, and how we determine whether that information is actually relevant. Then, we still have to consider how the model uses the retrieved information, as well as how we evaluate the quality of the final response.
As always, the AI model is just one part of a larger system. We can’t expect a general-purpose model to know everything about our product, our company or our users—but we can build a system around it that knows where to find that information and surfaces it to the model as needed.
Log in
