RAG (Retrieval-Augmented Generation)
Also called: Retrieval-Augmented Generation · Retrieval Augmented Generation · grounding
Letting an AI read your documents before it answers, so it works from your facts instead of its guesses.
Why it matters
A model on its own only knows what it absorbed during training. It has never seen your pricing sheet, your contracts, or last quarter's handover notes. RAG is what closes that gap, and it is the difference between an assistant that sounds plausible and one your team can actually rely on.
How it works
Three steps, and none of them are magic.
Retrieve. Your question is used to search a store of your own documents (contracts, wikis, tickets, transcripts) and the most relevant passages come back. This is search. It has the same failure modes as search.
Augment. Those passages are placed into the model’s prompt alongside your question, usually with an instruction along the lines of: answer using only what follows.
Generate. The model reads the passages and writes an answer. Because the facts are sitting in front of it, it does not have to reach into its training data and reconstruct something that sounds right.
The name describes the sequence exactly. Retrieval, then augmented generation.
Where it goes wrong in practice
Almost every RAG system that disappoints its users disappoints them at the retrieval step, not the generation step. The model is doing its job faithfully: it is answering from the passages it was handed. The passages were the wrong ones.
That has consequences for how you evaluate one. If you only read the final answers, you cannot tell a retrieval failure from a reasoning failure, and you will spend weeks tuning the wrong half of the system. Log what was retrieved. Read it. In our experience the first honest look at the retrieved passages is the most useful hour of the whole project.
The second common failure is quieter. The system works, the answers are good, and six months later the source documents have drifted out of date while nobody was watching. A RAG system is only as current as the corpus behind it, and corpora rot. Somebody has to own that.
What it is good for
Question-answering over an internal knowledge base. Customer support that quotes the actual policy. Onboarding, where a new hire can ask the questions they are slightly embarrassed to ask a colleague. Contract and document review, where the answer must be traceable to a clause someone can go and read.
The common thread: the value is in your information, and the model is the interface to it. Where there is no proprietary information (general drafting, brainstorming, translation), RAG adds cost and no accuracy.
The question to ask before building one
Not “can we build a RAG system?” You can. The question is: when this thing gives an answer, who is accountable for whether it was right, and how would they check?
If the answer is “the model”, the project is not ready. If the answer is a named person who can click through to the source passage in two seconds, you have designed something that will survive contact with your organisation.
Common questions
Is RAG the same as fine-tuning?
No. Fine-tuning changes the model's behaviour: its tone, its format, the shape of its answers. RAG changes what the model knows at the moment it answers. If your problem is that the AI writes in the wrong style, fine-tuning may help. If your problem is that the AI does not know your refund policy, you need RAG. Most teams reach for fine-tuning when RAG is what they needed.
Do we need RAG if the model has a large context window?
Sometimes not. If your entire knowledge base fits comfortably in the context window and rarely changes, you can simply paste it in. RAG earns its complexity when the corpus is too large to pass every time, changes often enough that a static copy goes stale, or has permissions attached: different people are allowed to see different documents.
Does RAG stop hallucination?
It reduces it substantially. It does not eliminate it. A model given the right passage can still summarise it wrongly, and a retrieval step that surfaces the wrong document will produce a confident answer built on the wrong facts. RAG moves the failure from 'the model invented something' to 'the system retrieved the wrong thing', a failure that is far easier to inspect and fix.
What does a RAG system need from us before it works?
Documents that are findable, current, and permissioned. Most RAG projects that stall do not stall on the retrieval code. They stall because nobody could say which version of the policy document was authoritative, or because the useful knowledge lived in people's inboxes rather than in any system at all.
Working through this with your team?
Olaia Labs runs AI training and automation programmes for SMEs. Start with a diagnostic: we look at where your team already uses AI, where it breaks, and what is worth automating.
Request a diagnostic