RAG in plain English: letting the AI look it up

In November 2022 a passenger asked Air Canada's website chatbot how bereavement fares worked. The bot explained, confidently, that he could book at full price and claim the discount afterwards. The airline's own policy page said the opposite. When he was refused, he took it to a Canadian tribunal, which ruled in February 2024 (Moffatt v. Air Canada) that the airline was responsible for what its chatbot said and ordered it to pay the difference.

The chatbot was not lying. It was doing what a large language model does by default: writing the most plausible-sounding answer from memory. The fix most teams reach for has an awkward name, retrieval-augmented generation (RAG), and a simple idea: make the model read the real policy before it answers.

Picture a new colleague on their first day. They have read a great deal about your industry, they write beautifully, and they want to be helpful. Ask them how many days of parental leave the company offers and they will give you an answer immediately, in a reassuring tone, based on what companies like yours usually do. It might even be right.

A language model is that new hire. Everything it knows was absorbed during training and folded into its internal settings, its parameters. That built-in knowledge is called parametric knowledge, and it has two hard limits:

• It stops at a date. Training ended at the model's knowledge cutoff. Your handbook update from last month is not in there.
• It never saw your private documents at all. Your HR wiki, your contracts and your internal runbooks were not on the reading list.

When a question falls into either gap, the model does not usually say “I don't know”. It fills the gap with something fluent. A fluent, wrong answer is called a hallucination, and it is exactly what happened at Air Canada.

Now give that new hire a rule: before you answer, find the relevant page in the handbook, read it, and quote it. Their answers get slower by a few seconds and dramatically more trustworthy. That rule is RAG, and its name describes the three steps:

• Retrieval: search your documents for the few passages that relate to the question.
• Augmented: paste those passages into the instructions the model receives, next to the question.
• Generation: the model writes its answer from those passages, and says which ones it used.

Nobody retrains the model. Updating what it knows becomes as simple as editing a document, which is why RAG is the default design for assistants over company knowledge.

Modern models can read a lot at once: the amount of text they can take in a single request, their context window, now runs to hundreds of thousands of words for some models. For a small, stable handbook, pasting the whole thing in is a perfectly good design.

For a company wiki of 50,000 pages it is not. Every request would cost as much as reading a shelf of books, take far longer to start answering, and models are measurably worse at using facts buried in the middle of a very long input (lost in the middle). A person with the right page open answers better than one with the whole shelf on the desk.

To find the right page in a fraction of a second, a RAG system prepares the documents in advance. It cuts each one into short passages, usually a paragraph or two, a step called chunking. Think of copying a manual onto index cards.

Card size is a real decision. A card that covers parental leave, sick leave and holiday entitlement at once will turn up for all three questions and confuse each answer. A card that just says “up to 16 weeks” is useless without knowing what lasts 16 weeks. Good cards hold one idea, with enough context to make sense on their own.

Plain keyword search finds cards that contain your words. That works well for names, codes and exact phrases, and it is still part of almost every real system. But people rarely use the document's words: someone asks “how long can I take off when the baby arrives?” and the policy says “parental leave entitlement”.

So most systems also turn every card into an embedding: a list of numbers that captures roughly what the text means. Imagine a huge map where cards about similar topics sit near each other. “Time off when the baby arrives” lands next to “parental leave” even though they share no words. The question is placed on the same map and the nearest cards are pulled. That is semantic search; running it alongside keyword search is called hybrid search, and it is the usual production choice.

Because the answer comes from specific cards, it can point to them: “16 weeks at full pay (source: Parental leave policy, section 2)”. That is attribution, and it changes your relationship with the tool. You no longer have to trust the answer; you can check it in one click. An answer that says only what its sources say is called grounded.

• If search misses the right card, the answer cannot be right. In practice, most bad RAG answers are search failures, not model failures.
• Wrong documents give wrong answers. If the wiki still has last year's policy, the assistant will quote it faithfully.
• The model can still misread. Having the right page in front of it makes invention much rarer, not impossible.
• Some questions are not about text. “How many people joined last quarter?” belongs in a database query, not a document search.

The most underrated behaviour of a good RAG system is saying “I couldn't find that in our documents.” If nothing relevant turns up, the honest move is to decline rather than guess, a behaviour called abstention. Had Air Canada's bot been required to answer only from the bereavement policy page, or to say it didn't know, the tribunal case would not exist.