What RAG actually is, for people buying it rather than building it

RAG stands for retrieval-augmented generation, and it means one simple thing: before the AI answers, it looks something up. Your documents are searched, the relevant passages are handed to the language model, and the model is told to answer using those passages rather than from its general knowledge.

That is the whole concept. Everything technical about it is implementation detail — but the implementation detail determines whether it works, and this is a guide to the questions that expose it.

Why does the term appear in every proposal?

Because it is the standard way to make a language model answer from a specific body of knowledge, and almost every business AI project needs that.

Without it, the model answers from what it absorbed during training — which does not include your pricing, your policies or your product range, and which is frozen at some point in the past. With it, the model works from material you control and can update this afternoon.

So the presence of the term in a proposal tells you very little. Everyone says it. What matters is what sits behind it.

What separates a working system from a demonstration?

Demonstrations are easy. A vendor loads twenty clean pages, asks three questions they chose, and it works beautifully. Your business has four hundred inconsistent documents and customers who ask questions nobody anticipated.

In a demonstration	In your business
Clean, curated documents	PDFs from 2019 with tables split across pages
Questions the vendor chose	Questions phrased in ways nobody predicted
One version of each fact	Three policy versions, all findable
English	Hindi, Gujarati, mixed-script messages
Nobody asks anything unanswerable	Half the questions are outside the documents

The last row is the important one. Any system looks good when every question has an answer in the material. The measure of a real one is what it does when the answer is not there.

What should you ask a vendor?

Seven questions. You do not need to understand the technology to judge the answers.

1. “Show me it answering a question that is not in the documents.” The single most revealing request. It should decline and offer a handover. If it produces a confident answer, it will do that to your customers.

2. “Can you show me which passage produced that answer?” If the system cannot cite its source internally, you cannot audit a disputed answer, and you will have disputed answers.

3. “How do we update a fact?” The answer should be “edit the document”. If it involves the vendor, or a retraining cycle, you have bought a dependency.

4. “How do you know a change has not broken something?” They should describe a test set of questions with known answers, run on every change. If the answer is “we check it manually”, quality will drift.

5. “What happens to questions it cannot answer?” They should be logged and reviewable. If they vanish, you lose the most valuable output the system produces.

6. “Where does our data go?” Which provider, which region, whether it is retained, whether it could be used for training. Ask before signing, and get it in the contract.

7. “Let me test it with our own documents.” Not their sample set. A vendor confident in the system will agree; one who resists is telling you something.

What will it cost you beyond the software?

Three things, all routinely underestimated.

Document preparation. Usually the largest task. Finding what exists, deciding which version is correct, converting print-formatted files into something machine-readable, and writing down what only exists in someone’s head.

Supervised launch. Someone reviewing answers for the first weeks and feeding corrections back. Skip this and the errors reach customers instead.

Ongoing ownership. A named person reviewing unanswered questions monthly and keeping the knowledge base current. Without it the system is accurate at launch and mediocre within a year.

Per-query costs are real but usually the smallest line. The labour is what makes or breaks the business case.

Common questions

Is RAG better than fine-tuning?

They do different jobs. Retrieval supplies facts; fine-tuning shapes style and format. Most business applications need current facts far more than a distinctive voice, so retrieval is usually the right starting point and often the only thing needed.

Does it eliminate wrong answers?

No. It reduces them substantially by grounding answers in real material, but a system can still retrieve the wrong passage or summarise one badly. Judge a vendor on how they handle being wrong, not on whether they claim it cannot happen.

How long does implementation take?

The technical build is rarely the constraint. Document preparation and the supervised launch period set the timeline, and both depend on the state of your material — which nobody can estimate before looking at it.

Can we build this ourselves?

A basic version, yes — the components are widely available and a capable developer can assemble one. The difficulty is not assembly, it is evaluation, guardrails, handover and the discipline to keep it accurate. Those are the parts that make it safe to put in front of customers.

More on this: building a chatbot that answers from your documents and when a language model is the wrong tool. See also OpenAI integration.