Building a chatbot that answers from your own documents/knowledge base

A chatbot that answers from your own documents does not “know” your business — it looks things up before answering. The user asks a question, the system finds the relevant passages in your material, and the language model is instructed to answer using only those passages. Without that retrieval step, the model answers from general knowledge and will confidently invent a refund policy that sounds exactly like yours.

That architectural difference is the whole thing. Everything else is refinement.

Why can you not simply train it on your data?

You can, and for most businesses it is the wrong approach.

Training or fine-tuning teaches a model a style and a set of patterns. It does not give it a reliable memory of facts, and it cannot be updated when your pricing changes on Monday. You would have to retrain.

Retrieval keeps your knowledge in documents you control. Change the document, and the next answer changes. No retraining, no delay, and — critically — you can see which passage produced any given answer, which matters enormously the first time a customer disputes what the bot told them.

	Fine-tuning	Retrieval
Updating a fact	Retrain	Edit the document
Can you see why it answered that?	No	Yes — the source passage
Handles frequently changing data	Poorly	Well
Good for teaching tone and format	Yes	Partly
Setup effort	High	Moderate

Most business chatbots need current facts far more than they need a distinctive voice. Retrieval first; fine-tune later, if at all.

What actually happens when someone asks a question?

1. The question is converted into a numerical representation that captures its meaning rather than its exact words — so “can I send it back” finds your returns policy even though it shares no vocabulary with it.
2. Your documents, processed the same way in advance, are searched for the closest passages.
3. The best few passages are selected — a handful, not everything that matched.
4. Those passages plus the question go to the language model, with an instruction to answer only from what was supplied and to say so if the answer is not there.
5. The answer is returned, ideally with a reference to where it came from.

Step four is where most implementations are weak. If the instruction does not firmly constrain the model to the supplied passages, it will fill gaps from general knowledge — and the resulting answer reads exactly like the correct ones.

Where do these projects actually go wrong?

Almost never in the model. Nearly always in the documents.

The knowledge does not exist in writing. The most common discovery. Your best support agent knows the answers; nothing is documented. Retrieval cannot find what was never written down, and this is the point at which many projects pause for a month.

The documents contradict each other. Three versions of the returns policy in different files, all findable. The bot answers from whichever matched best, which may be the one from 2023.

Everything is in PDFs built for print. Multi-column layouts, tables split across pages, text embedded in images. Extraction produces something that reads like scrambled fragments, and retrieval over scrambled fragments returns nonsense.

Passages are cut in the wrong places. Documents are split into chunks for retrieval. Split badly and a policy’s condition ends up in one chunk and its exception in another — so the bot cheerfully states the rule without the exception.

Nobody owns the knowledge base. It is accurate at launch and drifts within months. Without a named owner and a review rhythm, accuracy decays quietly and nobody notices until a customer complains.

What has to be built beyond retrieval?

Four things that separate a demo from something you can put in front of customers.

A way to say “I do not know”. The bot must be able to decline. This is the single most important behaviour and the hardest to get right, because the natural failure mode of these systems is confident plausibility. If retrieval returns nothing relevant, the correct output is an admission and a handover — not an attempt.

Handover with context. When it hands to a person, the conversation goes too. A customer who has explained their problem twice will not explain it a third time.

Evaluation. A set of real questions with known correct answers, run every time anything changes. Without it you cannot tell whether an adjustment improved things or quietly broke a category of question that used to work.

Logging of unanswered questions. What it could not answer is more valuable than what it could. That log is your content roadmap, and reviewing it monthly is what keeps the system improving rather than decaying.

How should you start?

Narrow. Pick one category of question with clear documented answers — order status, warranty terms, product specifications — and get that right before widening.

A bot that answers one thing reliably is useful and builds trust. A bot that attempts everything and is right most of the time teaches customers not to believe it, and that reputation is difficult to reverse.

This is how we approach it under AI chatbots for WhatsApp and web — knowledge audit first, narrow scope, supervised launch, monthly review of what it could not answer.

Common questions

How much documentation do we need to start?

Less than you think, if it is well-scoped. Twenty accurate pages covering your most common questions beats a thousand pages of mixed-quality material. Quality and currency matter far more than volume.

Can it read our existing PDFs?

Usually, with preparation. PDFs designed for print extract badly, and scanned documents need OCR first. Budget time for document preparation — it is routinely the largest task in the project and the one nobody plans for.

Will it work in Hindi and other Indian languages?

Generally yes, including answering in one language from documents written in another. Quality varies by language, so test with real questions from real customers rather than assuming parity with English.

What stops it from giving a wrong answer confidently?

Nothing entirely. You reduce it — constrain answers to retrieved passages, require a source, refuse when retrieval is weak, review during a supervised launch. Anyone promising elimination is overselling. The right question is what happens when it is wrong, not whether it ever will be.

More on this: what RAG actually is and when a language model is the wrong tool. For how we build these, see AI chatbots.