Why plain chatbots make things up
A language model answers from patterns learned during training. It has never seen your price list, your refund policy or last week’s new branch timings, so when asked it guesses, often confidently. Retrieval-augmented generation fixes this by giving the model the right reference material at the moment of the question. The idea was formalised in a 2020 research paper by Patrick Lewis and colleagues, which showed that pairing a model with a searchable document index produced more factual answers than the model alone.
How does a RAG chatbot work, step by step?
- Collect the source material: website pages, PDFs, SOPs, product catalogues, FAQ sheets, past support replies.
- Clean and split it into chunks of a few hundred words, each tagged with its source, date and access level.
- Embed each chunk: an embedding model turns the text into a list of numbers that captures its meaning.
- Store the embeddings in a vector database, or in a search index that supports both keyword and meaning-based search.
- Retrieve: when a customer asks "do you deliver to Noida on Sunday?", the question is embedded and the closest chunks are fetched.
- Generate: the model receives the question plus the retrieved chunks with an instruction to answer only from them and to say "I don’t know" otherwise.
- Cite and log: the answer shows or stores which document it came from, so wrong answers can be traced and fixed.
Good systems add a re-ranking step that re-orders retrieved chunks by relevance, and a hybrid search that combines keywords (useful for product codes and names) with meaning-based search (useful for loosely phrased questions).
When does a business actually need RAG?
| Situation | Do you need RAG? | Why |
|---|---|---|
| Fewer than 30 FAQs that rarely change | Usually no | Put them straight into the bot’s instructions |
| Hundreds of products, policies or SOP pages | Yes | Too much to fit in instructions; must be looked up |
| Information changes weekly (stock, schedules, prices) | Yes, or a live API lookup | Update documents without rebuilding the bot |
| Staff knowledge base for a multi-branch business | Yes | Answers with citations build trust with staff |
| Questions need live account data (order status, balance) | Use tool calls, not just RAG | Live data belongs in an API call, not a document |
| Regulated advice (medical, legal, financial) | RAG plus strict hand-over | The bot can quote policy, a professional must advise |
Many real assistants combine both: RAG for policies and product knowledge, and tool calls into Shopify, Zoho or a booking system for live data. That combination is how most AI agents answer questions about a specific customer.
RAG vs fine-tuning vs long context
| Approach | What it is | Good for | Weak at |
|---|---|---|---|
| RAG | Look up relevant passages at question time | Large or changing knowledge, citing sources | Questions needing the whole document set at once |
| Fine-tuning | Further train a model on your examples | Tone, format and specialised classification | Storing facts that change; hard to update |
| Long context | Paste whole documents into every prompt | Small, stable knowledge (a few documents) | Cost and speed at scale; no access control |
| Tool calls | Query a live system through its API | Order status, stock, balances, slots | Unstructured policy and how-to knowledge |
Most business chatbots end up as RAG plus tool calls. Fine-tuning is rarely the first step for an SME.
Build options: no-code platform or custom?
No-code chatbot builders let you upload documents and get a working website bot in a day. They suit a single channel, public information and modest volume. A custom build, often on n8n or code with a vector database such as Postgres with pgvector, makes sense when you need WhatsApp plus other channels, role-based access, integrations with your CRM or order system, Hindi and English documents, or control over where data is stored. A sensible path is to prototype on a no-code tool with real questions, learn where it fails, then decide whether a custom build is worth it.
What drives the cost of a RAG chatbot?
Model usage for embedding documents is usually cheap; for most SMEs it is a small part of the bill. The real cost drivers are elsewhere:
| Cost driver | Low end | High end |
|---|---|---|
| Data preparation | A clean website and a few tidy PDFs | Scanned documents, tables, Hindi and English mixed, conflicting versions |
| Channels | Website widget only | WhatsApp, website, internal Slack or Teams, voice |
| Integrations | None, documents only | CRM, order system, ticketing, single sign-on |
| Access control | Everyone sees everything | Answers filtered by branch, role or customer |
| Volume | A few hundred questions a month | Tens of thousands, needing caching and monitoring |
| Accuracy work | Basic testing | Evaluation sets, re-ranking, weekly review of failed answers |
For typical ranges and a line-by-line breakdown, see AI chatbot cost in India. If the bot will live on WhatsApp, the messaging fees often matter more than the AI fees; see WhatsApp AI agent cost.
How accurate is a RAG chatbot?
It is only as accurate as the documents it retrieves and the instructions it follows. Most wrong answers trace back to one of four causes: the answer was not in the documents, two documents disagreed, retrieval fetched the wrong chunk, or the model filled a gap instead of admitting it did not know.
- Build a test set of 50 to 100 real customer questions with the correct answers before launch.
- Remove outdated and duplicate documents; keep one source of truth per topic.
- Instruct the bot to answer only from retrieved text and to say when it does not know.
- Show or log the source for every answer.
- Route low-confidence answers and sensitive topics to a person.
- Review a sample of real conversations weekly and fix documents, not just prompts.
- Re-run the test set every time you change the model, prompts or documents.
Questions to ask before you buy a RAG chatbot
- Which documents will it index, and who keeps them up to date?
- How does it handle a question the documents do not answer? Show me.
- Can it show the source for every answer, and can I see the logs?
- How are Hindi, English and Hinglish questions handled against English documents?
- Can answers be limited by branch, role or customer?
- Where are the documents, embeddings and chat logs stored, and for how long?
- What does it cost per month at my expected volume, including the model, hosting and the channel?
- If I leave, do I keep the index, the prompts and the conversation history?
A vendor who cannot demonstrate the "I don’t know" behaviour on your own documents in the first meeting is selling a demo, not a system.
Data privacy and the DPDP Act
If your documents or chats contain personal data, the Digital Personal Data Protection Act, 2023 applies. Its Rules were notified in November 2025, with most operational duties phased in over 18 months, so businesses have until around May 2027 for the core obligations. Confirm current timelines with your legal adviser.
In practice for a RAG chatbot: avoid indexing personal data you do not need; apply access controls so a customer can never retrieve another customer’s records; tell users what you collect and why; use business AI plans that do not train on your data; set a retention period for chat logs; and know where your vector database is hosted. For sensitive data, self-hosted or private deployment options are available at higher cost.
How Silvane can help
Silvane builds RAG chatbots and assistants as part of its custom AI agents and AI customer support services, on your website, WhatsApp or internal tools, with sources cited and hand-over to your team. It starts with a ₹199 discovery call and audit for founding members (normally ₹15,000). Builds start from ₹50,000 and run plans from ₹15,000 a month; AI and hosting usage are billed at cost and GST is extra.
Sources: Lewis et al., Retrieval-Augmented Generation (2020) · DPDP Rules notified, phased over 18 months (YourStory) · Anthropic: Building effective agents