Knowledge Base Chatbot: How It Works and How to Build One

TL;DR: A knowledge base chatbot is an AI chatbot that answers questions only from content you give it: help articles, FAQs, policies, product pages and documents. It works by retrieval: it searches your content for the few passages that match the question, then writes an answer from them. Its quality depends far more on how your content is written and kept current than on which AI model it uses.

A knowledge base chatbot is a chatbot that retrieves answers from a defined set of your own sources instead of relying on what a language model learned in training. When a customer asks "What is your return window?", the bot searches your returns policy, finds the passage with the number, and answers in a sentence, ideally with a link to the source. The technique behind almost every product in this category, from Intercom Fin and Zendesk AI agents to Chatbase and custom builds on the OpenAI or Anthropic APIs, is called retrieval-augmented generation (RAG).

Why tie a bot to your own content?

A chatbot needs a knowledge base because a general model does not know your prices, policies or stock, and will guess when asked. A guessed refund policy is worse than no answer. Tying the bot to your sources gives three things a general chatbot can't:

• Current facts: change the policy page and the next answer changes.
• Traceability: each answer points back to the passage it came from, so a wrong answer can be fixed at the source.
• A clear boundary: a question with no matching passage becomes "I don't know, let me connect you to the team," not an invention.

Consider a small furniture shop with 40 help articles. Customers ask the same ten questions about delivery areas, assembly and returns all day. A knowledge base chatbot answers those from the articles at 2 a.m., and hands the custom-order questions to a person.

How does retrieval work, step by step?

A knowledge base chatbot works in two phases: indexing, done once per content change, and retrieval, done on every question.

Indexing. The system splits each document into chunks, converts every chunk into a vector embedding that represents its meaning, and stores them in a vector database. Anthropic describes the usual chunk size as "no more than a few hundred tokens" (Anthropic: Contextual Retrieval). OpenAI's file search defaults to 800-token chunks with a 400-token overlap; you can set max_chunk_size_tokens between 100 and 4096, with chunk_overlap_tokens no more than half of it (OpenAI: Retrieval).

Retrieval. When a visitor asks something, the question is embedded the same way, and the closest chunks are pulled back. OpenAI's search returns 10 results by default and up to 50 with max_num_results. Good systems combine meaning-based search with keyword search (BM25), because embeddings can miss exact strings. Anthropic's example is a user searching for "Error code TS-999": embeddings find articles about error codes in general, BM25 finds the one with that exact code. The top passages are then placed in the model's prompt with an instruction to answer only from them.

Do you always need retrieval?

You do not always need retrieval. Anthropic notes that a knowledge base smaller than 200,000 tokens, about 500 pages, can be placed in the prompt whole, and prompt caching makes that faster (over 2x lower latency) and up to 90% cheaper. Most small-business sites are well under that size. Retrieval still helps once you have many sources, frequent updates, or need to show which passage an answer came from.

Worked example: what happens to a 30-page help centre?

Here is how a help centre of 30 articles, averaging 1,200 tokens each, becomes searchable with OpenAI's defaults:

At 36,000 tokens this help centre would also fit in a single prompt, which is why small knowledge bases rarely fail on retrieval mechanics. They fail on content.

Products set these limits for you. LuckPanda's Chatbot, for example, holds up to 60 sources per bot, searches all of them for each question and uses the best three passages; a website source reads 5, 10, 20 or 50 pages at a link depth of 0 to 3 (LuckPanda: help centre and FAQ).

How do you prepare content for a knowledge base chatbot?

Prepare content so that each passage answers one question on its own, because the bot only sees a few passages at a time, never your whole article.

1. One topic per article. "Returns and exchanges" and "Shipping" belong on separate pages. A chunk that mixes both gets retrieved for the wrong question.
2. Put the rule in a sentence. "Returns are accepted within 30 days of delivery" retrieves and quotes cleanly. A rule buried in a table image or a scanned PDF does not.
3. Repeat the subject. A chunk saying "It costs $40" loses its meaning when cut from the heading above. Write "Express delivery costs $40." Anthropic's Contextual Retrieval adds this context to each chunk automatically and cut failed retrievals by 49%, and by 67% with reranking.
4. Use the customer's words. If buyers say "refund" and your page says "reimbursement," add the common term.
5. Add exact rules as FAQ entries too. A warranty term or cancellation window stated once as a short FAQ answer is the most reliable source a bot can have.
6. Remove stale and duplicate pages. Two versions of a price list guarantee that some answers use the old one.
7. Add separate sites as separate sources. A help centre at help.example.com is a different site; a crawl of the main domain will not follow links into it.

How do you keep the knowledge base current?

Keep it current by updating the source, not the bot. Re-sync schedules vary: Intercom's Fin re-syncs website sources once a week (Intercom: Add your content for Fin), so a price change on your site may take up to seven days to reach the bot unless you trigger a manual re-sync. After any price, policy or hours change, re-sync and ask the bot the affected question.

How is it different from rule-based bots and article suggestions?

A knowledge base chatbot writes an answer; older tools only point to one.

What causes wrong answers?

A knowledge base chatbot usually gives a wrong answer because a source is missing, old, contradictory or unclear, not because the model is weak. Check in this order:

• Missing: the answer isn't in any source. Add it.
• Stale: two pages disagree. Delete or update one.
• Unretrievable: the fact sits in an image, a scanned PDF or a page the crawl didn't reach.
• Ambiguous: the chunk lost its subject (see step 3 above).
• No fallback: the bot was not told to say "I don't know." Configure a human handoff for unmatched questions.

When you're ready to try one on your own pages, LuckPanda Chatbot builds a free preview from your website address; correct a wrong answer once and it uses your wording from then on. For what it stores, see security. Running Squarespace? See how to add a chatbot to Squarespace.