Building a Knowledge Base That Actually Works: Lessons from 2,000+ Deployments

After watching over 2,000 businesses deploy AI agents on Chatzuri, a pattern emerges in the deployments that underperform: the model is fine, the channel integration is fine, the prompt is reasonable — but the knowledge base is a mess. The agent hallucinates, gives outdated answers, or retrieves the wrong document for the user's question. Almost all of it traces back to how the knowledge base was built.

Mistake 1: Uploading Documents Without Cleaning Them First

A PDF exported from a presentation deck carries a lot of noise: slide headers, footers, page numbers, formatting artifacts. When these get chunked and indexed, they pollute the vector space. A retrieval system searching for 'refund policy' might rank a chunk that says 'Slide 14 | Refund Policy Overview' — which contains almost no useful content.

Before uploading any document, strip all structural artifacts. Export policies and FAQs as clean plain text or structured Markdown. Use clear section headers. If you're uploading PDFs, preview the extracted text first — most tools let you see what the parser produces before indexing.

Mistake 2: Chunks That Are Too Long or Too Short

Chunking strategy is the single most impactful technical decision in knowledge base design. Chunks that are too long (3,000+ characters) carry too much noise and make it hard for the retrieval system to find the exact relevant passage. Chunks that are too short (50–100 characters) lose context and confuse the LLM when it tries to compose an answer.

A useful default: 400–600 characters with 50-character overlap between adjacent chunks. For long technical documents, chunk by section (heading + body). For FAQs, chunk by question-answer pair. The overlap prevents the retriever from cutting answers in half at chunk boundaries.

Mistake 3: No Versioning or Staleness Handling

Product terms change. Pricing updates. Policy documents get revised. A knowledge base with no versioning will confidently tell customers the price you charged two years ago. At best this is embarrassing; at worst it's a contractual dispute.

• Tag every chunk with a document version and last-updated date
• Configure the retrieval system to weight recent versions over old ones
• Set a review cadence — quarterly at minimum — to identify outdated content
• When policy changes, don't append the update — replace the chunk and re-index

Mistake 4: Covering Only the Happy Path

Most teams build their knowledge base from their marketing copy and official policy documents. These cover what's supposed to happen. Customers ask about what happens when things go wrong: failed payments, delayed orders, account lockouts, refund disputes. If these scenarios aren't in the knowledge base, the agent will either hallucinate an answer or admit it doesn't know — neither is good.

Mistake 5: Treating the Knowledge Base as a One-Time Project

A knowledge base is a living system, not a launch artifact. Every conversation your agent has is a signal about what the knowledge base is missing or getting wrong. Review your agent's low-confidence responses weekly in the early months. The conversations where the agent says 'I'm not sure about that' or retrieves an irrelevant chunk are a direct map to your next knowledge base update.

Chatzuri's analytics show that agents with a regular knowledge base review process reach 90%+ resolution rates in 60 days. Agents that are deployed and never updated plateau at around 70% and stay there.