June 22, 2026
How to Build a Knowledge Base Your AI Agent Can Actually Use
You can have the best AI agent on the market, but if it's reading from a 60-page PDF that hasn't been updated since 2022, your customers are going to get bad answers. The knowledge base is where most AI projects quietly fail — not the model, not the prompt, not the integration. The content.
Here's how to build one that actually works.
Why the Knowledge Base Decides Everything
When a customer asks your AI agent a question, the agent doesn't pull the answer out of thin air. It searches your knowledge base for the most relevant chunks of text, then uses those chunks to construct a response. This is called retrieval-augmented generation, or RAG, and the "retrieval" part is doing more work than people realize.
If the agent retrieves the wrong chunk, it gives the wrong answer — confidently. If it retrieves a chunk that's outdated, it tells the customer something that hasn't been true for six months. If it retrieves nothing useful at all, it either hallucinates or gives a generic non-answer that makes you look amateur.
So the quality of your KB sets the ceiling on the quality of your agent. A great KB with a decent model will outperform a mediocre KB with the best model on the planet.
What Actually Works: Three Content Patterns
After building these systems for small businesses, three content patterns consistently produce good retrieval. Use these as your default.
1. Q&A pairs in your customer's exact phrasing
This is the highest-leverage content you can write. Find the questions customers actually ask — from your inbox, your chat logs, your phone notes — and write them down in their words, not yours.
Customers don't ask "What is your return policy?" They ask "can i return something i bought 3 weeks ago" or "do i get my money back if it doesn't fit." Both questions should exist in your KB, with the same answer attached.
## Can I return an item after 30 days?
We accept returns within 60 days of purchase for unused items
in original packaging. After 60 days, we can offer store credit
on a case-by-case basis. Contact support@example.com with your
order number to start a return.
**Related questions:**
- How long do I have to return something?
- Can I get my money back if it doesn't fit?
- What's your return policy?
That "Related questions" block isn't decoration. It dramatically improves retrieval because the embedding model can match against any of those phrasings.
2. Policy docs with clear headers
For longer policies — shipping, refunds, warranty, service terms — break them into short sections under specific H2 or H3 headers. Each section should answer one question and stand alone.
Bad:
Our shipping policy includes a variety of options depending on your location, the items ordered, and the time of year, with most orders shipping within 2-5 business days from our warehouse in Ohio, though peak season may extend this...
Good:
## How long does shipping take?
Standard shipping: 2-5 business days within the continental US.
## Do you ship internationally?
We ship to Canada and Mexico. International orders take 7-14
business days and require a signature on delivery.
## What about peak season?
From November 15 through December 31, add 3-5 days to all
estimates due to carrier volume.
Each chunk the agent retrieves now contains a complete thought. No more "well, the answer is in there somewhere across three paragraphs."
3. Product specs in structured format
If you sell products or services with specs, plans, or tiers, structure them like data, not prose.
## Plan: Starter
- **Price:** $29/month
- **Users included:** Up to 3
- **Storage:** 10 GB
- **Support:** Email, 24-hour response
- **Best for:** Solo operators and small teams testing the platform
This is gold for retrieval. The agent can pull this entire chunk when someone asks about pricing, user limits, storage, support response times, or which plan fits a small team. One well-structured block answers a dozen questions.
What Breaks Retrieval (And Quietly Ruins Your Agent)
Three failure patterns show up over and over. If your KB has any of these, fix them before you blame the model.
Dense paragraphs with no headers. When the chunker breaks your document into pieces, it's working with character counts. A 2,000-word block of unbroken prose gets sliced into chunks that start and end mid-thought. The agent retrieves a chunk that begins "...this only applies if the customer has..." with no context. Headers act as natural break points and give each chunk a topic.
Ambiguous language. "We typically respond quickly" is useless. "We respond within 24 business hours" is retrievable and verifiable. If a human reader would have to ask a follow-up question, the AI is going to give a wishy-washy answer.
Version conflicts inside one document. This is the silent killer. Someone updates the pricing on page 1 but forgets the FAQ on page 8 still says $19/month. The agent retrieves both. Now your customer is getting two different prices in the same conversation.
The fix: one fact, one place. If pricing lives in pricing.md, don't restate it anywhere else. Link to it instead. When the price changes, you update one file.
What to Exclude (And Why It Matters)
The hard rule: if you wouldn't want a customer to read it verbatim, it doesn't belong in the agent's knowledge base.
This sounds obvious. It isn't. People dump their whole internal wiki into the agent on day one and then panic when it surfaces things like:
- Internal pricing notes ("we can go as low as $X if they push back")
- Competitor comparisons written for sales reps
- Process docs that reveal limits ("escalate to Sebastian if they ask about refunds over $500")
- Personnel info, vendor costs, margin notes
- Draft policies that haven't been approved
- Anything containing "TODO," "DRAFT," or "do not share"
The agent doesn't know what's confidential. It just retrieves what's most relevant to the question. If a customer asks "can I get a discount?" and your internal sales doc says "we'll go to 40% off if pressed," guess what your agent might tell them.
Keep two separate knowledge bases if you need to: one for the customer-facing agent, one for an internal operations agent. Don't share files between them.
The Format Hierarchy: Markdown Wins
Not all file formats parse equally. Here's the actual ranking based on how much structure survives ingestion:
1. Markdown (.md) — the gold standard. Headers, lists, code blocks, and links all parse cleanly. The structure you write is the structure the agent sees. Easy to version-control, easy to edit, easy to diff when something changes.
2. HTML — almost as good. Clean semantic HTML parses well. Messy HTML with inline styles and tracking pixels doesn't. If you're pulling from a website, strip it to the article content first.
3. Plain text (.txt) — works but loses hierarchy. Fine for simple Q&A lists. Bad for anything with structure.
4. Word docs (.docx) — usable but quirky. Headers and lists usually survive. Tables sometimes do. Embedded images and complex formatting usually don't. Tracked changes and comments occasionally leak in. Convert to Markdown when you can.
5. PDF — the worst common format. PDFs are designed for printing, not parsing. Multi-column layouts get jumbled. Tables become unreadable strings. Headers lose their hierarchy. Footnotes get mixed into body text. If your only source material is PDFs, plan on a real conversion effort, not just dumping them in.
If you remember one thing: write your KB in Markdown, in a folder structure that mirrors your topics. pricing.md, shipping.md, returns.md, faq.md. Boring, effective.
Chunk Size: The Setting Nobody Talks About
When your documents get ingested, they're split into chunks — usually 500 to 1,500 tokens each, with some overlap between adjacent chunks. This setting matters more than people realize.
Too small (under 300 tokens) and chunks lose context. The agent retrieves a piece that says "this option is only available on the higher tier" with no idea what "this option" or "higher tier" refers to.
Too large (over 2,000 tokens) and chunks become noisy. The agent retrieves a giant blob where the actual answer is one sentence buried in unrelated content, and the model has to work harder to find it.
For most small-business KBs, chunks around 800-1,000 tokens with 100-200 token overlap work well. But this is something to tune against real questions, not set and forget. If your agent keeps missing answers that are clearly in the KB, chunk size is one of the first things to revisit.
The Living KB Problem
Here's what nobody tells you when you launch: a knowledge base is not a project, it's a process. Your business changes. Prices go up. Policies tighten. Products get added or discontinued. Hours shift for holidays. Every one of those changes needs to land in the KB, or the agent starts lying to your customers in your name.
Build a monthly review cadence. Block 60-90 minutes on the calendar. Walk through:
- Pricing and plans. Anything change? Any promotional pricing that ended?
- Policies. Returns, refunds, warranties, terms of service — any updates?
- Products and services. Anything added, removed, or renamed?
- Hours and contact info. Holidays, new phone numbers, new support email?
- Top 10 questions from the last month. Are they all answered well in the KB? Add the ones that aren't.
- Recent agent failures. Most platforms log conversations where the agent gave up or got bad feedback. Read those. They tell you exactly what's missing.
Put one person in charge. If "everyone" owns the KB, nobody does, and three months from now you're back to a stale document and a frustrated customer base.
For fast-moving info like inventory, hours, or promotional pricing, don't put it in the KB at all. Connect the agent to the live source (your booking system, your storefront API, your CRM) so it pulls the current value at conversation time. The KB is for things that change monthly. Live integrations are for things that change daily.
A Realistic First-Week KB
If you're starting from scratch, you don't need 200 documents. You need maybe 8-12 well-written ones. A typical small-business starting set looks like:
/kb
about.md — who you are, what you do, where
contact.md — how to reach a human, hours, response times
pricing.md — plans, packages, what's included
services.md — detailed offerings, what each includes
faq.md — top 20-30 customer questions in their words
returns.md — refund and return policy
shipping.md — or scheduling, or whatever applies
warranty.md — or service guarantees
process.md — what happens after they buy / book
policies.md — terms, privacy notes a customer might ask about
That's it. That's enough to launch. You add more as real customer questions reveal gaps.
How This Fits Into an AI Agent Build
When I build a custom AI agent through the Pilot Agent service, KB buildout is the Week 1 deliverable — before any prompt engineering, before any integrations, before any UI work. We pull your existing content, restructure it into clean Markdown, identify what's missing based on the questions your customers actually ask, and write the gaps. By the end of week one, your agent has source material it can actually retrieve from.
Skip this step and you're tuning a Ferrari with bad gas.
If you're ready to launch an AI agent that gives accurate, on-brand answers — and want the knowledge base built right the first time — the AI Pilot Agent is where to start. Or get in touch and we'll talk through what your specific KB needs to cover.