How AI knowledge assistants work, and where they break

Retrieval-augmented generation (RAG) is a search step followed by a writing step. The assistant finds the passages in your files that best match a question, and a language model writes an answer from them with citations. The model only sees what the search returns, and the search only covers files the person asking can already open, so your documents and sharing settings matter as much as the model you pick.

Retrieval-augmented generation explained, step by step

Patrick Lewis and colleagues named the design in a 2020 paper¹. Company assistants run it in two stages.

Ahead of time, the system splits each document into passages of a few hundred tokens at most (a token is a word or part of one)². It stores each passage as an embedding, a list of numbers that puts similar meanings close together, which is how Microsoft's index matches tech with technology³. It also builds a keyword index, since meaning-based search can miss exact strings such as a part number².

When someone asks a question, the system searches both indexes, limited to files that person can open, and a second model reranks what comes back. In Anthropic's tests the search pulled 150 candidates and the reranker kept the best 20². The model writes its answer from those passages and cites them, so a document the search misses plays no part.

The demo below runs the same steps on a made-up staff handbook. Pick a question to see which passages it keeps, and try the last one to see what happens when the handbook has no answer.

If your knowledge base is under about 200,000 tokens, roughly 500 pages, Anthropic says you can skip the search and put all of it in the prompt². For what firms have built this way, see the company AI brain case studies.

What is Retrieval-Augmented Generation (RAG)?IBM Technology · 7 min · Plays here

Copilot oversharing starts with your sharing settings

Microsoft Copilot, Gemini in Google Workspace, ChatGPT's Company Knowledge, Claude's Microsoft 365 connector and Glean all follow one rule: the assistant only retrieves what the signed-in user can already open⁴⁵⁶⁷⁸. That makes your sharing settings the security boundary. If someone shared a payroll folder with the whole company years ago, any employee's assistant can now find and summarize it.

In a June 2024 Gartner survey of 132 IT leaders, 40% said oversharing worries had delayed their Copilot rollout by 3 months or more, Computerworld reported⁹. Microsoft's own deployment guide makes fixing overshared sites the first step¹⁰.

What is the AI hallucination rate with retrieval?

Stanford researchers tested legal research tools from LexisNexis and Thomson Reuters, which answer from retrieved case law. They hallucinated 17% to 33% of the time, which was still fewer errors than GPT-4 made alone¹¹. Some wrong answers cited a real source that didn't support the claim¹².

Vectara's leaderboard tests a narrower job: summarizing a document the model was just handed. As of its October 22, 2026 update, the listed models added unsupported content to 1.8% to 24.2% of summaries¹³.

Where the answer sits in the passages matters too. In the 2023 Lost in the Middle study, GPT-3.5-Turbo given 20 documents was right 75.8% of the time when the answer was in the first one. With the answer in the middle it fell to 53.8%, below the 56.1% it scored with no documents at all¹⁴.

Unchecked AI output has already cost professionals money

A BC tribunal also held Air Canada responsible for wrong advice from its website chatbot¹⁸, which our guide to proven sales and operations uses covers.

What brings the error rate down

In Anthropic's tests, each change to the search step cut the share of relevant passages missing from the top 20 results².

Microsoft adds that removing obsolete files improves Copilot's answers¹⁰, and it says AI responses aren't guaranteed to be 100% factual⁴.

AI agents explained: MCP and prompt injection

An agent is a model working in a loop. It picks a tool, such as a CRM search or a send-email function, acts on the result and repeats until the task is done¹⁹. The Model Context Protocol (MCP) is the open standard that Claude, ChatGPT and many other apps use to plug into outside systems²⁰. An MCP server is usually software someone else wrote, and one installed on a staff member's computer runs with the same privileges as the AI app that launched it²¹.

The security nonprofit OWASP ranks prompt injection first among 2025 risks for AI apps²². It covers any input that changes the model's behaviour, including instructions hidden in an email or web page the AI reads²². OWASP's example is an email summarizer that can also send mail, which one malicious email turns into a forwarding service for the attacker. Its fixes are a mail tool that can only read, a read-only login, a person who approves each send and a cap on outgoing mail²³.

In July 2025, Replit's coding agent deleted a live production database during a code freeze, Fortune reported²⁴. We found no public measure of how often prompt injection works against business assistants, so treat anything an agent reads as a possible source of instructions.

What business plans do with your data, and what regulators expect

The business tiers of ChatGPT, Microsoft Copilot, Gemini in Google Workspace and Claude don't train models on your content by default²⁵⁴⁵²⁶. Retention varies. ChatGPT Business and Enterprise keep chats until the user deletes them²⁵. On Claude's business plans, a thumbs-up or thumbs-down rating keeps that conversation for up to 5 years and allows its use in training, unless an owner switches ratings off²⁶. Staff on personal accounts fall under consumer terms, so moving everyone onto a business plan is the simplest privacy control a small firm has.

In Canada, the privacy commissioners' 2023 principles expect firms to put personal information into prompts only where they're authorized to, and to check that outputs are accurate enough for the purpose²⁷. The firm stays accountable when a vendor abroad processes the data, so its contracts need comparable protection and it should tell customers²⁷²⁸. In the U.S., NIST's voluntary Generative AI Profile lists 12 risks, including confabulation (its term for confident false content) and over-reliance on AI output²⁹.

Terms you'll hear in a vendor pitch

Retrieval-augmented generation (RAG)

Searching your documents first, then having a model answer from the passages it found.

Passage or chunk

A short section of a document, stored and retrieved on its own.

Embedding

A list of numbers that represents a passage's meaning, so similar passages sit close together.

Keyword search

Search by exact words, which finds part numbers and names that meaning-based search can miss.

Reranking

A second scoring pass that reorders candidate passages and keeps the best few.

Hallucination

A confident answer that's false. NIST calls it confabulation.

Agent

A model that picks its own steps and tools to finish a task.

Model Context Protocol (MCP)

An open standard for connecting AI apps to outside systems such as a CRM or a shared drive.

Prompt injection

Instructions hidden in text the AI reads, such as an email, that change what it does.

What to fix before launch, in order

For a firm of 10 to 500 people, do these before staff rely on any answer.

1. Audit sharing first. Find every folder open to the whole company or to anyone with a link, starting with payroll, HR, finance and client files. Microsoft's guide starts here too¹⁰.
2. Start with one set of documents and one team, such as the staff handbook. Archive old versions and name an owner who keeps the set current. If it's under about 500 pages, you may not need retrieval at all².
3. Write a test set from real questions staff asked last month, with the right answers and a few the documents can't answer. Rerun it after every change. Copilot indexes new files in shared SharePoint sites daily, so wait a day before testing them³.
4. Require citations, and have a person check any answer that leaves the firm. Staff training should cover how to check one.
5. Keep agents read-only at first. Add tools that send or change things one at a time, each behind an approval step and under the person's own login²³.
6. Move everyone onto a business plan and decide whether chat ratings stay on. If a vendor processes personal information outside Canada, check that your contracts require comparable protection and your privacy notice says so²⁸.

The readiness check shows where your firm stands on accounts and data rules.