Design a RAG System

The ask

Staff ask questions in plain words. The assistant answers from the organisation's documents, such as policies, manuals and project pages.

Every answer must point to its sources. A reader can open the cited document and check the answer in seconds.

A language model can also produce a hallucination, an answer that sounds fluent but is not true. Questions about internal documents make that failure likely, because the model never saw those documents in training.

The design therefore retrieves first and generates second. The model's job narrows to reading supplied passages and writing from them.

Estimate the scale

The corpus holds 10,000,000 documents. The average document runs to 2,500 tokens.

A chunk is a short passage cut from a document. Chunking is the splitting of documents into such passages, here 500 tokens each.

2,500 / 500 = 5 chunks per document.

10,000,000 documents × 5 chunks = 50,000,000 chunks in total.

Each chunk is stored as an embedding of 4 KB.

50,000,000 × 4 KB = 200,000,000,000 bytes, which is 200 GB of vectors.

The answer budget is about 2 seconds from question to finished answer.

Data and features

Documents arrive from the organisation's stores, such as file systems, wikis and ticket systems. An ingestion pipeline collects new and changed documents.

Chunking keeps each chunk at about one idea. A chunk of 500 tokens holds a few paragraphs, small enough to quote and large enough to keep its meaning.

Chunks overlap slightly at the edges. A sentence split across a boundary stays whole in at least one chunk.

Every chunk carries metadata beside its text. The metadata names the source document, its date and the readers allowed to see it.

Each chunk is embedded once and written to a vector index. A changed document is re-chunked and re-embedded, and its old chunks are removed.

Updates reach the index in minutes. Answers can cite a policy changed this morning, which retraining a model could never match.

High-level design

The system has two paths. An offline path turns documents into indexed chunks. An online path turns a question into an answer with citations.

On the online path, the question is embedded and matched against the index. The best chunks are placed in a prompt, and the language model writes the answer from them.

App servers hold the question flow. The vector index, the document store and the model fleet each scale on their own numbers.

The document store keeps the original documents. Citation links point there, so a reader always lands on the full source.

Repeated questions are common inside one organisation. A cache of recent question-and-answer pairs cuts model cost, and it is cleared when a cited document changes.

Training the model

The language model is not retrained for the corpus. Its training stays general, and the organisation's knowledge lives in the index instead.

That split is the economic core of the design. Indexing a changed document costs a few embeddings. Retraining would cost a full training run and would still go stale.

The embedding model is picked by offline tests. Sample questions with known source documents show which model retrieves the right chunks.

Fine-tuning stays available for the language model's tone and answer format. The facts never depend on it, because the facts live in the chunks.

Serving predictions

The question arrives and is embedded with the same model used for the chunks.

Vector search returns the top 50 chunks by similarity.

Reranking is a second, heavier scoring pass over those chunks. It re-orders them by how directly each one answers the question, and it keeps the top 5.

The top 5 chunks go into the prompt.

5 chunks × 500 tokens = 2,500 tokens of evidence, well inside the 8,000-token context window, beside the question and short instructions.

The whole answer must land in about 2 seconds. Retrieval and reranking get a small share of that budget, and generation gets the rest.

The answer streams to the reader as tokens are produced. Citations travel with the answer, attached to the sentences they support.

Grounding and citations

Grounding means every claim in the answer is supported by a retrieved chunk. A grounded answer can be checked, sentence by sentence, against its citations.

The prompt instructs the model to answer from the chunks only. Facts outside the chunks stay out of the answer, even facts the model holds from its training.

When the chunks do not answer the question, the assistant says so. That refusal is the main defence against hallucination.

A checker compares each answer sentence with its cited chunk. Sentences without support are removed or flagged before the answer is shown.

Deep dive: who may see what

Documents carry permissions, and chunks inherit them. A chunk from a restricted document is restricted in the index.

Retrieval filters by the questioner's permissions. The filter runs inside the search, before reranking and before any chunk reaches a prompt.

Filtering after generation is too late. The model has already read the chunk, and its answer can leak the content in other words.

Permissions arrive as group memberships from the organisation's sign-in system. A person in the finance group retrieves finance chunks, and nobody else does.

Every retrieval is logged with the user and the chunks used. The log makes leaks traceable and permission mistakes correctable.

The answer cache is keyed by permission group as well as by question. One group's cached answer is never served to another group.

Evaluation and trade-offs

Offline evaluation uses questions with known source documents. Retrieval scores measure whether the right chunks reach the top 5.

Raters grade answers for support. An answer scores well only when its citations actually back its sentences.

Online signals include citations opened, answers copied and questions re-asked. A re-asked question is a quiet failure mark.

Larger chunks carry more context and dilute the match. Smaller chunks match sharply and can lose the surrounding meaning.

The 500-token chunk size is the balance the design starts from, and tests move it.

The index is the knowledge, and it can change every minute. The model stays fixed, which is what makes the answers both current and checkable.

From corpus to prompt