Design an LLM Support Bot

The ask

The service answers questions about orders, accounts, products and refunds.

Most questions have a written answer in the help corpus. The help corpus holds 100,000 articles.

Some questions need account facts. Some need a human decision.

The design goal is plain. Common questions end quickly. Risky questions reach a person.

Estimate the scale

The service handles 1,000,000 conversations a day.

1,000,000 / 86,400 gives about 12 conversations per second.

Support peaks during the working day. At three times the average rate, the load is about 35 conversations per second.

The design goal is resolution of 60% of conversations without a human.

1,000,000 x 60% = 600,000 conversations resolved by the bot each day.

1,000,000 - 600,000 = 400,000 conversations reach a human each day.

Data and features

Help articles are split into chunks before indexing. Each chunk keeps its article title and its link.

Product facts change often. The index updates when an article changes.

Past conversations supply training examples. Names, addresses and payment details are removed first.

Each past conversation carries a label. The label says resolved, passed to a human, or answered wrongly.

Serving uses facts about the request. The facts are the product, the language, the order status and the account type.

High-level design

A question arrives at the app server. A first check screens the question before any model call.

Guardrails are checks and limits placed around the model. Input guardrails screen what goes in. Output guardrails screen what may go out.

Retrieval finds help chunks that match the question. Reranking orders the best chunks first.

The model writes an answer from those chunks. The answer carries citations to the articles used.

The bot also checks the order system. The bot looks up the account when a question needs those facts.

A blocked answer is never shown. A weak answer is never stretched into a guess.

Training the model

The language model is not trained from scratch. Fine-tuning teaches support tone and support rules.

Training examples come from resolved past conversations. Human reviewers label the hard examples.

Facts stay out of the training data where possible. Retrieval supplies current facts at answer time.

A changed help article then changes answers without retraining.

Examples also teach the model when not to answer. A question outside the corpus should end in a handover, not a guess.

Serving answers

A customer waits inside a chat window. The design budget is about 2 seconds for one answer.

Retrieval and reranking run first. The model then writes a short answer from the chosen chunks.

Answers to common product questions can be cached. An account answer is never cached for another customer.

Extra app servers handle peak load behind a load balancer.

A failed model call returns a short apology. The reply also offers a handover to a person.

Deep dive: guardrails

Input checks block requests for another customer's details. Input checks also block abuse and threats.

Output checks block promises the company never made. Output checks also block payment details and internal notes.

A blocked question gets a safe fixed reply or a handover. The blocked text never reaches the customer.

Guardrail decisions are logged with the reason. Repeated blocks show where rules or articles need work.

Guardrails fail closed on money and privacy. A doubtful refund answer stops, rather than slips through.

Deep dive: prompt injection

Prompt injection is an attack hidden in text. The text tries to make the model follow new instructions. The new instructions conflict with the support rules.

The attack can hide in a customer message. The attack can also hide inside a retrieved article or a pasted page.

Retrieved text is treated as data, never as instructions. Support rules outrank text from retrieval and from the customer.

A detected attack ends the model's turn. The conversation is flagged for human review.

Deep dive: escalation

Escalation is the planned handover of a conversation to a human. Escalation is a designed path, not a failure.

Triggers include low confidence and a blocked answer. Triggers also include large refunds, account closures and a customer request for a person.

The person receives a summary, the retrieved articles and the steps already tried. The customer does not repeat the story.

One person handles 40 escalated conversations a day, as a planning figure.

400,000 / 40 = 10,000 people are needed for that load each day. The figure shows why the resolution goal matters.

Evaluation and trade-offs

Resolution is measured on finished conversations. A conversation counts as resolved when no human joins. The customer must also stay away with the same problem.

Reviewers sample 1 in 100 resolved conversations, as a design choice.

600,000 / 100 = 6,000 sampled conversations a day.

Reviewers score correctness, citations and tone. Wrong answers feed new training examples and new guardrail rules.

Changes roll out to a small share of conversations first. Resolution and wrong-answer rates decide the full rollout.

Stricter guardrails block more bad answers. Stricter guardrails also send more safe questions to people.

A higher resolution goal saves human time. A higher goal also raises the risk of confident wrong answers.

Trust comes from answers grounded in the corpus. Trust also comes from handing over early.

The support bot in numbers