Design an AI Agent System

The ask

An agent is a program that uses a language model to choose the next action for a task.

Tasks include researching a topic, fixing a failed job and preparing a report.

A chat answer ends with text. An agent task ends with work done in other systems.

The customer gives a goal and limits. The system plans, acts and reports the result.

Estimate the scale

The service starts 1,000,000 agent tasks a day.

1,000,000 / 86,400 gives about 12 tasks started per second.

A planning figure is 8 model calls for one average task.

1,000,000 x 8 = 8,000,000 model calls a day at that average.

One task may make at most 20 model calls, as a hard bound.

1,000,000 x 20 = 20,000,000 model calls a day if every task hit the bound.

20,000,000 / 86,400 gives about 231 model calls per second at the bound.

Capacity is planned between the average and the bound.

Data and features

Records of finished tasks are the main training data. Each record holds the goal, the steps, the results and the final outcome.

The outcome is the label for a task record. Human reviewers label tasks that failed or caused a problem.

Each available action has a written description. The description states what the action does, its inputs and its risks.

Serving facts include the customer's permissions. Serving facts also include the remaining call budget for the task.

Task records are stored per customer. One customer's records never appear in another customer's context.

High-level design

A task arrives through an API and enters a queue. A worker takes the task and starts the loop.

Tool use is the model asking the system to run a named action. The system runs the action and returns the result.

A separate checker enforces budgets and permissions. The model cannot raise its own limits.

Agent memory is the information kept between steps and between tasks. Different kinds of memory are kept for different lengths of time.

Each step is written to a log before the next step starts. A crashed worker can resume from the log.

The final answer summarises the work done. The answer also lists the evidence behind the result.

Training the model

The base language model is fine-tuned on task records. Training teaches step planning and action choice.

Training also teaches stopping when the goal is met. Extra steps waste calls and add risk.

Failed tasks are valuable examples. Reviewers mark the step where a failed task went wrong.

Good records show recovery after a failed action. The model learns to change approach, not to repeat blindly.

Serving tasks

Workers hold no task state in memory. Task state lives in a database, so any worker can continue a task.

A task can run for minutes. The customer gets progress updates while the task runs.

A retried action carries the same idempotency key as the first try. A payment action therefore runs once, even after a crash.

A task that passes its time limit stops and reports. A stopped task is a result, not a hang.

Deep dive: the bounded loop

The model reads the goal and the results so far.

The model chooses one action, or decides the task is done.

The checker approves or blocks the chosen action.

The system runs the approved action and records the result.

The loop has hard bounds. One task may make at most 20 model calls.

A task that hits the bound stops and reports its progress. Work is handed back, not lost.

Bounds also cover time and spending. Any bound can stop the loop.

A confused model can repeat a failing action. The bound turns an endless loop into a stopped task.

Deep dive: scoped tools and approval

Scoped means limited to the actions and data a task needs. Each task receives a small set of allowed actions.

An action outside the set cannot be run. The checker blocks the action before any system changes.

Reading actions are safer than writing actions. Writing actions get tighter limits and more checks.

Human approval is required before a task spends more than 100 dollars. Approval is also required before deleting data or sending a message outside the customer's organisation.

An approval request shows the action, the reason and the expected result. Silence is not approval. The task waits or stops.

Deep dive: memory tiers

Working memory holds the current goal and the step results. Working memory clears when the task ends.

Project memory holds facts for ongoing work for one customer. Project memory is stored only with the customer's permission.

Long-term memory holds stable preferences, such as a report format. Long-term memory changes slowly and is reviewed by the customer.

Each tier has its own lifetime and its own access rules.

Wrong long-term memory poisons later tasks. Customers can view and delete stored memories.

Stored text can carry a prompt injection from an older task. Memory is screened like any other input.

Deep dive: the autonomy ladder

The autonomy ladder is a set of levels. Each level raises what the system may do alone.

Level one suggests actions and takes none.

Level two acts after approval for each action.

Level three acts within set limits and reports afterwards.

A customer moves up a level after a clean record on real tasks.

A serious mistake moves the setting back down a level. Autonomy is earned again, not assumed.

Evaluation and trade-offs

A fixed set of test tasks runs before each release. Each test task has a known safe result.

Serving reports show success rate, calls per task and cost per finished task. Fewer calls for the same success means a cheaper task.

New versions start on internal tasks. Customer tasks follow after clean internal results.

Tighter bounds stop runaway tasks early. Tighter bounds also cut off slow, careful work.

More autonomy saves human time. More autonomy also gives a mistake more room to spread.

Scoped actions keep a mistake small. Approval keeps the largest actions human.

The agent system in numbers