Design a Code Judging System

The ask

A problem states a task, input rules and output rules. A user submits a program as the answer.

The system runs the program against hidden tests. Each test feeds one input and expects one correct output.

A verdict is the result label for a submission. Common verdicts are accepted, wrong answer, too slow and crashed.

Submitted programs are untrusted. A program may loop forever, use huge memory or try to read hidden data.

Estimate the scale

1,000,000 submissions arrive a day.

1,000,000 / 86,400 = 12 submissions per second.

One judging run takes 2 seconds of one machine's CPU.

12 x 2 = 24 judge machines busy on average.

A contest peak runs at 100 times the average rate.

1,000,000 / 86,400 x 100 = about 1,157 submissions per second at that peak.

API and data

The submit operation names the problem, the language and the program text. The response returns a submission identity.

A status operation returns the state of one submission. A result operation returns the verdict, the run time and the memory used.

A submission record holds the user, the problem, the language, the program text and the final verdict.

Test data is stored apart from problems and never leaves the judging side. A user sees test outcomes, not hidden inputs.

High-level design

A front service accepts a submission and places it on a queue. The queue feeds a fleet of judge machines.

A sandbox is an isolated box for running untrusted programs. The box has strict limits and no path to the rest of the system.

A judge machine pulls one submission, builds the program and runs the tests inside a sandbox. Results go to a result store.

The front service reads results from the store. Judging stays off the request path, so a slow run never blocks a page.

Deep dive: sandboxing untrusted programs

Sandboxing is the practice of running untrusted work inside a sealed box with hard limits. The box is thrown away after the run.

The box allows no network access. The program cannot call out and cannot reach other services.

The box caps CPU time and memory. A program past a cap is stopped and judged by that limit.

The box exposes a small empty file area. Hidden tests are fed in as input, never as readable files.

The box limits how many processes a program may start. One submission cannot exhaust the judge machine.

Judging correctness

The program is built once for a submission. A build failure ends judging with a build verdict.

Each test runs the built program with the test input. The output is compared with the expected output.

Some problems accept many correct outputs. A checker program then validates the output against the input instead of comparing text.

Judging stops at the first failing test in practice mode. Contest mode records the full score.

Contest peaks and fairness

Contest submissions arrive in bursts near the start and near the end. The queue holds the burst in arrival order.

At peak the wait is the designed answer, with a wait budget of about 30 seconds. Extra judge machines are added before known contests.

Practice submissions use a separate queue. A contest burst cannot starve everyday judging, and everyday load cannot delay a contest.

Evaluation and trade-offs

A code judging system is a queue feeding sealed boxes. Safety comes from the sandbox, and fairness comes from the queue.

A fresh box for every run costs startup time. Reusing a box risks leftover state from untrusted work.

More judge machines cut waits and sit idle between contests. The queue smooths the gap.

Strict output comparison is simple and brittle. Checker programs cost more to write and judge fairly.

Judging scale