Design Object Detection

The ask

A self-driving car must know what sits around it right now.

The answer lists each object, its kind and a box around it. A frame is one still picture from a camera, and a box marks one object's edges in that frame.

Kinds include people, other vehicles, signs, lights and lane markings.

A late answer is a wrong answer, because the car keeps moving.

The design budget is about 100 milliseconds from camera to answer.

Estimate the scale

One car carries 8 cameras. Each camera produces 30 frames per second.

8 × 30 = 240 frames per second per car.

The fleet is 1,000,000 cars.

240 × 1,000,000 = 240,000,000 frames per second across the fleet.

No network can carry that stream, so detection must run on the car itself.

The design uploads 1 frame in 10,000 for retraining data. 240,000,000 ÷ 10,000 = 24,000 frames per second.

Data and features

Training frames come from the uploaded stream and from test cars driven by people.

People draw boxes around objects in saved frames. Those boxes and kinds are the labels for training.

A feature is one measured fact from the frame, such as an edge or a shape. Motion across frames is a feature too.

Rare cases are scarce, such as a person in a costume or a fallen tree.

Frames where the model was unsure are worth more labels than easy highway frames.

Weather, light and place are recorded with each frame, so gaps in the training data show up.

High-level design

Cameras feed a computer inside the car.

The detection model runs on that computer and answers inside the car.

A tracker follows each object across frames. One sighting becomes a moving path.

The car's driving software reads the object list and plans around it.

A small upload path sends chosen frames and model doubts to the data center.

Servers in the data center store frames, arrange labels and train new models.

Training the model

Training uses labelled frames from many places, weather kinds and times of day.

The model learns to place boxes tightly and to name the kind correctly.

Training balances missed objects against false alarms. A false alarm is an object reported where none exists.

New models train on the newest uploads, so rare cases grow over time.

A new model must beat the current one on held-back frames before any car gets it.

Serving predictions

Serving happens on the car, inside the budget of about 100 milliseconds.

The on-car computer runs the model on every frame from all 8 cameras.

A frame that waits too long is dropped, because its answer would describe the past.

Model updates arrive as downloads while the car is parked or charging.

Each car reports how often the model was unsure, without sending every frame.

A failure in the model makes the car slow down and stop safely.

Deep dive: the hardest frames

Night rain, glare and fog hide the edges the model relies on.

A partly hidden person still counts, even when only a head and a shoulder show.

Within the upload rate, doubtful frames go first, so hard cases reach training often.

A synthetic frame is a made-up frame built by software, not a camera. Synthetic frames can fill rare gaps, such as a deer on a city street.

Hard frames are kept as a fixed test set that every new model must face.

A model that improves on easy frames but slips on hard ones does not ship.

Evaluation and trade-offs

Offline tests count missed objects and false alarms on held-back frames.

Misses on people count far more than misses on signs, so scores are kept per kind.

On-road tests run a new model in shadow first. Shadow running means the new model answers alongside the old one, but the car follows the old answers.

A bigger model spots more and runs slower, so its size is capped by the frame rate.

The design spends its budget on the car, because the fleet stream can never ride the network.

Object detection at a glance