Design Text-to-Video Generation

The ask

The product takes a written description and returns a short generated video clip.

Text-to-video is that turning of words into moving pictures.

The first version makes one 5-second clip per prompt, without sound.

Results arrive minutes later. The product is a creation tool, not a live conversation.

Quality means the clip matches the description and moves smoothly. Flicker and warped shapes fail the clip.

Estimate the scale

The service expects 1,000,000 video requests a day.

1,000,000 requests ÷ 86,400 seconds = 12 requests per second.

The rate sounds small. The cost per request is not.

One clip is 5 seconds of video. Generating it takes about 5 minutes, or 300 seconds, of GPU time.

12 requests per second × 300 seconds = 3,600 clips being generated at any moment.

1,000,000 clips × 5 seconds = 5,000,000 seconds of new video a day.

Data and features

Training data pairs video clips with written descriptions of what happens in them.

The model must learn two things at once. It learns what things look like, and how they move over time.

Motion is the extra signal over still pictures. A description of a dog running left must match the direction and the speed, not only the dog.

Good video data is scarce. Clips need clear rights, steady quality and descriptions that name the action, not just the scene.

Duplicates and near-duplicates are removed. The model should not memorise a few famous clips.

People check samples by hand. Clips with warped motion or wrong descriptions are dropped.

High-level design

A user sends a prompt to an app server.

A first check rejects prompts asking for banned content.

The server creates a generation job, stores its state, and returns a job number at once.

GPU workers run jobs from a queue. Each job moves through queued, generating, checking, ready and failed states.

Finished clips go into blob storage, and delivery uses a CDN. The user gets a notification that the clip is ready.

Job state lives in a database. A user can close the app and collect the clip later.

Training the model

Video generation uses diffusion, the same method as image generation, extended across frames.

The model removes noise from a whole clip at once, guided by the prompt. Frames are made together, so movement stays consistent.

Training needs far more compute than image training. Every example is many pictures plus their motion.

Training runs in batches on large GPU clusters. Progress is saved often, so a failed run resumes instead of restarting.

A new version is tested on a fixed prompt set before release. People score flicker, warped faces and broken motion.

Serving predictions

1,000,000 clips × 300 seconds = 300,000,000 GPU-seconds of generation a day.

Capacity planning starts from GPU time, not request rate. Twelve requests a second still keep 3,600 clips in flight, and about 3,600 GPUs busy in a one-job-per-GPU design.

Jobs are admitted only while the queue stays under a stated length. Past that length, new requests are refused with a clear retry hint, instead of joining an endless wait.

A cheaper preview can be made first. A low-resolution draft lets the user fix the prompt before paying for the full clip.

Repeat views never touch a GPU. Finished clips are served from storage and the CDN.

Deep dive: the five-minute wait

Waiting is the central product problem. A user who hears nothing for 5 minutes assumes the job died.

Progress is shown as job state and elapsed time. A fake percentage would lie, because generation does not advance evenly.

An early draft frame proves the job is alive and on-topic. The user can cancel a wrong-headed clip before it burns more GPU time.

A notification arrives when the clip is ready. It reaches the user in the app, and by email if the user has left.

Cancellation stays possible at every state. The interface states plainly that GPU time already spent is gone.

A failed job is marked failed with a reason and a free retry. It is never left spinning.

Checking prompts and clips

Checks run twice. The prompt is checked before generation, and the finished clip is checked before delivery.

Checking a finished clip samples its frames. Checking every frame would cost almost as much as making the clip.

A second model, trained for content moderation, flags banned results in the sampled frames.

Prompts asking for a real person's likeness in a harmful scene are refused at the prompt stage.

Borderline clips go to human review before delivery.

Reports from viewers become labelled examples for the checking model.

Evaluation and trade-offs

Raters score match to the prompt, smoothness of motion, and stability of faces and objects.

A fixed prompt set tests every version. Action prompts are included, because still scenes are easy and motion is where versions differ.

Online signals include downloads, shares and re-generations with edited prompts. Heavy editing suggests the first clip missed the description.

Longer clips multiply GPU time, and higher resolution multiplies it again. The design buys quality with minutes and money, or speed with rougher clips.

The scale in numbers