Classic design · Staff
Text→video
Classic interview design. Key points: spatiotemporal DiT, video VAE, temporal consistency, FVD, and cost per second.
Short answer
Classic interview design. Key points: spatiotemporal DiT, video VAE, temporal consistency, FVD, and cost per second. OpenAI's technical report on Sora explores large-scale training of generative video models. They are text-conditional diffusion models, trained jointly on videos and images of varying durations, resolutions, and aspect ratios; the largest of them can generate a minute of high-fidelity video.
01 · CONTEXT
The problem of generating video from text
OpenAI's technical report on Sora explores large-scale training of generative video models. They are text-conditional diffusion models, trained jointly on videos and images of varying durations, resolutions, and aspect ratios; the largest of them can generate a minute of high-fidelity video.
The topic is also a chapter in ByteByteGo's GenAI interview book, devoted to text-to-video generation. The report warns that it does not include model or implementation details; so whatever goes beyond it in this case is proposed design, not a description of Sora.
02 · FROM VIDEO TO PATCHES
A single representation for video and images
The central idea is to treat video the way a language model treats text: instead of text tokens, Sora uses visual patches. To get to them, the video is first compressed into a lower-dimensional latent space and then decomposed into spacetime patches.
A compression network takes the raw video and returns a latent compressed in time and space. Sora is trained on and generates within this compressed space, and a corresponding decoder maps the generated latents back to pixels.
From the compressed video, a sequence of spacetime patches is extracted to serve as transformer tokens. Because this representation accepts varying resolutions, durations, and aspect ratios, it also serves for images, which are single-frame videos.
The descriptive captioner produces text captions for the videos in the training set.
The compression network takes the raw video to a latent representation compressed in time and space.
The spacetime patches come out of the compressed latent and act as transformer tokens.
The transformer receives noisy patches and the text conditioning, and is trained to predict the clean patches.
03 · TRAINING AND SCALE
Diffusion with a transformer and native size
The model is a diffusion transformer: it receives noisy patches, plus conditioning such as the text prompt, and is trained to predict the original clean patches. The report concludes that diffusion transformers also scale well as video models, and that sample quality improves markedly with more training compute.
Earlier approaches usually resize, crop, or trim videos to a standard size. The report says that training at native size brings several benefits.
| Criterion | Standardize the size | Native size |
|---|---|---|
| Data preparation | Resizes, crops, or trims videos to a standard size | Trains on the data at its native size |
| Enquadramento | With square crops, the subject sometimes appears only partially | Improves composition and framing |
Another benefit is sampling flexibility: you can prototype at smaller sizes before generating at full resolution, with the same model.
04 · CAPTIONS, CONSISTENCY, AND LIMITS
What the text controls and where the model fails
Training text-to-video requires many videos with corresponding captions. The report trains a highly descriptive captioner and uses it to caption all videos in the training set; descriptive captions improve text fidelity and overall quality.
At inference time, a GPT converts short user prompts into longer, more detailed captions, which are sent to the video model.
Maintaining temporal consistency in long videos is a known challenge of video generation. The report says Sora often, but not always, manages to model short- and long-range dependencies.
05 · SERVING A REQUEST
A service design for the interview
This service design is a proposal for the interview, assembled from the pieces described in the report; it does not describe OpenAI's infrastructure. The proposal uses a job queue, delivers the result through CDN and treats the cost per second of video as a design metric.
The user submits a short prompt; an expander, playing the role of the report's GPT, converts it into a detailed caption for the video model.
Highlighted in the diagram: the job queue and the GPU worker, connected by an asynchronous edge.
On the worker, the video size is set by laying out randomly initialized patches in a grid of the appropriate size, and the model, trained to predict clean patches, processes them.
The decoder maps the generated latents back to pixel space.
Highlighted in the diagram: the storage of the final video and the CDN that delivers it to the user.
Frequently asked questions
What is Text→video?
Classic interview design. Key points: spatiotemporal DiT, video VAE, temporal consistency, FVD, and cost per second.
What does the Text→video design cover?
It is organized in these sections: The problem of generating video from text, A single representation for video and images, Diffusion with a transformer and native size, What the text controls and where the model fails, and A service design for the interview.
How long is Text→video and what level is it?
It takes about 17 minutes to read and is rated Staff level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.