Classic design · Senior
Self-driving vehicle perception
Classic interview design. Key points: semantic segmentation, sensor labeling, and critical safety evaluation.
Short answer
Classic interview design. Key points: semantic segmentation, sensor labeling, and critical safety evaluation. Designing the perception of an autonomous vehicle is a ML system design problem: raw data from several sensors must become driving decisions. The source organizes the stack into four blocks.
01 · CONTEXT
The stack of a self-driving car
Designing the perception of an autonomous vehicle is a ML system design problem: raw data from several sensors must become driving decisions. The source organizes the stack into four blocks. Perception turns raw sensor data into an understanding of the world, prediction estimates the future trajectory of agents, planning makes decisions and generates the trajectory, and the controller generates the commands to accelerate, brake, and steer.
The raw sensor data reaches perception, which turns it into an understanding of the world.
Prediction receives the detected objects and estimates each agent's future trajectory.
Planning receives the route, the map, and the prediction of neighbors and generates the trajectory.
The controller translates the trajectory into acceleration, brake, and steering commands.
The same source notes that latency is on the order of milliseconds for some tasks.
02 · SENSORS
Camera, LiDAR, and RADAR
No sensor covers all conditions, which is why the system combines several. The table summarizes what the source says about each one.
| Criterion | Camera | LiDAR | RADAR |
|---|---|---|---|
| What it offers | Classification, segmentation, and localization. | Measures distance with lasers or light and adds depth, as a point cloud. | Uses radio waves and works in any condition. |
| Limitation | Problems at night and in extreme conditions such as fog and heavy rain. | Works in the dark, but still fails with rain or fog noise. | Very noisy, lower spatial resolution, and interference from other radio systems. |
03 · PERCEPTION
Detect, segment, and track
Perception detects traffic lights, pedestrians, signs, lanes, and parking spots. In image detection, two-stage detectors use a region proposal network to find regions of interest, and one-stage detectors skip proposal generation and output the boxes directly, which makes them attractive for real time.
With the point cloud, three-dimensional detection inherits ideas from image detection and requires a lot of computation. To follow objects over time, tracking can use probabilistic methods such as the EKF or ML models.
In segmentation, semantic segmentation classifies each pixel of the image into a class, and instance segmentation combines object detection with semantic segmentation to classify the pixels of each instance. The public description of Educative's module on this topic covers image segmentation in self-driving cars.
04 · PREDICTION AND PLANNING
From motion to decision
Motion prediction estimates where each object will be, given several past frames, and takes perception and the high-definition map as input. The challenge cited is multimodality: the future is uncertain.
Planning receives the route, the context map, and the prediction of neighboring agents. The source also describes a motion validation that checks for collision and red lights, rejecting and ranking the options.
| Abordagem | Example in the source | What it learns jointly |
|---|---|---|
| Perception with prediction | Fast & Furious and IntentNet, both from Uber. | Detection, tracking, and short-term motion prediction (Fast & Furious); detected objects, trajectory, and intent (IntentNet). |
| Prediction with planning | ChauffeurNet, from Waymo. | Prediction and planning in a single network, through imitation learning. |
| End to end | Reinforcement learning from wayve.ai. | A driving policy, with no high-definition map and no hand-written rules. |
05 · DATA AND LABELING
Where the labels come from
For the example of predicting pedestrian crossing, the source recalls that predicting pedestrian actions is harder than predicting vehicle actions, since it depends on pose and activity. Collection must be diverse, with varied locations, behaviors, traffic, and lighting.
Na rotulagem annotation starts with human labelers and evolves into an automatic labeling system. Each frame gets a box, pose, activity tags, and pedestrian attributes, and each video gets weather conditions and time of day. Data augmentation includes rotation, mirroring, and brightness adjustment.
The fleet sends its sensor logs to storage.
The logs enter a queue and the labeling service consumes that queue.
With the labeled data, training generates models that go to the model registry.
Evaluation compares the system's decisions with those of an expert driver on historical logs.
Approved models return to the fleet, and the source recommends updating and retraining with new data.
In later iterations, the cases where the driver had to intervene are sent to the labelers and enter the dataset. To run in real time, the source suggests optimizing the model with pruning, quantization, and TensorRT.
06 · CRITICAL EVALUATION
Per-component and end-to-end metrics
The source separates metrics by component and end to end. For components, object detection uses mAP and crossing detection uses precision, recall, and F1. End to end, the metric is manual intervention, in addition to simulation errors over historical logs of scenes recorded with an expert driver, comparing the system's decision with the driver's.
Frequently asked questions
What is Self-driving vehicle perception?
Classic interview design. Key points: semantic segmentation, sensor labeling, and critical safety evaluation.
What does the Self-driving vehicle perception design cover?
It is organized in these sections: The stack of a self-driving car, Camera, LiDAR, and RADAR, Detect, segment, and track, From motion to decision, Where the labels come from, and Per-component and end-to-end metrics.
How long is Self-driving vehicle perception and what level is it?
It takes about 16 minutes to read and is rated Senior level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.