Real architecture · Mid
Netflix: Linux performance analysis in 60 s
Real production architecture. Key points: Monitoring and Observability and Performance Antipatterns.
Short answer
Real production architecture. Key points: Monitoring and Observability and Performance Antipatterns. You log into a Linux server that has a performance problem. Netflix's post starts from a direct question: what should you check in the first minute? Netflix has a large Linux EC2 cloud and several tools for monitoring and observability.
01 · CONTEXT
A slow server and one minute to look
You log into a Linux server that has a performance problem. Netflix's post starts from a direct question: what should you check in the first minute? Netflix has a large Linux EC2 cloud and several tools for monitoring and observability.
Atlas handles cloud monitoring and Vector handles on-demand instance analysis. According to the post, these tools solve most problems, but sometimes you need to log into an instance and run the standard Linux tools. The command list is for that moment.
There is a Linux instance with a performance problem, and the question is what to check first.
Atlas does the cloud monitoring.
Vector does on-demand analysis of an instance.
When those tools are not enough, the engineer logs into the instance and runs standard Linux tools.
02 · THE TEN COMMANDS
One command for each resource
The checklist gathers ten commands that, in sixty seconds, give an overview of resource usage and running processes. The table links each command to the resource it helps examine and to what the post highlights.
| Comando | Recurso | What the post highlights |
|---|---|---|
| uptime | Overall load | The load average includes processes blocked in uninterruptible I/O |
| dmesg | tail | Kernel errors | Look for errors that could cause performance problems |
| vmstat 1 | CPU queue | The r column above the CPU count is saturation |
| mpstat -P ALL 1 | CPU per core | Shows CPU time per CPU and lets you check for imbalance |
| pidstat 1 | Processos | In the example, two java processes account for the CPU consumption |
| iostat -xz 1 | Disco | Utilization above 60% usually leads to poor performance; await is the average I/O time in milliseconds |
| free -m | Memory | Linux uses free memory for caches and can reclaim it quickly |
| sar -n DEV 1 | Network interfaces | In the example, eth0 receive reaches 22 Mbytes/s |
| sar -n TCP,ETCP 1 | TCP | Retransmits are a sign of a network or server problem |
| top | Overview | Makes it hard to see patterns over time |
03 · PRODUCTION EXAMPLE
Following the signals on a real server
The post illustrates the commands with outputs from a production system. Follow the example's signals, one by one, on the server under investigation.
Uptime shows the one-minute load average at 30, against 19 for the fifteen-minute one: a recent increase.
In vmstat, the CPUs appear well above 90% average utilization.
Pidstat points out which processes consume the CPU: two java processes, one of them with almost 16 CPUs.
On the traffic arriving at the server, sar shows eth0 receive at 22 Mbytes/s.
04 · HOW TO READ THE SIGNALS
Errors and saturation first
The post says to look for errors and saturation, which are easy to interpret, and only then look at utilization. Saturation is when a resource has more load than it can serve. This checklist is part of the USE method, which checks utilization, saturation and errors for all resources, and the post recommends paying attention to when a resource has already been checked and ruled out. This reasoning ties into the study of performance antipatterns.
05 · LIMITS AND NEXT STEPS
When the minute is not enough
To go deeper, the post points to a tutorial that walks through more than 40 commands. The post says that monitoring tools solve most problems and that only sometimes is it necessary to log into the instance.
Frequently asked questions
What is Netflix: Linux performance analysis in 60 s?
Real production architecture. Key points: Monitoring and Observability and Performance Antipatterns.
What does the Netflix: Linux performance analysis in 60 s design cover?
It is organized in these sections: A slow server and one minute to look, One command for each resource, Following the signals on a real server, Errors and saturation first, and When the minute is not enough.
How long is Netflix: Linux performance analysis in 60 s and what level is it?
It takes about 15 minutes to read and is rated Mid level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.