Linux Performance Checklist
USE Method
USE (Utilization, Saturation and Errors) is a methodology to analyze performance of any system. In summary, the idea is to look at utilization, saturation and errors for each resource in the data path to understand the system bottlenecks.
Checklist
Following is my reference checklist and one-liners I use for Linux performance analysis.
uptime
uptime provides the load average of the system for last 1, 5 and 15 mins. It's a quick way to check the load trend.
top
top provides real-time view of the running processes. However, it may miss the short lived processes. To capture the short lived processes, use atop. htop is similar to top but provides better UI.
ps: report snapshot of current processes
# list every process with process hierarchy
ps -ef f
# list every process with thread IDs
ps -e -fL
# list every process and output specific columns
ps -e -o pid,tid,cmd
pidstat: report stats for linux tasks
# display stats for tasks whose cmd name includes string cmd
pidstat -C cmd
# report io stats
pidstat -d
# display process command name and all its arguments
pidstat -l
# select tasks (processes) for which stats are to be reported
pidstat -p <pid1,pid2,...>
# report realtime priority and scheduling information
pidstat -R
# report page faults and memory utilization
pidstat -r
# report stack utilization
pidstat -s
# display stats for the associated threads
pidstat -t
# report CPU utilization
pidstat -u
# report values of some kernel tables
pidstat -v
# report task switching activity
pidstat -w
# reports most useful information
pidstat -p <pid> -huvdr
# display 5 reports at 2 second interval
pidstat -p 23047 -huvdr 2 5
mpstat: report processors related stats
# display 5 reports of stats for all processors at 2 second interval
mpstat -P ALL 2 5
vmstat: report virtual memory statistics
# display report every 2 seconds
# in: interrupts per second
# cs: context switches per second
vmstat -Sm 2
free: display amount of free and used memory in the system
# display report every 2 seconds
free -h -s 2
sar: collect, report or save system activity information
# report per process stats for specified processor or processors
sar -P ALL 2 2
# report system load and preassure-stall stats
sar -q CPU 2 2
sar -q LOAD 2 2
# report CPU utilization
sar -u ALL 1 1
# report paging statistics
sar -B 2 5
# report activity for each block device
sar -d 2 2
# report hugepage utilization stats
sar -H 2 2
# report network stats
sar -n DEV,EDEV 2 2
# report pressure-stall stats
sar -q MEM 1 1
# report memory utilization stats
sar -r ALL 1 1
iostat: report CPU stats and IO stats for devices and partitions
# display block io stats every 1 second
# first report is stats since boot
iostat -xmdz 1
# display block io stats every 1 second omitting the stats since boot
iostat -xmdzy 1
# display block io stats in human readable format
iostat -xmdzyh
ss: utility to investigate sockets
# show socket summary
ss -s
# display TCP sockets
ss -tna
# display Unix Domain sockets
ss -x
# display listening TCP sockets
ss -ltn
# display processes using sockets
ss -pn # all processes
ss -tpn # processes using TCP sockets
ss -xpn # processes using Unix Domain sockets
# display sockets in established state
# TCP states: listen, sync_recv, established, close_wait,
# last_ack, closed, sync_sent, established, fin_wait_1,
# fin_wait_2, time_wait, closed
ss -t state established # TCP sockets
ss -x state established # Unix Domain sockets
# display timer information for TCP connections
ss -to
# display only TCP sockets with destination port <port>
ss -tn dst :<port>
# display only SRC sockets with source IP <IP>
ss -tn src <IP>
# display internal TCP information for sockets with destination
# port <port>
ss -ti dst :<port>
iftop: display bandwidth usage on an interface by host
# display bandwidth usage for an interface in interactive mode
sudo iftop -i <interface> -Nn
vmtouch: virtual memory toucher
# check how many pages of a file is resident in memory
vmtouch -v <file_path>
# bring pages from disk to memory. page is guaranteed to be brought
# to memory but might be evicted before the command finishes.
vmtouch -tf <file|dir>
# evict mapped pages from file system cache. pages are guaranteed
# to be evicted but might be brought back before the command finishes.
vmtouch -ef <file|dir>
# bring pages from disk and lock them in memory.
vmtouch -lf <file|dir>
pcstat: get page cache statistics for files
# show all open maps for a given pid
sudo pcstat -pid <pid>