Linux Performance Analysis: USE Method, perf, eBPF, and System Profiling

The USE Method

For every resource, check:

• Utilization: % of time the resource is busy
• Saturation: amount of extra work queued
• Errors: error count

CPU Analysis

# Overall CPU utilization
top -1        # Per-CPU breakdown, press 1
htop          # Interactive, better than top
mpstat -P ALL 1  # Per-CPU stats every second

# CPU saturation (run queue)
vmstat 1      # r column = processes waiting for CPU
              # r > CPU count = saturation!
uptime        # 1/5/15 minute load averages

# What's using CPU
pidstat 1     # Per-process CPU every second
perf top      # Real-time kernel/user profiling
perf record -F 99 -a -g -- sleep 10  # Record flame graph data
perf report   # Analyze recording

Memory Analysis

# Memory overview
free -h              # Total/used/free/buffers/cache
cat /proc/meminfo    # Detailed breakdown

# Memory saturation (swapping = BAD)
vmstat 1             # si/so columns = swap in/out
iostat -xz 1         # Check swap I/O

# Memory leaks - per process
pmap -x <pid>        # Process memory map
smem --sort rss      # Sort by Resident Set Size (RSS)
valgrind --leak-check=full ./app  # Find leaks

# Page faults
perf stat -e page-faults ./app

Disk I/O Analysis

# I/O overview
iostat -xz 1         # %util, await (ms), r/s, w/s
                     # util > 90% = disk saturated
                     # await > 10ms = slow I/O

# What's doing I/O
iotop -ao            # Processes generating I/O
pidstat -d 1         # Per-process I/O stats

# File-level tracing (eBPF)
opensnoop            # Shows all open() system calls
filelife             # Short-lived files
ext4slower 10        # ext4 ops slower than 10ms

Network Analysis

# Network stats
netstat -s           # Protocol statistics
ss -s                # Socket summary
ss -ntlp             # Listening TCP sockets

# Network I/O
sar -n DEV 1         # Network interface stats
nethogs              # Per-process bandwidth
iftop                # Per-connection bandwidth

# Connection analysis
ss -n 'state time-wait' | wc -l   # TIME_WAIT connections
ss -n 'state close-wait' | wc -l  # CLOSE_WAIT (potential leak)

strace for System Call Tracing

# Trace a running process
strace -p <pid>
strace -p <pid> -e trace=read,write,open,close  # Filter syscalls

# Trace with timing
strace -T -p <pid>   # Show time spent in each syscall

# Trace a command
strace -o output.txt -T -tt ls /tmp

perf Flame Graphs

# 1. Record CPU profile
perf record -F 99 -g -p <pid> -- sleep 30

# 2. Generate flame graph (requires FlameGraph tools)
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg

# View in browser
open flame.svg

eBPF Tools (BCC/bpftrace)

# All execve calls (new processes)
execsnoop

# TCP connection tracing
tcpconnect   # Show new TCP connections
tcpaccept    # Show incoming connections

# DNS queries
dnsnoop

# Custom bpftrace script
bpftrace -e '
tracepoint:syscalls:sys_enter_write {
  @[comm] = count();
}
interval:s:5 {
  print(@);
  clear(@);
}
'

Quick Performance Checklist

# 60-second analysis checklist
uptime               # Load average trend
dmesg | tail         # Kernel messages
vmstat 1 3           # Virtual memory stats
mpstat -P ALL 1      # CPU breakdown
pidstat 1 3          # Process stats
iostat -xz 1 3       # Disk I/O
free -h              # Memory
sar -n DEV 1 3       # Network stats
sar -n TCP,ETCP 1    # TCP stats
top                  # System overview