strace, lsof, and ps: Linux Process and System Debugging

Essential tools for understanding what's happening inside your Linux system.

ps - Process Snapshot

# All processes with details
ps aux

# Process tree
ps auxf

# Find specific process
ps aux | grep nginx

# Processes by user
ps -u www-data

# Custom format
ps -eo pid,ppid,user,%cpu,%mem,cmd --sort=-%mem | head -20

# Watch processes live
watch -n1 'ps aux --sort=-%cpu | head -10'

top / htop

# Interactive CPU/memory monitor
top

# Better version
htop

# Sort by memory in top
top -o %MEM

# Key bindings in htop
# F2: Setup, F3: Search, F4: Filter
# F5: Tree view, F6: Sort, F9: Kill

lsof - List Open Files

# All open files
lsof

# Files opened by process
lsof -p 1234

# Files opened by user
lsof -u www-data

# Network connections
lsof -i

# TCP connections
lsof -i TCP

# Specific port
lsof -i :8080

# Files in directory
lsof +D /var/log/nginx

# Find process using a file
lsof /path/to/file

netstat / ss

# All listening ports
ss -tlnp
netstat -tlnp

# All connections
ss -anp

# Check specific port
ss -tlnp | grep ':80'

# Show routing table
netstat -r
ip route show

strace - System Call Tracer

# Trace process system calls
strace ls /tmp

# Trace running process
strace -p 1234

# Show only file-related calls
strace -e trace=file ls /tmp

# Show only network calls
strace -e trace=network curl http://example.com

# Count and summarize calls
strace -c ls /tmp

# Follow forks
strace -f nginx

# Save to file
strace -o /tmp/trace.log myapp

Practical Debugging Scenarios

# Find what's using port 3000
lsof -i :3000
# or
fuser 3000/tcp

# Why is disk full?
lsof | grep deleted | awk '{print $7, $9}' | sort -rn | head

# Find slow system calls
strace -T -e trace=file myapp 2>&1 | awk '{print $NF, $0}' | sort -rn | head

# Memory leak investigation
cat /proc/1234/status | grep -E "VmRSS|VmSize"

# File descriptor limits
cat /proc/1234/limits | grep "open files"
ulimit -n

These tools give you deep visibility into system behavior for debugging production issues.