高性能：BPF performance Tools 学习笔记-1

本系统列内容来自 性能优化大神Brendan Gregg 出的新书， 目前国内还没有引入。

为了抢先体验到大佬的又一部力作，我直接去oreilly官方网站的看的电子书。

Brendan Gregg 书里提到 Netflix和Facebook都将这些工具作为生产服务器的标配工具，因此我们也可以在经过充分的测试后，考虑将这些工具做到初始化镜像模板里面。

废话不说。直接放图，每个level都有对应的perf分析工具：

image.png

每个工具的作用如下：

•argdist:     Display function parameter values as a histogram or frequency count. 

•bashreadline:     Print entered bash commands system wide. 

•biolatency:     Summarize block device I/O latency as a histogram. 

•biotop:     Top for disks:     Summarize block device I/O by process. 

•biosnoop:     Trace block device I/O with PID and latency. 

•bitesize:     Show per process I/O size histogram. 

•bpflist:     Display processes with active BPF programs and maps. 

•btrfsdist:     Summarize btrfs operation latency distribution as a histogram. 

•btrfsslower:     Trace slow btrfs operations. 

•capable:     Trace security capability checks. 

•cachestat:     Trace page cache hit/miss ratio. 

•cachetop:     Trace page cache hit/miss ratio by processes. 

•compactsnoop:     Trace compact zone events with PID and latency. 

•cpudist:     Summarize on- and off-CPU time per task as a histogram. Examples

•cpuunclaimed:     Sample CPU run queues and calculate unclaimed idle CPU. Examples

•criticalstat:     Trace and report long atomic critical sections in the kernel. Examples

•dbslower:     Trace MySQL/PostgreSQL queries slower than a threshold. 

•dbstat:     Summarize MySQL/PostgreSQL query latency as a histogram. 

•dcsnoop:     Trace directory entry cache (dcache) lookups. 

•dcstat:     Directory entry cache (dcache) stats. 

•deadlock:     Detect potential deadlocks on a running process. 

•drsnoop:     Trace direct reclaim events with PID and latency. 

•execsnoop:     Trace new processes via exec() syscalls. 

•exitsnoop:     Trace process termination (exit and fatal signals). 

•ext4dist:     Summarize ext4 operation latency distribution as a histogram. 

•ext4slower:     Trace slow ext4 operations. 

•filelife:     Trace the lifespan of short-lived files. 

•fileslower:     Trace slow synchronous file reads and writes. 

•filetop:     File reads and writes by filename and process. Top for files. 

•funccount:     Count kernel function calls. 

•funclatency:     Time functions and show their latency distribution. 

•funcslower:     Trace slow kernel or user function calls. 

•gethostlatency:     Show latency for getaddrinfo/gethostbyname[2] calls. 

•hardirqs:     Measure hard IRQ (hard interrupt) event time. 

•inject:     Targeted error injection with call chain and predicates 

•killsnoop:     Trace signals issued by the kill() syscall. 

•klockstat:     Traces kernel mutex lock events and display locks statistics. 

•llcstat:     Summarize CPU cache references and misses by process. 

•mdflush:     Trace md flush events. 

•memleak:     Display outstanding memory allocations to find memory leaks. 

•mountsnoop:     Trace mount and umount syscalls system-wide. 

•mysqld_qslower:     Trace MySQL server queries slower than a threshold. 

•nfsslower:     Trace slow NFS operations. 

•nfsdist:     Summarize NFS operation latency distribution as a histogram. 

•offcputime:     Summarize off-CPU time by kernel stack trace. 

•offwaketime:     Summarize blocked time by kernel off-CPU stack and waker stack. 

•oomkill:     Trace the out-of-memory (OOM) killer. 

•opensnoop:     Trace open() syscalls. 

•pidpersec:     Count new processes (via fork). 

•profile:     Profile CPU usage by sampling stack traces at a timed interval. 

•reset-trace:     Reset the state of tracing. Maintenance tool only. 

•runqlat:     Run queue (scheduler) latency as a histogram. 

•runqlen:     Run queue length as a histogram. 

•runqslower:     Trace long process scheduling delays. 

•shmsnoop:     Trace System V shared memory syscalls. 

•sofdsnoop:     Trace FDs passed through unix sockets. 

•slabratetop:     Kernel SLAB/SLUB memory cache allocation rate top. 

•softirqs:     Measure soft IRQ (soft interrupt) event time. 

•solisten:     Trace TCP socket listen. 

•sslsniff:     Sniff OpenSSL written and readed data. 

•stackcount:     Count kernel function calls and their stack traces. 

•syncsnoop:     Trace sync() syscall. 

•syscount:     Summarize syscall counts and latencies. 

•tcpaccept:     Trace TCP passive connections (accept()). 

•tcpconnect:     Trace TCP active connections (connect()). 

•tcpconnlat:     Trace TCP active connection latency (connect()). 

•tcpdrop:     Trace kernel-based TCP packet drops with details. 

•tcplife:     Trace TCP sessions and summarize lifespan. 

•tcpretrans:     Trace TCP retransmits and TLPs. 

•tcpstates:     Trace TCP session state changes with durations. 

•tcpsubnet:     Summarize and aggregate TCP send by subnet. 

•tcptop:     Summarize TCP send/recv throughput by host. Top for TCP. 

•tcptracer:     Trace TCP established connections (connect(), accept(), close()). 

•tplist:     Display kernel tracepoints or USDT probes and their formats. 

•trace:     Trace arbitrary functions, with filters. 

•ttysnoop:     Watch live output from a tty or pts device. 

•ucalls:     Summarize method calls or Linux syscalls in high-level languages. 

•uflow:     Print a method flow graph in high-level languages. 

•ugc:     Trace garbage collection events in high-level languages. 

•uobjnew:     Summarize object allocation events by object type and number of bytes allocated. 

•ustat:     Collect events such as GCs, thread creations, object allocations, exceptions and more in high-level languages. 

•uthreads:     Trace thread creation events in Java and raw pthreads. 

•vfscount vfscount.c:     Count VFS calls. 

•vfsstat vfsstat.c:     Count some VFS calls, with column output. 

•wakeuptime:     Summarize sleep to wakeup time by waker kernel stack. 

•xfsdist:     Summarize XFS operation latency distribution as a histogram. 

•xfsslower:     Trace slow XFS operations. 

•zfsdist:     Summarize ZFS operation latency distribution as a histogram. 

•zfsslower:     Trace slow ZFS operations. 

上面提到的工具非常多，我们可以稍微精简些。如下：

不同level推荐的分析工具：