... in conjunction with '-i' filter for protocol in state -U ......... show unix domain sockets ('@' indicates abstract sock name, see unix(7)) file flags: R/W/RW ..... read/write/read-write CR ......... create AP ......... append TR ......... truncate -s protocols TCP, UDP -s states (TCP) CLOSED, IDLE, BOUND, LISTEN, ESTABLISHED, SYN_SENT, SYN_RCDV, ESTABLISHED, CLOSE_WAIT, FIN_WAIT1, CLOSING, LAST_ACK, FIN_WAIT_2, TIME_WAIT -s states (UDP) Unbound, Idle","breadcrumbs":"Resource analysis & monitor » lsof » lsof(8)","id":"200","title":"lsof(8)"},"201":{"body":"","breadcrumbs":"Resource analysis & monitor » lsof » Examples","id":"201","title":"Examples"},"202":{"body":"Show open files with file flags for process: lsof +fg -p ","breadcrumbs":"Resource analysis & monitor » lsof » File flags","id":"202","title":"File flags"},"203":{"body":"Show open tcp connections for $USER: lsof -a -u $USER -i TCP Note : -a ands the results. If -a is not given all open files matching $USER and all tcp connections are listed ( ored ).","breadcrumbs":"Resource analysis & monitor » lsof » Open TCP connections","id":"203","title":"Open TCP connections"},"204":{"body":"Show open connections to localhost for $USER: lsof -a -u $USER -i @localhost","breadcrumbs":"Resource analysis & monitor » lsof » Open connection to specific host","id":"204","title":"Open connection to specific host"},"205":{"body":"Show open connections to port :1234 for $USER: lsof -a -u $USER -i :1234","breadcrumbs":"Resource analysis & monitor » lsof » Open connection to specific port","id":"205","title":"Open connection to specific port"},"206":{"body":"lsof -i 4TCP -s TCP:ESTABLISHED","breadcrumbs":"Resource analysis & monitor » lsof » IPv4 TCP connections in ESTABLISHED state","id":"206","title":"IPv4 TCP connections in ESTABLISHED state"},"207":{"body":"This may help to find which processes keep devices busy when trying to unmount and the device is currently busy. # Assuming /proc is a mount point.\nlsof /proc","breadcrumbs":"Resource analysis & monitor » lsof » List open files in a mounted directory.","id":"207","title":"List open files in a mounted directory."},"208":{"body":"ss [option] [filter] [option] -p ..... Show process using socket -l ..... Show sockets in listening state -4/-6 .. Show IPv4/6 sockets -x ..... Show unix sockets -n ..... Show numeric ports (no resolve) -O ..... Oneline output per socket [filter] dport/sport PORT .... Filter for destination/source port dst/src ADDR ........ Filter for destination/source address and/or .............. Logic operator ==/!= ............... Comparison operator (EXPR) .............. Group exprs","breadcrumbs":"Resource analysis & monitor » ss » ss(8)","id":"208","title":"ss(8)"},"209":{"body":"Show all tcp IPv4 sockets connecting to port 443: ss -4 'dport 443' Show all tcp IPv4 sockets that don't connect to port 443 or connect to address 1.2.3.4. ss -4 'dport != 443 or dst 1.2.3.4'","breadcrumbs":"Resource analysis & monitor » ss » Examples","id":"209","title":"Examples"},"21":{"body":"Note: The trick with bash I/O redirection is to interpret from left-to-right. # stdout & stderr to file\ncommand >file 2>&1\n# equivalent\ncommand &>file # stderr to stdout & stdout to file\ncommand 2>&1 >file The article Bash One-Liners Explained, Part III: All about redirections contains some nice visualization to explain bash redirections.","breadcrumbs":"Tools » bash » I/O redirection","id":"21","title":"I/O redirection"},"210":{"body":"pidstat [opt] [interval] [cont] -U [user] show username instead UID, optionally only show for user -r memory statistics -d I/O statistics -h single line per process and no lines with average","breadcrumbs":"Resource analysis & monitor » pidstat » pidstat(1)","id":"210","title":"pidstat(1)"},"211":{"body":"pidstat -r -p [interval] [count] minor_pagefault: Happens when the page needed is already in memory but not allocated to the faulting process, in that case the kernel only has to create a new page-table entry pointing to the shared physical page (not required to load a memory page from disk). major_pagefault: Happens when the page needed is NOT in memory, the kernel has to create a new page-table entry and populate the physical page (required to load a memory page from disk).","breadcrumbs":"Resource analysis & monitor » pidstat » Page fault and memory utilization","id":"211","title":"Page fault and memory utilization"},"212":{"body":"pidstat -d -p [interval] [count]","breadcrumbs":"Resource analysis & monitor » pidstat » I/O statistics","id":"212","title":"I/O statistics"},"213":{"body":"pgrep [opts] -n only list newest matching process -u only show matching for user -l additionally list command -a additionally list command + arguments -x match exactly","breadcrumbs":"Resource analysis & monitor » pgrep » pgrep(1)","id":"213","title":"pgrep(1)"},"214":{"body":"For example attach gdb to newest zsh process from $USER. gdb -p $(pgrep -n -u $USER zsh)","breadcrumbs":"Resource analysis & monitor » pgrep » Debug newest process","id":"214","title":"Debug newest process"},"215":{"body":"pmap [opts] Dump virtual memory map of process. Compared to /proc//maps it shows the size of the mappings.\nopts: -p show full path in the mapping -x show details (eg RSS usage of each segment)","breadcrumbs":"Resource analysis & monitor » pmap » pmap(1)","id":"215","title":"pmap(1)"},"216":{"body":"pstack Dump stack for all threads of process.","breadcrumbs":"Resource analysis & monitor » pstack » pstack(1)","id":"216","title":"pstack(1)"},"217":{"body":"time strace ltrace perf OProfile callgrind","breadcrumbs":"Trace and Profile » Trace and Profile","id":"217","title":"Trace and Profile"},"218":{"body":"# statistics of process run\n/usr/bin/time -v ","breadcrumbs":"Trace and Profile » time » /usr/bin/time(1)","id":"218","title":"/usr/bin/time(1)"},"219":{"body":"strace [opts] [prg] -f .......... follow child processes on fork(2) -ff ......... follow fork and separate output file per child -p .... attach to running process -s ... max string size, truncate of longer (default: 32) -e ... expression for trace filtering -o ... log output into -c .......... dump syscall statitics at the end -C .......... like -c but dump regular ouput as well -k .......... dump stack trace for each syscall -P ... only trace syscall accesing path -y .......... print paths for FDs -tt ......... print absolute timestamp (with us precision) -r .......... print relative timestamp -z .......... log only successful syscalls -Z .......... log only failed syscalls -n .......... print syscall numbers -y .......... translate fds (eg file path, socket) -yy ......... translate fds with all information (eg IP) -x .......... print non-ASCII chars as hex string : trace=syscall[,syscall] .... trace only syscall listed trace=file ................. trace all syscall that take a filename as arg trace=process .............. trace process management related syscalls trace=signal ............... trace signal related syscalls signal ..................... trace signals delivered to the process","breadcrumbs":"Trace and Profile » strace » strace(1)","id":"219","title":"strace(1)"},"22":{"body":"j>&i Duplicate fd i to fd j, making j a copy of i. See dup2(2) . Example: command 2>&1 >file duplicate fd 1 to fd 2, effectively redirecting stderr to stdout redirect stdout to file","breadcrumbs":"Tools » bash » Explanation","id":"22","title":"Explanation"},"220":{"body":"Trace open(2) & socket(2) syscalls for a running process + child processes: strace -f -e trace=open,socket -p Trace signals delivered to a running process: strace -e signal -e 'trace=!all' -p ","breadcrumbs":"Trace and Profile » strace » Examples","id":"220","title":"Examples"},"221":{"body":"ltrace [opts] [prg] -f .......... follow child processes on fork(2) -p .... attach to running process -o ... log output into -l . show who calls into lib matched by -C .......... demangle","breadcrumbs":"Trace and Profile » ltrace » ltrace(1)","id":"221","title":"ltrace(1)"},"222":{"body":"List which program/libs call into libstdc++: ltrace -l '*libstdc++*' -C -o ltrace.log ./main","breadcrumbs":"Trace and Profile » ltrace » Example","id":"222","title":"Example"},"223":{"body":"perf list show supported hw/sw events perf stat -p .. show stats for running process -I ... show stats periodically over interval -e ... filter for events perf top -p .. show stats for running process -F ... sampling frequency -K ........ hide kernel threads perf record -p ............... record stats for running process -F ................ sampling frequency --call-graph .. [fp, dwarf, lbr] method how to caputre backtrace fp : use frame-pointer, need to compile with -fno-omit-frame-pointer dwarf: use .cfi debug information lbr : use hardware last branch record facility -g ..................... short-hand for --call-graph fp -e ................ filter for events perf report -n .................... annotate symbols with nr of samples --stdio ............... report to stdio, if not presen tui mode -g graph,0.5,caller ... show caller based call chains with value >0.5 Useful : page-faults minor-faults major-faults cpu-cycles` task-clock","breadcrumbs":"Trace and Profile » perf » perf(1)","id":"223","title":"perf(1)"},"224":{"body":"","breadcrumbs":"Trace and Profile » perf » Flamegraph","id":"224","title":"Flamegraph"},"225":{"body":"perf record -g -e cpu-cycles -p \nperf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > cycles-flamegraph.svg","breadcrumbs":"Trace and Profile » perf » Flamegraph with single event trace","id":"225","title":"Flamegraph with single event trace"},"226":{"body":"perf record -g -e cpu-cycles,page-faults -p \nperf script --per-event-dump\n# fold & generate as above","breadcrumbs":"Trace and Profile » perf » Flamegraph with multiple event traces","id":"226","title":"Flamegraph with multiple event traces"},"227":{"body":"operf -g -p -g ...... caputre call-graph information opreport [opt] FILE show time spent per binary image -l ...... show time spent per symbol -c ...... show callgraph information (see below) -a ...... add column with time spent accumulated over child nodes ophelp show supported hw/sw events","breadcrumbs":"Trace and Profile » OProfile » OProfile","id":"227","title":"OProfile"},"228":{"body":"Callgrind is a tracing profiler to record the function call history of a target program. It is part of the valgrind tool suite. Profiling data is collected by instrumentation rather than sampling of the target program. Callgrind does not capture the actual time spent in a function but computes the cost of a function based on the instructions fetched (Ir = Instruction read). Therefore effects like slow IO are not reflected, which should be kept in mind when analyzing callgrind results. By default the profiler data is dumped when the target process is terminating, but callgrind_control allows for interactive control of callgrind. # Run a program under callgrind.\nvalgrind --tool=callgrind -- # Interactive control of callgrind.\ncallgrind_control [opts] opts: -b ............. show current backtrace -e ............. show current event counters -s ............. show current stats --dump[=file] .. dump current collection -i=on|off ...... turn instrumentation on|off Results can be analyzed by using one of the following tools callgrind_annotate (cli) kcachegrind (ui) The following is a collection of frequently used callgrind options. valgrind --tool=callgrind [opts] -- opts: --callgrind-out-file= .... output file, rather than callgrind.out. --dump-instr= .......... annotation on instrucion level, allows for asm annotations --instr-atstart= ....... control if instrumentation is enabled from beginning of the program --separate-threads= .... create separate output files per thread, appends - to the output file","breadcrumbs":"Trace and Profile » callgrind » callgrind","id":"228","title":"callgrind"},"229":{"body":"Programmatically enable/disable instrumentation using the macros defined in the callgrind header. #include