... in conjunction with '-i' filter for protocol in state -U ......... show unix domain sockets ('@' indicates abstract sock name, see unix(7)) file flags: R/W/RW ..... read/write/read-write CR ......... create AP ......... append TR ......... truncate -s protocols TCP, UDP -s states (TCP) CLOSED, IDLE, BOUND, LISTEN, ESTABLISHED, SYN_SENT, SYN_RCDV, ESTABLISHED, CLOSE_WAIT, FIN_WAIT1, CLOSING, LAST_ACK, FIN_WAIT_2, TIME_WAIT -s states (UDP) Unbound, Idle","breadcrumbs":"Resource analysis & monitor » lsof » lsof(8)","id":"206","title":"lsof(8)"},"207":{"body":"","breadcrumbs":"Resource analysis & monitor » lsof » Examples","id":"207","title":"Examples"},"208":{"body":"Show open files with file flags for process: lsof +fg -p ","breadcrumbs":"Resource analysis & monitor » lsof » File flags","id":"208","title":"File flags"},"209":{"body":"Show open tcp connections for $USER: lsof -a -u $USER -i TCP Note : -a ands the results. If -a is not given all open files matching $USER and all tcp connections are listed ( ored ).","breadcrumbs":"Resource analysis & monitor » lsof » Open TCP connections","id":"209","title":"Open TCP connections"},"21":{"body":"# default value\nbar=${foo:-some_val} # if $foo set, then bar=$foo else bar=some_val # alternate value\nbar=${foo:+bla $foo} # if $foo set, then bar=\"bla $foo\" else bar=\"\" # check param set\nbar=${foo:?msg} # if $foo set, then bar=$foo else exit and print msg # indirect\nFOO=foo\nBAR=FOO\nbar=${!BAR} # deref value of BAR -> bar=$FOO # prefix\n${foo#prefix} # remove prefix when expanding $foo\n# suffix\n${foo%suffix} # remove suffix when expanding $foo # substitute\n${foo/pattern/string} # replace pattern with string when expanding foo\n# pattern starts with\n# '/' replace all occurences of pattern\n# '#' pattern match at beginning\n# '%' pattern match at end Note: prefix/suffix/pattern are expanded as pathnames .","breadcrumbs":"Tools » bash » Parameter","id":"21","title":"Parameter"},"210":{"body":"Show open connections to localhost for $USER: lsof -a -u $USER -i @localhost","breadcrumbs":"Resource analysis & monitor » lsof » Open connection to specific host","id":"210","title":"Open connection to specific host"},"211":{"body":"Show open connections to port :1234 for $USER: lsof -a -u $USER -i :1234","breadcrumbs":"Resource analysis & monitor » lsof » Open connection to specific port","id":"211","title":"Open connection to specific port"},"212":{"body":"lsof -i 4TCP -s TCP:ESTABLISHED","breadcrumbs":"Resource analysis & monitor » lsof » IPv4 TCP connections in ESTABLISHED state","id":"212","title":"IPv4 TCP connections in ESTABLISHED state"},"213":{"body":"This may help to find which processes keep devices busy when trying to unmount and the device is currently busy. # Assuming /proc is a mount point.\nlsof /proc","breadcrumbs":"Resource analysis & monitor » lsof » List open files in a mounted directory.","id":"213","title":"List open files in a mounted directory."},"214":{"body":"ss [option] [filter] [option] -p ..... Show process using socket -l ..... Show sockets in listening state -4/-6 .. Show IPv4/6 sockets -x ..... Show unix sockets -n ..... Show numeric ports (no resolve) -O ..... Oneline output per socket [filter] dport/sport PORT .... Filter for destination/source port dst/src ADDR ........ Filter for destination/source address and/or .............. Logic operator ==/!= ............... Comparison operator (EXPR) .............. Group exprs","breadcrumbs":"Resource analysis & monitor » ss » ss(8)","id":"214","title":"ss(8)"},"215":{"body":"Show all tcp IPv4 sockets connecting to port 443: ss -4 'dport 443' Show all tcp IPv4 sockets that don't connect to port 443 or connect to address 1.2.3.4. ss -4 'dport != 443 or dst 1.2.3.4'","breadcrumbs":"Resource analysis & monitor » ss » Examples","id":"215","title":"Examples"},"216":{"body":"pidstat [opt] [interval] [cont] -U [user] show username instead UID, optionally only show for user -r memory statistics -d I/O statistics -h single line per process and no lines with average","breadcrumbs":"Resource analysis & monitor » pidstat » pidstat(1)","id":"216","title":"pidstat(1)"},"217":{"body":"pidstat -r -p [interval] [count] minor_pagefault: Happens when the page needed is already in memory but not allocated to the faulting process, in that case the kernel only has to create a new page-table entry pointing to the shared physical page (not required to load a memory page from disk). major_pagefault: Happens when the page needed is NOT in memory, the kernel has to create a new page-table entry and populate the physical page (required to load a memory page from disk).","breadcrumbs":"Resource analysis & monitor » pidstat » Page fault and memory utilization","id":"217","title":"Page fault and memory utilization"},"218":{"body":"pidstat -d -p [interval] [count]","breadcrumbs":"Resource analysis & monitor » pidstat » I/O statistics","id":"218","title":"I/O statistics"},"219":{"body":"pgrep [opts] -n only list newest matching process -u only show matching for user -l additionally list command -a additionally list command + arguments -x match exactly","breadcrumbs":"Resource analysis & monitor » pgrep » pgrep(1)","id":"219","title":"pgrep(1)"},"22":{"body":"* match any string\n? match any single char\n\\\\ match backslash\n[abc] match any char of 'a' 'b' 'c'\n[a-z] match any char between 'a' - 'z'\n[^ab] negate, match all not 'a' 'b'\n[:class:] match any char in class, available: alnum,alpha,ascii,blank,cntrl,digit,graph,lower, print,punct,space,upper,word,xdigit With extglob shell option enabled it is possible to have more powerful patterns. In the following pattern-list is one ore more patterns separated by | char. ?(pattern-list) matches zero or one occurrence of the given patterns\n*(pattern-list) matches zero or more occurrences of the given patterns\n+(pattern-list) matches one or more occurrences of the given patterns\n@(pattern-list) matches one of the given patterns\n!(pattern-list) matches anything except one of the given patterns Note: shopt -s extglob/shopt -u extglob to enable/disable extglob option.","breadcrumbs":"Tools » bash » Pathname","id":"22","title":"Pathname"},"220":{"body":"For example attach gdb to newest zsh process from $USER. gdb -p $(pgrep -n -u $USER zsh)","breadcrumbs":"Resource analysis & monitor » pgrep » Debug newest process","id":"220","title":"Debug newest process"},"221":{"body":"ps [opt] opt: --no-header .... do not print column header -o ....... comma separated list of output columns -p ....... only show pid -T ............. list threads Set PS_FORMAT env variable to setup default output columns. Frequently used output columns pid process id\npgid process group id\ntid thread id comm name of process\ncmd name of process + args (full) etime elapsed time (since process started)\nuser user owning process pcpu cpu utilization (percent)\npmem physical resident set (rss) (percent)\nrss physical memory (in kb)\nvsz virtual memory (in kb)","breadcrumbs":"Resource analysis & monitor » ps » ps(1)","id":"221","title":"ps(1)"},"222":{"body":"for tid in $(ps -o tid --no-header -T -p 31084); do taskset -c -p $tid;\ndone","breadcrumbs":"Resource analysis & monitor » ps » Example: Use output for scripting","id":"222","title":"Example: Use output for scripting"},"223":{"body":"pmap [opts] Dump virtual memory map of process. Compared to /proc//maps it shows the size of the mappings.\nopts: -p show full path in the mapping -x show details (eg RSS usage of each segment)","breadcrumbs":"Resource analysis & monitor » pmap » pmap(1)","id":"223","title":"pmap(1)"},"224":{"body":"pstack Dump stack for all threads of process.","breadcrumbs":"Resource analysis & monitor » pstack » pstack(1)","id":"224","title":"pstack(1)"},"225":{"body":"time strace ltrace perf OProfile callgrind","breadcrumbs":"Trace and Profile » Trace and Profile","id":"225","title":"Trace and Profile"},"226":{"body":"# statistics of process run\n/usr/bin/time -v ","breadcrumbs":"Trace and Profile » time » /usr/bin/time(1)","id":"226","title":"/usr/bin/time(1)"},"227":{"body":"strace [opts] [prg] -f .......... follow child processes on fork(2) -ff ......... follow fork and separate output file per child -p .... attach to running process -s ... max string size, truncate of longer (default: 32) -e ... expression for trace filtering -o ... log output into -c .......... dump syscall statitics at the end -C .......... like -c but dump regular ouput as well -k .......... dump stack trace for each syscall -P ... only trace syscall accesing path -y .......... print paths for FDs -tt ......... print absolute timestamp (with us precision) -r .......... print relative timestamp -z .......... log only successful syscalls -Z .......... log only failed syscalls -n .......... print syscall numbers -y .......... translate fds (eg file path, socket) -yy ......... translate fds with all information (eg IP) -x .......... print non-ASCII chars as hex string : trace=syscall[,syscall] .... trace only syscall listed trace=file ................. trace all syscall that take a filename as arg trace=process .............. trace process management related syscalls trace=signal ............... trace signal related syscalls signal ..................... trace signals delivered to the process","breadcrumbs":"Trace and Profile » strace » strace(1)","id":"227","title":"strace(1)"},"228":{"body":"Trace open(2) & socket(2) syscalls for a running process + child processes: strace -f -e trace=open,socket -p Trace signals delivered to a running process: strace -e signal -e 'trace=!all' -p ","breadcrumbs":"Trace and Profile » strace » Examples","id":"228","title":"Examples"},"229":{"body":"ltrace [opts] [prg] -f .......... follow child processes on fork(2) -p .... attach to running process -o ... log output into -l . show who calls into lib matched by -C .......... demangle","breadcrumbs":"Trace and Profile » ltrace » ltrace(1)","id":"229","title":"ltrace(1)"},"23":{"body":"Note: The trick with bash I/O redirection is to interpret from left-to-right. # stdout & stderr to file\ncommand >file 2>&1\n# equivalent\ncommand &>file # stderr to stdout & stdout to file\ncommand 2>&1 >file The article Bash One-Liners Explained, Part III: All about redirections contains some nice visualization to explain bash redirections.","breadcrumbs":"Tools » bash » I/O redirection","id":"23","title":"I/O redirection"},"230":{"body":"List which program/libs call into libstdc++: ltrace -l '*libstdc++*' -C -o ltrace.log ./main","breadcrumbs":"Trace and Profile » ltrace » Example","id":"230","title":"Example"},"231":{"body":"perf list show supported hw/sw events & metrics -v ........ print longer event descriptions --details . print information on the perf event names and expressions used internally by events perf stat -p ..... show stats for running process -I ...... show stats periodically over interval -e ...... select event(s) -M ..... print metric(s), this adds the metric events --all-user ... configure all selected events for user space --all-kernel . configure all selected events for kernel space perf top -p .. show stats for running process -F ... sampling frequency -K ........ hide kernel threads perf record -p ............... record stats for running process -F ................ sampling frequency --call-graph .. [fp, dwarf, lbr] method how to caputre backtrace fp : use frame-pointer, need to compile with -fno-omit-frame-pointer dwarf: use .cfi debug information lbr : use hardware last branch record facility -g ..................... short-hand for --call-graph fp -e ................ select event(s) --all-user ... configure all selected events for user space --all-kernel . configure all selected events for kernel space perf report -n .................... annotate symbols with nr of samples --stdio ............... report to stdio, if not presen tui mode -g graph,0.5,callee ... show callee based call chains with value >0.5 Useful : page-faults minor-faults major-faults cpu-cycles` task-clock","breadcrumbs":"Trace and Profile » perf » perf(1)","id":"231","title":"perf(1)"},"232":{"body":"Events to sample are specified with the -e option, either pass a comma separated list or pass -e multiple times. Events are specified in the following form name[:modifier]. The list and description of the modifier can be found in the perf-list(1) manpage under EVENT MODIFIERS. # L1 i$ misses in user space\n# L2 i$ stats in user/kernel space mixed\n# Sample specified events.\nperf stat -e L1-icache-load-misses:u \\ -e l2_rqsts.all_code_rd:uk,l2_rqsts.code_rd_hit:k,l2_rqsts.code_rd_miss:k \\ -- stress -c 2 The --all-user and --all-kernel options append a :u and :k modifier to all specified events. Therefore the following two command lines are equivalent. # 1)\nperf stat -e cycles:u,instructions:u -- ls # 2)\nperf stat --all-user -e cycles,instructions -- ls","breadcrumbs":"Trace and Profile » perf » Select specific events","id":"232","title":"Select specific events"},"233":{"body":"In case perf does not provide a symbolic name for an event, the event can be specified in a raw form as r + UMask + EventCode. The following is an example for the L2_RQSTS.CODE_RD_HIT event with EventCode=0x24 and UMask=0x10 on my laptop with a sandybridge uarch. perf stat -e l2_rqsts.code_rd_hit -e r1024 -- ls\n# Performance counter stats for 'ls':\n#\n# 33.942 l2_rqsts.code_rd_hit\n# 33.942 r1024","breadcrumbs":"Trace and Profile » perf » Raw events","id":"233","title":"Raw events"},"234":{"body":"The intel/perfmon repository provides a performance event databases for the different intel uarchs. The table in mapfile.csv can be used to lookup the corresponding uarch, just grab the family model from the procfs. cat /proc/cpuinfo | awk '/^vendor_id/ { V=$3 } /^cpu family/ { F=$4 } /^model\\s*:/ { printf \"%s-%d-%x\\n\",V,F,$3 }' The table in performance monitoring events describes how events are sorted into the different files.","breadcrumbs":"Trace and Profile » perf » Find raw performance counter events (intel)","id":"234","title":"Find raw performance counter events (intel)"},"235":{"body":"Perf also defines some own symbolic names for events. An example is the cache-references event. The perf_event_open(2) manpage gives the following description. perf_event_open(2) PERF_COUNT_HW_CACHE_REFERENCES Cache accesses. Usually this indicates Last Level Cache accesses but this may vary depending on your CPU. This may include prefetches and coherency messages; again this depends on the design of your CPU. The sysfs can be consulted to get the concrete performance counter on the given system. cat /sys/devices/cpu/events/cache-misses\n# event=0x2e,umask=0x41","breadcrumbs":"Trace and Profile » perf » Raw events for perfs own symbolic names","id":"235","title":"Raw events for perfs own symbolic names"},"236":{"body":"","breadcrumbs":"Trace and Profile » perf » Flamegraph","id":"236","title":"Flamegraph"},"237":{"body":"perf record -g -e cpu-cycles -p \nperf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > cycles-flamegraph.svg","breadcrumbs":"Trace and Profile » perf » Flamegraph with single event trace","id":"237","title":"Flamegraph with single event trace"},"238":{"body":"perf record -g -e cpu-cycles,page-faults -p \nperf script --per-event-dump\n# fold & generate as above","breadcrumbs":"Trace and Profile » perf » Flamegraph with multiple event traces","id":"238","title":"Flamegraph with multiple event traces"},"239":{"body":"","breadcrumbs":"Trace and Profile » perf » Examples","id":"239","title":"Examples"},"24":{"body":"j>&i Duplicate fd i to fd j, making j a copy of i. See dup2(2) . Example: command 2>&1 >file duplicate fd 1 to fd 2, effectively redirecting stderr to stdout redirect stdout to file","breadcrumbs":"Tools » bash » Explanation","id":"24","title":"Explanation"},"240":{"body":"#define NOP4 \"nop\\nnop\\nnop\\nnop\\n\"\n#define NOP32 NOP4 NOP4 NOP4 NOP4 NOP4 NOP4 NOP4 NOP4\n#define NOP256 NOP32 NOP32 NOP32 NOP32 NOP32 NOP32 NOP32 NOP32\n#define NOP2048 NOP256 NOP256 NOP256 NOP256 NOP256 NOP256 NOP256 NOP256 int main() { for (unsigned i = 0; i < 2000000; ++i) { asm volatile(NOP2048); }\n} perf stat -e cycles,instructions ./noploop\n# Performance counter stats for './noploop':\n#\n# 1.031.075.940 cycles\n# 4.103.534.341 instructions # 3,98 insn per cycle","breadcrumbs":"Trace and Profile » perf » Determine theoretical max instructions per cycle","id":"240","title":"Determine theoretical max instructions per cycle"},"241":{"body":"The following gives an example for a scenario where we have the following calls main -> do_foo() -> do_work() main -> do_bar() -> do_work() perf report --stdio -g graph,callee # Children Self Command Shared Object Symbols\n# ........ ........ ....... .................... .................\n#\n# 49.71% 49.66% bench bench [.] do_work\n# |\n# --49.66%--_start <- callstack bottom\n# __libc_start_main\n# 0x7ff366c62ccf\n# main\n# |\n# |--25.13%--do_bar\n# | do_work <- callstack top\n# |\n# --24.53%--do_foo\n# do_work perf report --stdio -g graph,callee # Children Self Command Shared Object Symbols\n# ........ ........ ....... .................... .................\n#\n# 49.71% 49.66% bench bench [.] do_work\n# |\n# ---do_work <- callstack top\n# |\n# |--25.15%--do_bar\n# | main\n# | 0x7ff366c62ccf\n# | __libc_start_main\n# | _start <- callstack bottom\n# |\n# --24.55%--do_foo\n# main\n# 0x7ff366c62ccf\n# __libc_start_main\n# _start <- callstack bottom","breadcrumbs":"Trace and Profile » perf » Caller vs callee callstacks","id":"241","title":"Caller vs callee callstacks"},"242":{"body":"intel/perfmon - intel PMU event database per uarch intel/perfmon-html - a html rendered version of the PMU events with search intel/perfmon/mapfile.csv - processor family to uarch mapping linux/perf/events - x86 PMU events known to perf tools linux/arch/events - x86 PMU events linux kernel wikichip - computer architecture wiki perf-list(1) - manpage perf_event_open(2) - manpage","breadcrumbs":"Trace and Profile » perf » References","id":"242","title":"References"},"243":{"body":"operf -g -p -g ...... caputre call-graph information opreport [opt] FILE show time spent per binary image -l ...... show time spent per symbol -c ...... show callgraph information (see below) -a ...... add column with time spent accumulated over child nodes ophelp show supported hw/sw events","breadcrumbs":"Trace and Profile » OProfile » OProfile","id":"243","title":"OProfile"},"244":{"body":"Callgrind is a tracing profiler to record the function call history of a target program. It is part of the valgrind tool suite. Profiling data is collected by instrumentation rather than sampling of the target program. Callgrind does not capture the actual time spent in a function but computes the cost of a function based on the instructions fetched (Ir = Instruction read). Therefore effects like slow IO are not reflected, which should be kept in mind when analyzing callgrind results. By default the profiler data is dumped when the target process is terminating, but callgrind_control allows for interactive control of callgrind. # Run a program under callgrind.\nvalgrind --tool=callgrind -- # Interactive control of callgrind.\ncallgrind_control [opts] opts: -b ............. show current backtrace -e ............. show current event counters -s ............. show current stats --dump[=file] .. dump current collection -i=on|off ...... turn instrumentation on|off Results can be analyzed by using one of the following tools callgrind_annotate (cli) kcachegrind (ui) The following is a collection of frequently used callgrind options. valgrind --tool=callgrind [opts] -- opts: --callgrind-out-file=