10 files changed, 182 insertions, 0 deletions
diff --git a/src/SUMMARY.md b/src/SUMMARY.md
index f31e217..d0a4e62 100644
--- a/src/SUMMARY.md
+++ b/src/SUMMARY.md
@@ -1,3 +1,14 @@
 # Summary
 
 - [gdb](./gdb.md)
+- [debug & trace](./debug_trace.md)
+    - [strace](./strace.md)
+    - [lsof](./lsof.md)
+    - [pidstat](./pidstat.md)
+    - [time](./time.md)
+    - [pmap](./pmap.md)
+    - [pstack](./pstack.md)
+    - [perf](./perf.md)
+    - [OProfile](./oprofile.md)
+
+
diff --git a/src/debug_trace.md b/src/debug_trace.md
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/src/debug_trace.md
diff --git a/src/lsof.md b/src/lsof.md
new file mode 100644
index 0000000..a7ed2d1
--- /dev/null
+++ b/src/lsof.md
@@ -0,0 +1,36 @@
+# lsof(8)
+
+```markdown
+lsof
+  -a ......... AND slection filters instead ORing (OR: default)
+  -p <pid> ... list open file descriptors for process
+  +fg ........ show file flags for file descripros
+  -n ......... don't convert network addr to hostnames
+  -P ......... don't convert network port to know service names
+  -i <@h[:p]>. show connections to h (hostname|ip addr) with optional port p
+```
+
+```markdown
+file flags:
+  R/W/RW ..... read/write/read-write
+  CR ......... create
+  AP ......... append
+  TR ......... truncate
+```
+
+# Examples
+
+Show open files with file flags:
+```markdown
+lsof +fg -p <pid>
+```
+
+Show open tcp connections from user:
+```markdown
+lsof -a -u $USER -i tcp
+```
+
+Show open connections to 'localhost' for user:
+```markdown
+lsof -a -u $USER -i @localhost
+```
diff --git a/src/oprofile.md b/src/oprofile.md
new file mode 100644
index 0000000..15e06f6
--- /dev/null
+++ b/src/oprofile.md
@@ -0,0 +1,15 @@
+# OProfile
+
+```markdown
+operf -g -p <pid>
+  -g ...... caputre call-graph information
+
+opreport [opt] FILE
+     ...... show time spent per binary image
+  -l ...... show time spent per symbol
+  -c ...... show callgraph information (see below)
+  -a ...... add column with time spent accumulated over child nodes
+
+ophelp
+     ...... show supported hw/sw events
+```
diff --git a/src/perf.md b/src/perf.md
new file mode 100644
index 0000000..003e164
--- /dev/null
+++ b/src/perf.md
@@ -0,0 +1,56 @@
+# perf(1)
+
+```markdown
+perf list
+  ......... show supported hw/sw events
+
+perf stat
+  -p <pid> .. show stats for running process
+  -I <ms> ... show stats periodically over interval <ms>
+  -e <ev> ... filter for events
+
+perf top
+  -p <pid> .. show stats for running process
+  -F <hz> ... sampling frequency
+  -K ........ hide kernel threads
+
+perf record
+  -p <pid> ............... record stats for running process
+  -F <hz> ................ sampling frequency
+  --call-graph <method> .. [fp, dwarf, lbr] method how to caputre backtrace
+                           fp   : use frame-pointer, need -fno-omit-frame-pointer
+                           dwarf: use .cfi debug information
+                           lbr  : use hardware last branch record facility
+  -g ..................... short-hand for --call-graph fp
+  -e <ev> ................ filter for events
+
+perf report
+  -n .................... annotate symbols with nr of samples
+  --stdio ............... report to stdio, if not presen tui mode
+  -g graph,0.5,caller ... show caller based call chains with value >0.5
+```
+
+## Useful `perf` events
+
+```markdown
+useful <ev>:
+  page-faults
+  minor-faults
+  major-faults
+  cpu-cycles`
+  task-clock
+ ```
+
+## [`Flamegraph`](https://github.com/brendangregg/FlameGraph)
+
+```markdown
+# flamegraph for single event trace
+perf record -g -p <pid> -e cpu-cycles
+perf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > cycles-flamegraph.svg
+
+# flamegraphs for multiple events trace
+perf record -g -p <pid> -e cpu-cycles,page-faults
+perf script --per-event-dump
+# fold & generate as above
+```
+
diff --git a/src/pidstat.md b/src/pidstat.md
new file mode 100644
index 0000000..c17a2e6
--- /dev/null
+++ b/src/pidstat.md
@@ -0,0 +1,13 @@
+# pidstat(1)
+
+Trace minor/major page faults.
+```markdown
+pidstat -r -p <pid> [interval]
+  minor_pagefault: happens when the page needed is already in memory but not
+                   allocated to the faulting process, in that case the kernel
+                   only has to create a new page-table entry pointing to the
+                   shared physical page
+  major_pagefault: happends when the page needed is NOT in memory, the kernel
+                   has to create a new page-table entry and populate the
+                   physical page
+```
diff --git a/src/pmap.md b/src/pmap.md
new file mode 100644
index 0000000..d2e0e79
--- /dev/null
+++ b/src/pmap.md
@@ -0,0 +1,7 @@
+# pmap(1)
+
+```markdown
+pmap <pid>
+  ............. dump virtual memory map of process.
+                compared to /proc/<pid>/maps it shows the size of the mappings
+```
diff --git a/src/pstack.md b/src/pstack.md
new file mode 100644
index 0000000..06cbade
--- /dev/null
+++ b/src/pstack.md
@@ -0,0 +1,6 @@
+# pstack(1)
+
+```markdown
+pstack <pid>
+  ............. dump current stack of process + threads
+```
diff --git a/src/strace.md b/src/strace.md
new file mode 100644
index 0000000..ff0f9d3
--- /dev/null
+++ b/src/strace.md
@@ -0,0 +1,32 @@
+# strace(1)
+
+```markdown
+strace [opts] [prg]
+  -f .......... follow child processes on fork(2)
+  -p <pid> .... attach to running process
+  -s <size> ... max string size (default: 32)
+  -e <expr> ... expression for trace filtering
+  -o <file> ... log output into <file>
+  -c .......... dump syscall statitics at the end
+```
+
+```markdown
+<expr>:
+  trace=syscall[,syscall] .... trace only syscall listed
+  trace=file ................. trace all syscall that take a filename as arg
+  trace=process .............. trace process management related syscalls
+  trace=signal ............... trace signal related syscalls
+  signal ..................... trace signals delivered to the process
+```
+
+# Examples
+
+Trace `'open & socket` syscalls for a running process + childs.
+```markdown
+strace -f -p <pid> -e trace=open,socket
+```
+
+Trace signals delivered to a running process.
+```markdown
+strace -f -p <pid> -e signal
+```
diff --git a/src/time.md b/src/time.md
new file mode 100644
index 0000000..be404fb
--- /dev/null
+++ b/src/time.md
@@ -0,0 +1,6 @@
+# /usr/bin/time(1)
+
+```markdown
+# statistics of process run
+/usr/bin/time -v <cmd>
+```