awk(1)

awk [opt] program [input]
-    -F <sepstr>        field separator string (can be regex)
-    program            awk program
-    input              file or stdin if not file given
-

Input processing

Input is processed in two stages:

Splitting input into a sequence of records. -By default split at newline character, but can be changed via the -builtin RS variable.
Splitting a record into fields. By default strings without whitespace, -but can be changed via the builtin variable FS or command line option --F.

Fields are accessed as follows:

$0 whole record
$1 field one
$2 field two
...

An example is the regex pattern /abc/ { print $1 } which prints the first -field if the record matches the regex /abc/. This form is actually a short -version for $0 ~ /abc/ { print $1 }, see the regex comparison operator -below.

Special pattern

awk provides two special patterns, BEGIN and END, which can be used -multiple times. Actions with those patterns are executed exactly once.

BEGIN actions are run before processing the first record
END actions are run after processing the last record

Special variables

RS record separator: first char is the record separator, by default -
FS field separator: regex to split records into fields, by default -
NR number record: number of current record
NF number fields: number of fields in the current record

Special statements & functions

-
printf "fmt", args...
-
Print format string, args are comma separated.
-
- %s string
- %d decimal
- %x hex
- %f float
-
Width can be specified as %Ns, this reserves N chars for a string. -For floats one can use %N.Mf, N is the total number including . and -M.
-
-
sprintf("fmt", expr, ...)
-
Format the expressions according to the format string. Similar as printf, -but this is a function and return value can be assigned to a variable.
-
-
strftime("fmt")
-
Print time stamp formatted by fmt.
-
- %Y full year (eg 2020)
- %m month (01-12)
- %d day (01-31)
- %F alias for %Y-%m-%d
- %H hour (00-23)
- %M minute (00-59)
- %S second (00-59)
- %T alias for %H:%M:%S
-
-
S ~ R, S !~ R
-
The regex comparison operator, where the former returns true if the string -S matches the regex R, and the latter is the negated form. -The regex can be either a -constant -or dynamic -regex.
-

Examples

Filter records

awk 'NR%2 == 0 { print $0 }' <file>
-

The pattern NR%2 == 0 matches every second record and the action { print $0 } -prints the whole record.

Negative patterns

awk '!/^#/ { print $1 }' <file>
-

Matches records not starting with #.

Range patterns

echo -e "a\nFOO\nb\nc\nBAR\nd" | \
-    awk '/FOO/,/BAR/ { print }'
-

/FOO/,/BAR/ define a range pattern of begin_pattern, end_pattern. When -begin_pattern is matched the range is turned on and when the -end_pattern is matched the range is turned off. This matches every record -in the range inclusive.

An exclusive range must be handled explicitly, for example as follows.

echo -e "a\nFOO\nb\nc\nBAR\nd" | \
-    awk '/FOO/,/BAR/ { if (!($1 ~ "FOO") && !($1 ~ "BAR")) { print } }'
-

Access last fields in records

echo 'a b c d e f' | awk '{ print $NF $(NF-1) }'
-

Access last fields with arithmetic on the NF number of fields variable.

Split on multiple tokens

echo 'a,b;c:d' | awk -F'[,;:]' '{ printf "1=%s | 4=%s\n", $1, $4 }'
-

Use regex as field separator.

Capture in variables

# /proc/<pid>/status
-#   Name:    cat
-#   ...
-#   VmRSS:   516 kB
-#   ...
-
-for f in /proc/*/status; do
-    cat $f | awk '
-             /^VmRSS/ { rss = $2/1024 }
-             /^Name/ { name = $2 }
-             END { printf "%16s %6d MB\n", name, rss }';
-done | sort -k2 -n
-

We capture values from VmRSS and Name into variables and print them at the -END once processing all records is done.

Capture in array

echo 'a 10
-b 2
-b 4
-a 1' | awk '{
-    vals[$1] += $2
-    cnts[$1] += 1
-}
-END {
-    for (v in vals)
-        printf "%s %d\n", v, vals[v] / cnts [v]
-}'
-

Capture keys and values from different columns and some up the values. -At the END we compute the average of each key.

Run shell command and capture output

cat /proc/1/status | awk '
-                     /^Pid/ {
-                        "ps --no-header -o user " $2 | getline user;
-                         print user
-                     }'
-

We build a ps command line and capture the first line of the processes output -in the user variable and then print it.

- -

bash(1)

Expansion

Generator

# generate sequence from n to m
-{n..m}
-# generate sequence from n to m step by s
-{n..m..s}
-
-# expand cartesian product
-{a,b}{c,d}
-

Parameter

# default value
-bar=${foo:-some_val}  # if $foo set, then bar=$foo else bar=some_val
-
-# alternate value
-bar=${foo:+bla $foo}  # if $foo set, then bar="bla $foo" else bar=""
-
-# check param set
-bar=${foo:?msg}  # if $foo set, then bar=$foo else exit and print msg
-
-# indirect
-FOO=foo
-BAR=FOO
-bar=${!BAR}  # deref value of BAR -> bar=$FOO
-
-# prefix
-${foo#prefix}  # remove prefix when expanding $foo
-# suffix
-${foo%suffix}  # remove suffix when expanding $foo
-
-# substitute
-${foo/pattern/string}  # replace pattern with string when expanding foo
-# pattern starts with
-# '/'   replace all occurences of pattern
-# '#'   pattern match at beginning
-# '%'   pattern match at end
-
-# set programmatically with priintf builtin
-printf -v "VAR1" "abc"
-NAME=VAR2
-printf -v "$NAME" "%s" "def"
-

-
Note: prefix/suffix/pattern are expanded as pathnames.
-

Pathname

*           match any string
-?           match any single char
-\\          match backslash
-[abc]       match any char of 'a' 'b' 'c'
-[a-z]       match any char between 'a' - 'z'
-[^ab]       negate, match all not 'a' 'b'
-[:class:]   match any char in class, available:
-              alnum,alpha,ascii,blank,cntrl,digit,graph,lower,
-              print,punct,space,upper,word,xdigit
-

With extglob shell option enabled it is possible to have more powerful -patterns. In the following pattern-list is one ore more patterns separated -by | char.

?(pattern-list)   matches zero or one occurrence of the given patterns
-*(pattern-list)   matches zero or more occurrences of the given patterns
-+(pattern-list)   matches one or more occurrences of the given patterns
-@(pattern-list)   matches one of the given patterns
-!(pattern-list)   matches anything except one of the given patterns
-

-
Note: shopt -s extglob/shopt -u extglob to enable/disable extglob -option.
-

I/O redirection

-
Note: The trick with bash I/O redirection is to interpret from left-to-right.
-

# stdout & stderr to file
-command >file 2>&1
-# equivalent
-command &>file
-
-# stderr to stdout & stdout to file
-command 2>&1 >file
-

-
The article Bash One-Liners Explained, Part III: All about -redirections -contains some nice visualization to explain bash redirections.
-

Explanation

j>&i
-

Duplicate fd i to fd j, making j a copy of i. See dup2(2).

Example:

command 2>&1 >file
-

duplicate fd 1 to fd 2, effectively redirecting stderr to stdout
redirect stdout to file

Process substitution (ref)

Process substitution allows to redirect the stdout of multiple processes at -once.

vim -d <(grep foo bar) <(grep foo moose)
-

Command grouping

Execute commands in a group with or without subshell. Can be used to easily -redirect stdout/stderr of all commands in the group into one file.

# Group commands without subshell.
-v=abc ; { v=foo; echo $v; } ; echo $v
-# foo
-# foo
-
-# Group commands with subshell.
-v=abc ; ( v=foo; echo $v; ) ; echo $v
-# foo
-# abc
-

Argument parsing with `getopts`

The getopts builtin uses following global variables:

OPTARG, value of last option argument
OPTIND, index of the next argument to process (user must reset)
OPTERR, display errors if set to 1

getopts <optstring> <param> [<args>]
-

<optstring> specifies the names of supported options, eg f:c -
- f: means -f option with an argument
- c means -c option without an argument
-
<param> specifies a variable name which getopts fills with the last parsed option argument
<args> optionally specify argument string to parse, by default getopts parses $@

Example

#!/bin/bash
-function parse_args() {
-    while getopts "f:c" PARAM; do
-        case $PARAM in
-            f) echo "GOT -f $OPTARG";;
-            c) echo "GOT -c";;
-            *) echo "ERR: print usage"; exit 1;;
-        esac
-    done
-    # users responsibility to reset OPTIND
-    OPTIND=1
-}
-
-parse_args -f xxx -c
-parse_args -f yyy
-

Regular Expressions

Bash supports regular expression matching with the binary operator =~. -The match results can be accessed via the $BASH_REMATCH variable:

${BASH_REMATCH[0]} contains the full match
${BASH_REMATCH[1]} contains match of the first capture group

INPUT='title foo : 1234'
-REGEX='^title (.+) : ([0-9]+)$'
-if [[ $INPUT =~ $REGEX ]]; then
-    echo "${BASH_REMATCH[0]}"    # title foo : 1234
-    echo "${BASH_REMATCH[1]}"    # foo
-    echo "${BASH_REMATCH[2]}"    # 1234
-fi
-

-
Caution: When specifying a regex in the [[ ]] block directly, quotes will be treated as part of the pattern. -[[ $INPUT =~ "foo" ]] will match against "foo" not foo!
-

Completion

The complete builtin is used to interact with the completion system.

complete                    # print currently installed completion handler
-complete -F <func> <cmd>    # install <func> as completion handler for <cmd>
-complete -r <cmd>           # uninstall completion handler for <cmd>
-

Variables available in completion functions:

# in
-$1              # <cmd>
-$2              # current word
-$3              # privous word
-
-COMP_WORDS      # array with current command line words
-COMP_CWORD      # index into COMP_WORDS with current cursor position
-
-# out
-COMPREPLY       # array with possible completions
-

The compgen builtin is used to generate possible matches by comparing word -against words generated by option.

compgen <option> <word>
-
-# usefule options:
-# -W <list>    specify list of possible completions
-# -d           generate list with dirs
-# -f           generate list with files
-# -u           generate list with users
-# -e           generate list with exported variables
-
-# compare "f" against words "foo" "foobar" "bar" and generate matches
-compgen -W "foo foobar bar" "f"
-
-# compare "hom" against file/dir names and generate matches
-compgen -d -f "hom"
-

Example

Skeleton to copy/paste for writing simple completions.

Assume a program foo with the following interface:

foo -c green|red|blue -s low|high -f <file> -h
-

The completion handler could be implemented as follows:

function _foo() {
-    local curr=$2
-    local prev=$3
-
-    local opts="-c -s -f -h"
-    case $prev in
-        -c) COMPREPLY=( $(compgen -W "green red blue" -- $curr) );;
-        -s) COMPREPLY=( $(compgen -W "low high" -- $curr) );;
-        -f) COMPREPLY=( $(compgen -f -- $curr) );;
-        *)  COMPREPLY=( $(compgen -W "$opts" -- $curr) );;
-    esac
-}
-
-complete -F _foo foo
-

- -

Notes

awk(1)

Input processing

Program

Special pattern

Special variables

Special statements & functions

Examples

Filter records

Negative patterns

Range patterns

Access last fields in records

Split on multiple tokens

Capture in variables

Capture in array

Run shell command and capture output

Notes

bash(1)

Expansion

Generator

Parameter

Pathname

I/O redirection

Explanation

Process substitution (ref)

Command grouping

Argument parsing with `getopts`

Example

Regular Expressions

Completion

Example

Notes

column(1)

Examples