Bash Pipeline Cost

Bash pipeline cost is the extra work of starting one process for every stage in a pipeline. Each stage forks a new process, so a long chain pays that startup cost again and again.

Three stages mean three processes

Each stage function logs one hit to a file. The final count shows three stages ran, not one.

: > calls.txt
stage() {
  echo hit >> calls.txt
  command cat
}
printf 'a\nb\n' | stage | stage | stage > /dev/null
echo "stages run: $(wc -l < calls.txt)"

Fewer stages, fewer process startups

The same data runs through three counted stages, then through one. The counts show the shorter pipeline starts fewer processes.

: > calls.txt
counted() {
  echo hit >> calls.txt
  command cat
}
printf 'x\n' | counted | counted | counted > /dev/null
echo "three stages: $(wc -l < calls.txt)"
: > calls.txt
printf 'x\n' | counted > /dev/null
echo "one stage: $(wc -l < calls.txt)"

An external command per line multiplies the cost

The loop calls an external counter once for each of two lines. A single call over the whole input does the same job once.

: > calls.txt
counted() {
  echo hit >> calls.txt
  command wc -l
}
printf 'aa\nbbbb\n' > words.txt
while IFS= read -r line; do printf '%s\n' "$line" | counted > /dev/null; done < words.txt
echo "per-line calls: $(wc -l < calls.txt)"
: > calls.txt
printf 'aa\nbbbb\n' | counted > /dev/null
echo "whole input calls: $(wc -l < calls.txt)"

Example

: > calls.txt
counted_grep() {
  echo hit >> calls.txt
  command grep "$@"
}
printf 'ok\nerror one\nok\nerror two\nok\n' > log.txt
slow=0
while IFS= read -r line; do printf '%s\n' "$line" | counted_grep -q error && slow=$((slow + 1)); done < log.txt
echo "slow shape errors: $slow"
echo "slow shape calls: $(wc -l < calls.txt)"
: > calls.txt
fast=$(counted_grep -c error log.txt)
echo "fast shape errors: $fast"
echo "fast shape calls: $(wc -l < calls.txt)"

Key points

Fork per stage: every command in a pipeline runs as its own process.
Startup adds up: ten stages pay ten process startups, not one.
Count calls: wrap a stage in a function that logs each run to prove how many times it ran.
Combine stages: one command with the right flags can replace a chain of small ones.
Per-line pipelines: calling an external command once per line multiplies the cost by the line count.
Builtins skip forks: a builtin runs inside the shell, so it costs no new process.

In this example, the script first creates a five-line log with two error lines. The slow shape pipes each line to a counted grep, so it finds 2 errors but pays 5 external calls. The fast shape runs one counted grep over the whole file, finds the same 2 errors, and pays only 1 call.

Try it Yourself

Edit the code, then press Run.

Running is simulated

: > calls.txt
counted_grep() {
  echo hit >> calls.txt
  command grep "$@"
}
printf 'ok\nerror one\nok\nerror two\nok\n' > log.txt
slow=0
while IFS= read -r line; do printf '%s\n' "$line" | counted_grep -q error && slow=$((slow + 1)); done < log.txt
echo "slow shape errors: $slow"
echo "slow shape calls: $(wc -l < calls.txt)"
: > calls.txt
fast=$(counted_grep -c error log.txt)
echo "fast shape errors: $fast"
echo "fast shape calls: $(wc -l < calls.txt)"

Exercise

A script pipes 100 lines through an external command, one line at a time. How many external calls does that shape make?

1 10 100 0