🎓 A1Learner
🔍
0 of 45 lessons completed

Bash Project: Parallel Runner

A parallel task runner is a script that runs several jobs at once within a fixed number of slots. It collects each job result and reports them in task order.

wait -n frees a slot when one job finishes

Beta finishes first, so wait -n returns and gamma starts while alpha still runs. The results still print in task order, not finish order.

run_task() {
  sleep "$2"
  echo "$1 done" > "result-$1.txt"
}
run_task alpha 0.3 &
run_task beta 0.1 &
wait -n
run_task gamma 0.1 &
wait
for task in alpha beta gamma; do
  cat "result-$task.txt"
done
echo "all tasks done"

A slot loop drains every job before reporting

The loop never holds more than two jobs, and a final wait drains the rest. The count 4 plus the four lines in order prove no task was lost.

run_task() {
  echo "$1 done" > "result-$1.txt"
}
tasks=("one" "two" "three" "four")
running=0
for task in "${tasks[@]}"; do
  run_task "$task" &
  running=$((running + 1))
  if [[ "$running" -ge 2 ]]; then
    wait -n
    running=$((running - 1))
  fi
done
wait
echo "ran ${#tasks[@]} tasks"
for task in "${tasks[@]}"; do
  cat "result-$task.txt"
done

Failures are collected, not allowed to stop the run

Gamma fails while the other jobs still run, and wait gathers them all. The failure list names only gamma, which proves failures were collected in order.

run_task() {
  if [[ "$1" == "gamma" ]]; then
    echo "failed" > "result-$1.txt"
    return 1
  fi
  echo "ok" > "result-$1.txt"
}
for task in alpha beta gamma delta; do
  run_task "$task" &
done
wait
failed=""
for task in alpha beta gamma delta; do
  if [[ "$(cat "result-$task.txt")" == "failed" ]]; then
    failed="$failed $task"
  fi
done
echo "failures:$failed"
echo "runner kept going after a failure"

Example

tasks=("alpha" "beta" "gamma" "delta")
run_task() {
  if [[ "$1" == "beta" || "$1" == "gamma" ]]; then
    sleep 0.1
  else
    sleep 0.2
  fi
  if [[ "$1" == "gamma" ]]; then
    echo "failed" > "result-$1.txt"
    return 1
  fi
  echo "ok" > "result-$1.txt"
}
running=0
for task in "${tasks[@]}"; do
  run_task "$task" &
  running=$((running + 1))
  if [[ "$running" -ge 2 ]]; then
    wait -n
    running=$((running - 1))
  fi
done
wait
passed=0
failed=0
for task in "${tasks[@]}"; do
  if [[ "$(cat "result-$task.txt")" == "ok" ]]; then
    echo "$task: ok"
    passed=$((passed + 1))
  else
    echo "$task: failed"
    failed=$((failed + 1))
  fi
done
echo "passed: $passed"
echo "failed: $failed"
if [[ "$failed" -gt 0 ]]; then
  exit 1
fi

Key points

  • Job slots: a cap on how many background jobs run at once.
  • wait -n: waits for one background job to finish, then frees one slot.
  • Collect then print: each job writes its own result file, and the main script prints results in task order.
  • Failure file: a job records ok or failed, so the main script can count failures after wait.
  • Final wait: a plain wait after the loop drains every remaining job before results print.
  • Never completion order: printing as jobs finish would shuffle the output from run to run.

In this example, a runner launches four tasks with at most two jobs at once, using wait -n to free a slot when one job finishes. Each job writes ok or failed to its own result file, and a final wait drains the rest. The runner then prints every result in task order, not completion order, counts three passes and one failure, and exits nonzero because gamma failed.

Try it Yourself

Edit the code, then press Run.

Running is simulated

Exercise

Why must a parallel runner print results in task order, not completion order?