Bash

Fix Hanging Bash Pipes and Truncated Loops Under Load

Your log processing pipeline sits there hanging, eating memory, or processing half your data while the upstream script insists it finished five minutes ago. You ran a loop through find, piped the results to xargs or a while read subshell, and your terminal went silent.

Here is the exact state of that broken pipeline.

text
$ find /var/log/app -type f -name "*.log" | while read -r file; do
    echo "Processing $file"
    ssh worker-01 "cat >> /backup/combined.log" < "$file"
done
Processing /var/log/app/app-01.log
$

The script exits after touching only one file, even though thirty log files match the pattern. Or worse, the process never exits at all:

text
$ ps aux | grep -E "find|xargs"
appuser  14210  0.0  0.1  12480  3100 pts/1  S+   23:14   0:00 xargs -I {} ./process_record.sh {}
appuser  14211  0.0  0.0      0     0 pts/1  Z+   23:14   0:00 [process_record.sh] <defunct>

The pipe buffers data silently, processes consume inputs meant for the loop itself, or null-byte separators get stripped by standard POSIX read buffers.

Why Pipelines Stall and Truncate

Three separate mechanics cause this behavior in standard Bash pipelines: standard input starvation, buffer deadlocks, and argument vector limits.

Standard input starvation is the most common reason a loop terminates early. When you pipe text into a while read construct, standard input for the entire loop block becomes that stream. If any binary inside that loop reads from stdin without an explicit redirect, it eats the rest of the stream meant for the read command. Programs like ssh, ffmpeg, mail, and mktemp eagerly consume standard input. In the first example above, ssh established a connection to worker-01 and slurped the remaining list of filenames straight out of standard input before read could run a second iteration.

The second failure point is pipe buffering. Linux defaults to a 64 KB pipe buffer (16 pages of 4096 bytes). When writing to a pipe, tools like grep, sed, and awk switch their internal C runtime buffers from line-buffered mode to block-buffered mode, usually allocating an 8 KB buffer. If the upstream process produces data slower than that buffer size, downstream utilities like fzf or custom parsers display nothing until the buffer fills up or the upstream process terminates. If the downstream consumer stops reading because it is waiting for an external lock, the upstream writer fills the 64 KB kernel pipe buffer and blocks on the write() system call indefinitely.

The third failure point is the combination of whitespace handling and delimiter collision. Default tokenization breaks paths containing spaces, tabs, or newlines unless you lock standard input down to null terminators using find -print0 paired with xargs -0 or read -r -d ''.

Fixing the Pipeline: Two Correct Patterns

The fix depends on whether your pipeline needs sequential execution with per-item shell logic, or parallel batch execution for heavy workloads.

Pattern 1: Sequential Shell Loop Using a Dedicated File Descriptor

To prevent internal commands from stealing your loop’s input data, move the input stream to an alternate file descriptor. Do not use file descriptor 0. Use file descriptor 3.

bash
#!/usr/bin/env bash
set -euo pipefail

# Route the input stream via file descriptor 3
while IFS= read -r -u 3 -d '' filepath; do
    echo "Target: ${filepath}"
    
    # Internal command can safely read stdin, or you can explicitly close stdin
    ssh -n worker-01 "logger -t app-ingest 'Ingesting file'"
    
    # Or explicitly pass the target without stdin interference
    ./process_record.sh "${filepath}" </dev/null

done 3< <(find /var/log/app -type f -name "*.log" -print0)

In this script, < <(...) provides process substitution, keeping the while loop in the parent shell environment instead of a subshell. Variables set inside this loop survive after the loop finishes. The -u 3 flag forces read to consume from file descriptor 3. The ssh -n flag redirects standard input from /dev/null, preventing it from draining other descriptors. The -d '' flag instructs read to use the ASCII NUL character (\0) as the record delimiter, preventing breakage from odd filenames.

To verify the input stream is correctly isolated, simulate a greedy consumer:

bash
while IFS= read -r -u 3 line; do
    echo "Loop read: ${line}"
    # This cat would normally consume the entire stream
    cat > /dev/null
done 3< <(printf "line1\nline2\nline3\n")

The terminal will output all three lines sequentially:

text
Loop read: line1
Loop read: line2
Loop read: line3

Pattern 2: High-Throughput Batch Processing with xargs

If you do not need shell builtins for every record, do not use a while read loop. Use xargs with explicit null terminators, bounded batch sizes, and parallel worker slots.

bash
find /var/log/app -type f -name "*.log" -print0 | \
    xargs -0 -P 4 -n 25 -I {} ./batch_loader.sh {}

This breaks down as follows:

  • -0 expects NUL characters as separators.
  • -P 4 spawns up to four concurrent process forks.
  • -n 25 groups 25 arguments per invocation.
  • -I {} replaces occurrences of the placeholder with the argument. Note that -I implicitly sets -L 1 on GNU systems, overriding -n 25. If you need batch grouping with multiple arguments, drop the replacement string:
bash
find /var/log/app -type f -name "*.log" -print0 | \
    xargs -0 -P 4 -n 25 ./batch_loader.sh

Here, ./batch_loader.sh receives 25 distinct paths as $1, $2, … $25.

Edge Case 1: Subshell Variable Loss in Piped Loops

One pervasive trap with pipe redirection is variable isolation caused by implicit subshell forking. When you write cat file | while read line, POSIX shells fork a subshell to run the right side of the pipe. Any variable mutated inside that loop disappears when the loop exits.

Look at this broken count accumulator:

bash
#!/usr/bin/env bash
count=0

find /var/log/app -type f -name "*.log" | while read -r file; do
    ((count++))
done

echo "Total processed: ${count}"

The output will always be zero:

text
Total processed: 0

To fix this while keeping your code clean, rewrite the loop to read from process substitution, or turn on the Bash 4.2+ lastpipe option. lastpipe executes the last command of a pipeline in the current shell environment, but it only works when job control is disabled.

bash
#!/usr/bin/env bash
set -euo pipefail
shopt -s lastpipe
set +m

count=0

find /var/log/app -type f -name "*.log" -print0 | while IFS= read -r -d '' file; do
    ((count++))
done

echo "Total processed: ${count}"

Output:

text
Total processed: 30

If you are running in an interactive shell or a platform where set +m is undesirable, use the redirection pattern shown in Pattern 1:

bash
count=0
while IFS= read -r -d '' file; do
    ((count++))
done < <(find /var/log/app -type f -name "*.log" -print0)
echo "Total processed: ${count}"

Edge Case 2: Silent Buffer Deadlocks in Interactive Pipes

When debugging pipes with tools like grep or sed chained into an interactive watcher like fzf or a custom python script, the pipeline will appear to freeze if the upstream writer produces low volumes of output.

Take this debugging pipe:

bash
tail -f /var/log/nginx/access.log | grep "500" | cut -d' ' -f1,7

If error 500 requests happen only once every ten seconds, nothing will appear on the screen. grep sees that its output destination is a pipe, not a terminal, and switches from line-buffering to a 4096-byte or 8192-byte block buffer. Those 500 lines will sit in RAM until enough errors occur to spill the buffer over the edge.

To force immediate flushes across the pipeline, use the correct flush switches for each tool:

For GNU grep:

bash
tail -f /var/log/nginx/access.log | grep --line-buffered "500" | cut -d' ' -f1,7

For GNU sed:

bash
tail -f /var/log/nginx/access.log | sed -u 's/foo/bar/'

For general utilities that lack native buffer-control switches, wrap them with stdbuf from coreutils:

bash
tail -f /var/log/nginx/access.log | \
    stdbuf -oL -eL grep "500" | \
    stdbuf -oL -eL cut -d' ' -f1,7

-oL configures the standard output buffer to flush on every newline. -eL sets standard error to flush on newlines as well.

To verify that buffering is gone, pipe a delayed stream into your command:

bash
for i in 1 2 3; do echo "$i"; sleep 2; done | stdbuf -oL grep "[0-9]"

Each number will print immediately, spaced exactly two seconds apart, without stalling until the loop completes.

More in Bash