Shell basics — pipes and redirection

Each shell command does one small job. grep only filters; sort only sorts; wc only counts.

That small tools can accomplish complex processing is thanks to the mechanism for connecting them. Pipes and redirection are it, and grasping these two lets you assemble most tasks.

What you'll learn in this chapter

• The three streams: standard input, standard output, and standard error
• How pipes connect commands
• The difference between > and >> in redirection
• Why standard error doesn't ride pipes, and how to handle it

Prerequisites

Any experience running commands in a terminal is enough.

The three streams

Every command has three I/O channels by default.

Name	Number	Role
Standard input (stdin)	0	Where data to process comes in
Standard output (stdout)	1	Where the processing result goes out
Standard error (stderr)	2	Where errors and warnings go out

The key is that output is split in two. Emitting results and errors through separate channels lets you pass only the results to the next step. You can leave errors on the screen while saving only the correct results to a file.

The numbers (0, 1, 2) are file descriptors, used to name the target in redirection.

Pipe — one command's output into the next's input

| connects the left command's standard output to the right command's standard input.

cat access.log | grep "POST" | wc -l

Data is transformed as it flows without creating intermediate files. Chain as many stages as you like. Each stage runs in its own process concurrently; the right side reads as the left side writes (Processes and threads).

Only standard output rides pipes. Standard error is a separate channel, so it bypasses the pipe and goes directly to the screen.

# When you specify a nonexistent file
cat missing.txt | grep buy
# → cat's error goes to the screen. grep receives nothing

That's intentional. Error messages flowing into downstream processing would be treated as data and misbehave.

Redirection — route to a file

Redirection switches an I/O channel's target to a file.

Symbol	Meaning
>	Route stdout to a file. Clears the file first, then writes (overwrite)
>>	Route stdout to a file. Appends to the end
<	Route a file into stdin
2>	Route stderr to a file
2>&1	Route stderr to the same target as stdout

The difference between > and >>

This gap directly causes real incidents.

# > wipes and writes each time
grep buy log_a.txt > out.txt   # out.txt = result of a
grep buy log_b.txt > out.txt   # out.txt = result of b only; a is gone

# >> accumulates
grep buy log_a.txt > out2.txt   # first time, use > to create (or >> is fine)
grep buy log_b.txt >> out2.txt  # out2.txt = a + b

For accumulating daily logs into one file, use >>. With >, each day's file has only that day's content and the past is gone.

Overwriting existing files with > causes many incidents, so when the target is an existing file, get in the habit of asking whether append would do.

Handling standard error

command > out.txt              # Only result to the file. Errors to screen
command 2> err.txt             # Only errors to the file. Result to screen
command > out.txt 2> err.txt   # Separate files
command > all.txt 2>&1         # Both to the same file
command 2>/dev/null            # Discard errors

Read 2>&1 as "redirect file descriptor 2 to the current target of file descriptor 1". The & is there to indicate "file descriptor 1", not "a file named 1".

Order matters.

command > all.txt 2>&1   # Correct. Point 1 at all.txt, then 2 at the same target
command 2>&1 > all.txt   # Wrong. Point 2 at the current 1 (screen), then move 1 to all.txt

Processing is left-to-right, so 2>&1 copies "the target of 1 at that moment". Writing it before you point 1 at the file leaves errors on the screen.

grep — filtering by line

The command you use most with pipes.

grep buy log.txt          # Extract lines containing "buy" from a file
cat log.txt | grep buy    # Extract from standard input (same result)

Option	Effect
-i	Ignore case
-v	Emit non-matching lines (invert)
-n	Prepend line numbers
-c	Emit only the count of matching lines
-r	Recursively search directories
-E	Use extended regular expressions

Since grep accepts a filename argument, cat file | grep pattern can be written as grep pattern file. The latter has one fewer process and is more efficient.

That said, starting from cat is widely used just because the pipe reads left-to-right. When you're iterating on a long pipeline, starting with cat is easier to edit. Neither is "correct".

Composition examples

# List the most frequent error occurrences
grep ERROR app.log | awk '{print $4}' | sort | uniq -c | sort -rn | head -10

# Append only that day's POST requests to a separate file
grep "2026-08-17" access.log | grep POST >> post-2026-08.log

# Record both output and errors while watching them on screen
./deploy.sh 2>&1 | tee deploy.log

tee "writes input to a file while also passing it to standard output". Use it to save contents mid-pipe. Once you move to scripts, environment variables and PATH become the next threshold.

Common misconceptions

"> appends." — > overwrites. >> appends. Using > on an existing file wipes its contents.

"Errors flow through the pipe." — Only standard output does. To route standard error through, merge with 2>&1.

"2>&1 > file puts errors in the file." — It doesn't. Order is reversed. Write > file 2>&1.

"cat file | grep x is wrong." — Not wrong. grep x file uses fewer processes, but the pipe form is a common assembly style.

Exercises

Q1. You ran the following daily batch for three days. On the morning of day 4, how many days' worth of results are in daily.txt? /var/log/app.log rotates at midnight, and the batch runs just before midnight.

#!/usr/bin/env bash
grep ERROR /var/log/app.log > daily.txt

Answer and explanation

Answer: One day's worth (only day 3's result)

> empties the file when opened. Running daily wipes what accumulated before, leaving only the last run's result.

To accumulate three days, use >>. Since the log being read changes daily, appending accumulates each day.

grep ERROR /var/log/app.log >> daily.txt

Choose > vs. >> by what the file should look like at the end. Accumulate → append. Rebuild every time → overwrite.

That said, only >> accumulates forever, so operations combine it with:

Means	Content
Split by period	Like >> errors-$(date +%Y%m).log for monthly files
Log rotation	Use logrotate and similar to cycle by size or period

Q2. You want to keep both the output and errors of a script in one log file. Which is right?

A: ./run.sh > log.txt
B: ./run.sh 2> log.txt
C: ./run.sh > log.txt 2>&1
D: ./run.sh 2>&1 > log.txt

Answer and explanation

Answer: C

• A — only stdout to file. stderr stays on screen
• B — only stderr to file. stdout stays on screen
• C — correct. Point 1 at log.txt, then point 2 at "the current target of 1" = log.txt
• D — order is reversed. Point 2 at "the current target of 1" (screen), then move 1 alone to log.txt. Result to file, errors to screen

Remembering that redirection is processed left to right stops this confusion.

To also see it on screen, use ./run.sh 2>&1 | tee log.txt.

Q3. ./build.sh | grep -i warning runs, but not one of the warnings you see on screen matches. Why?

Answer and explanation

Answer: Warnings go to standard error. Pipes carry only standard output.

Many tools emit results to stdout and warnings/errors to stderr. On screen they mix, so you can't distinguish visually.

To pass to grep, merge stderr into stdout.

./build.sh 2>&1 | grep -i warning

Conversely, this separation is by design. Error messages flowing into downstream processing as data would break aggregation and transformation. Because of "pass results forward, show anomalies to humans", pipes can safely chain.

To filter only stderr, drop stdout first, then pass.

./build.sh 2>&1 >/dev/null | grep -i warning

The order is peculiar. The pipe is set up before the command's own redirections, so at the time 2>&1 is evaluated, 1 points to the pipe. Point 2 there, then move 1 alone to /dev/null. Result: only stderr flows to the pipe.

This is bash behavior. In zsh, MULTIOS is enabled by default so >/dev/null doesn't replace 1's target but becomes an additional target, and stdout also flows to the pipe. To behave the same on both shells, drop stdout first.

{ ./build.sh >/dev/null; } 2>&1 | grep -i warning

Summary

• Commands have three channels: standard input, standard output, standard error
• Pipe | connects the left's stdout to the right's stdin. stderr does not flow through
• > overwrites, >> appends. > on an existing file wipes it
• 2>&1 is "point 2 at 1's current target". Write > file 2>&1 in that order
• grep takes a filename directly. Starting from cat is also common for ease of assembly

Related references

• Local development environment — Command examples for running this site locally

What to read next

• Shell scripting and environment variables — Pitfalls when moving to scripts