In the last post, I talked about command line tools that I use very often for search files. In this post, I will talk about command line tools for manipulating files or text content. Don’t forget to use man command to find detailed usage of a command.

The following are commands that I use often to manipulate the content of a file or text input:

echo display a line of text
cat concatenate files and print on the standard output
sed stream editor for filtering and transforming text
sort sort lines of text files
uniq filter out repeated lines
tr translate or delete characters
cut extract specific fields from each line of files

I’d like to share some useful examples that I learned over these years.

echo

You can add -e option to echo to enable backslash interpretation:

# \n will be recognized as new line
echo -e "Hello\nWorld!"

cat

You can use cat and redirect operator to add text of multiple lines into a file:

#!/bin/bash
cat > file1 << EOF
line 1
line 2
line 3
EOF

# or
cat << EOF > file1
line 1
line 2
line 3
EOF

sed

A basically example of sed is like this:

sed -i s/good/bad/g file1

which replaces all good with bad in file1. Note that without -i, the change will only be printed to stdout.

I find a lot of people don’t know that you can use different delimiters other than /:

sed -i s:good:bad:g file1

Here i’m using colon : instead of forward slash /. This is really helpful, for instance, if you want to replace a path using sed. It would be painful and error-prone if / is used. You need to escape all the forward slashes in your input like:

sed s/'\/usr\/local\/bin'/'\/bin'/g file1

However, if you change the delimiter to :, you can express it in a much cleaner way:

sed s:/usr/local/bin:/bin:g file1

Another useful tip is the -e option, which allows you to do multiple substitutions simultaneously:

sed -e s/good/bad/g -e s/hello/world/g -e ... file1

sort and uniq

uniq is often used together with sort, because uniq only filters consecutive duplicate lines:

sort file1 file2 | uniq > file3

# or use sort -u
sort -u file1 file2 > file3

tr

tr stands for translate. It’s really powerful when using it with the POSIX bracket expressions.

For example, to convert whitespaces in a file to tabs or vice versa, you can use tr:

$ echo "Hello World!" | tr [:space:] '\t'

Or convert text from upper cases to lower cases:

$ echo "Hello World!" | tr [:upper:] [:lower:]

You can also use tr to remove all new lines of text input or file and join the text into a string:

tr -s '\n' ' ' < file1