Exercise 8: More redirection and filtering: head, tail, awk, grep, sed

Now that you've tried Linux, I'll show you how Unix works. Watch it.

In fact, this means that in order to be proficient with Linux, you need to know how to get output from one program and provide it to another program, usually modifying it in the process. Usually, you can merge multiple programs by using pipes, which allow the output of one program to be connected to another. Like this:

What happened here is really simple. Almost every Linux program opens these three files at startup:

stdin - Standard input. This is where the program reads things.
stdout - Standard output. This is where the program writes things.
stderr - Standard error. This is where the procedure is wrong.

This is how it reads:

Starter 1 
    Start reading data from the keyboard
    Start writing errors to the monitor
    Starter 2 
        Start reading input from program 1
        Start writing errors to the monitor
        Starter 3 
            Start reading input from program 2
            Start writing errors to the monitor
            Start writing data to the monitor

There's another way to describe what's happening, if you like the South Park style of humour, but be careful: what you see is not invisible! Warning! You can't ignore it..

Let's consider the following pipeline, which accepts the output of ls-al and prints only the name and modification time of the file:

ls -al | tr -s ' ' | cut -d ' ' -f 8,9

This is an overview of what happened:

Start ls-al
    Get a list of files in the current directory
    Write an error to the monitor
    Write Output to Pipeline
    Start tr-s''
        Read input from ls-al through pipeline
        Keep only one space between two fields
        Write an error to the monitor
        Write Output to Pipeline
        Start cut-d'-f 8,9
            Read input from tr-s''through pipes
            Keep only fields 8 and 9 and throw away everything else.
            Write an error to the monitor
            Write out the output to the monitor

More specifically, this is what happens at every step:

Step 1: ls-al, we get a list of directories, each column is called a field.

user1@vm1:~$ ls -al
total 52
drwxr-xr-x 2 user1 user1 4096 Jun 18 14:16 .
drwxr-xr-x 3 root  root  4096 Jun  6 21:49 ..
-rw------- 1 user1 user1 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 user1 user1  220 Jun  6 21:48 .bash_logout
-rw-r--r-- 1 user1 user1 3184 Jun 14 12:24 .bashrc
-rw-r--r-- 1 user1 user1   64 Jun 18 14:16 hello.txt
-rw------- 1 user1 user1   89 Jun 18 16:26 .lesshst
-rw-r--r-- 1 user1 user1  634 Jun 15 20:03 ls.out
-rw-r--r-- 1 user1 user1  697 Jun  7 12:25 .profile
-rw-r--r-- 1 user1 user1  741 Jun  7 12:19 .profile.bak
-rw-r--r-- 1 user1 user1  741 Jun  7 13:12 .profile.bak1
-rw------- 1 user1 user1  666 Jun 18 14:16 .viminfo

Step 2: ls-al | tr-s', we only keep between two fields, because cut can't interpret multiple spaces as a way to separate multiple fields.

user1@vm1:~$ ls -al | tr -s ' '
total 52
drwxr-xr-x 2 user1 user1 4096 Jun 18 14:16 .
drwxr-xr-x 3 root root 4096 Jun 6 21:49 ..
-rw------- 1 user1 user1 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 user1 user1 220 Jun 6 21:48 .bash_logout
-rw-r--r-- 1 user1 user1 3184 Jun 14 12:24 .bashrc
-rw-r--r-- 1 user1 user1 64 Jun 18 14:16 hello.txt
-rw------- 1 user1 user1 89 Jun 18 16:26 .lesshst
-rw-r--r-- 1 user1 user1 634 Jun 15 20:03 ls.out
-rw-r--r-- 1 user1 user1 697 Jun 7 12:25 .profile
-rw-r--r-- 1 user1 user1 741 Jun 7 12:19 .profile.bak
-rw-r--r-- 1 user1 user1 741 Jun 7 13:12 .profile.bak1
-rw------- 1 user1 user1 666 Jun 18 14:16 .viminfo

Step 3: We only keep fields 8 and 9, which we want.

user1@vm1:~$ ls -al | tr -s ' ' | cut -d ' ' -f 8,9

14:16 .
21:49 ..
19:34 .bash_history
21:48 .bash_logout
12:24 .bashrc
14:16 hello.txt
16:26 .lesshst
20:03 ls.out
12:25 .profile
12:19 .profile.bak
13:12 .profile.bak1
14:16 .viminfo

Now you've learned how to get input from one program and pass it on to another, and how to convert it.

Do it in this way.

 1: ls -al | head -n 5
 2: ls -al | tail -n 5
 3: ls -al | awk '{print $8, $9}'
 4: ls -al | awk '{print $9, $8}'
 5: ls -al | awk '{printf "%-20.20s %s\n",$9, $8}'
 6: ls -al | grep bash
 7: ls -al > ls.out
 8: cat ls.out
 9: cat ls.out | sed  's/bash/I replace this!!!/g'

What will you see?

user1@vm1:~$ ls -al | head -n 5
total 52
drwxr-xr-x 2 user1 user1 4096 Jun 18 14:16 .
drwxr-xr-x 3 root  root  4096 Jun  6 21:49 ..
-rw------- 1 user1 user1 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 user1 user1  220 Jun  6 21:48 .bash_logout
user1@vm1:~$ ls -al | tail -n 5
-rw-r--r-- 1 user1 user1  636 Jun 18 17:52 ls.out
-rw-r--r-- 1 user1 user1  697 Jun  7 12:25 .profile
-rw-r--r-- 1 user1 user1  741 Jun  7 12:19 .profile.bak
-rw-r--r-- 1 user1 user1  741 Jun  7 13:12 .profile.bak1
-rw------- 1 user1 user1  666 Jun 18 14:16 .viminfo
user1@vm1:~$ ls -al | awk '{print $8, $9}'

14:16 .
21:49 ..
19:34 .bash_history
21:48 .bash_logout
12:24 .bashrc
14:16 hello.txt
16:26 .lesshst
17:52 ls.out
12:25 .profile
12:19 .profile.bak
13:12 .profile.bak1
14:16 .viminfo
user1@vm1:~$ ls -al | awk '{print $9, $8}'

. 14:16
.. 21:49
.bash_history 19:34
.bash_logout 21:48
.bashrc 12:24
hello.txt 14:16
.lesshst 16:26
ls.out 17:52
.profile 12:25
.profile.bak 12:19
.profile.bak1 13:12
.viminfo 14:16

user1@vm1:~$ ls -al | awk '{printf "%-20.20s %s\n",$9, $8}'

.                    14:16
..                   21:49
.bash_history        19:34
.bash_logout         21:48
.bashrc              12:24
hello.txt            14:16
.lesshst             16:26
ls.out               17:52
.profile             12:25
.profile.bak         12:19
.profile.bak1        13:12
.viminfo             14:16
user1@vm1:~$ ls -al | grep bash
-rw------- 1 user1 user1 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 user1 user1  220 Jun  6 21:48 .bash_logout
-rw-r--r-- 1 user1 user1 3184 Jun 14 12:24 .bashrc
user1@vm1:~$ ls -al > ls.out
user1@vm1:~$ cat ls.out
total 48
drwxr-xr-x 2 user1 user1 4096 Jun 18 14:16 .
drwxr-xr-x 3 root  root  4096 Jun  6 21:49 ..
-rw------- 1 user1 user1 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 user1 user1  220 Jun  6 21:48 .bash_logout
-rw-r--r-- 1 user1 user1 3184 Jun 14 12:24 .bashrc
-rw-r--r-- 1 user1 user1   64 Jun 18 14:16 hello.txt
-rw------- 1 user1 user1   89 Jun 18 16:26 .lesshst
-rw-r--r-- 1 user1 user1    0 Jun 18 17:53 ls.out
-rw-r--r-- 1 user1 user1  697 Jun  7 12:25 .profile
-rw-r--r-- 1 user1 user1  741 Jun  7 12:19 .profile.bak
-rw-r--r-- 1 user1 user1  741 Jun  7 13:12 .profile.bak1
-rw------- 1 user1 user1  666 Jun 18 14:16 .viminfo
user1@vm1:~$ cat ls.out | sed  's/bash/I replace this!!!/g'
total 48
drwxr-xr-x 2 user1 user1 4096 Jun 18 14:16 .
drwxr-xr-x 3 root  root  4096 Jun  6 21:49 ..
-rw------- 1 user1 user1 4865 Jun 15 19:34 .I replace this!!!_history
-rw-r--r-- 1 user1 user1  220 Jun  6 21:48 .I replace this!!!_logout
-rw-r--r-- 1 user1 user1 3184 Jun 14 12:24 .I replace this!!!rc
-rw-r--r-- 1 user1 user1   64 Jun 18 14:16 hello.txt
-rw------- 1 user1 user1   89 Jun 18 16:26 .lesshst
-rw-r--r-- 1 user1 user1    0 Jun 18 17:53 ls.out
-rw-r--r-- 1 user1 user1  697 Jun  7 12:25 .profile
-rw-r--r-- 1 user1 user1  741 Jun  7 12:19 .profile.bak
-rw-r--r-- 1 user1 user1  741 Jun  7 13:12 .profile.bak1
-rw------- 1 user1 user1  666 Jun 18 14:16 .viminfo

explain

• Print only the first five entries in the catalog list.
• Print only the last five entries in the catalog list.
• Print only the modification time and file name. Notice how I use awk, which is smarter than cut. The difference here is that cut can only interpret a single symbol (we have spaces here) as a way to separate fields (field separators), awk treats any number of spaces and TAB s as file separators, so there is no need to use tr to eliminate unnecessary spaces.
• Print the file name and modification time in this order. That's what cat can't do.
• Print file name and modification time on site. Notice how the output becomes clearer now.
• Print only the rows containing bash in the catalog list.
• Write the output of the directory list to the file ls.out.
• Print out ls.out. cat is the simplest available program that allows you to print out a file, no more. Despite this simplicity, it is very useful in building complex pipelines.
• Print out ls.out and replace all bash entries with I replace this!!!! sed is a powerful stream editor, and it's very, very useful.

Additional questions

• Open the manual pages of head, tail, awk, grep and sed. Don't be afraid, just remember that the manual page is always there. With some practice, you will be able to actually understand them.
• Find the grep option to print the line before or after it finds.
• Use Google to search the awk printf command and try to understand how it works.
• Read The Useless Use of Cat Award . Try some examples there.

Exercise 9: Bash: Task control, jobs, fg

Linux is a Multitask Operating system. This means that many programs run simultaneously. From the user's point of view, this means that you can run several programs at the same time, and bash must have tools to control the execution of multiple tasks for you. To be able to use this function, you need to learn the following commands:

• < CTRL > + Z - Place the currently running program in the background.
• jobs - Lists all background programs.
• fg - Bring the program to the front desk. fg accepts a number as a parameter, it can get a number from jobs, or if there is no parameter call, it will bring the last suspended program to the foreground.
• ctrl + c - Stops executing the currently running program at one time. Although I will not use it in this exercise, I must say that it may be very useful.

Now, you will learn how to use the built-in tools of bash to control program execution.

Do it in this way.

 1: less -S .profile
 2: <CTRL+z>
 3: less -S .bashrc
 4: <CTRL+z>
 5: less -S .bash_history
 6: <CTRL+z>
 7: jobs
 8: fg
 9: q
10: fg
11: q
12: fg
13: q
14: fg
15: jobs

What will you see?

user1@vm1:~$ less -S .profile
# exists.
# see /usr/share/doc/bash/examples/startup-files for
# the files are located in the bash-doc package.

# the default umask is set in /etc/profile; for setti
# for ssh logins, install and configure the libpam-um
#umask 022

# if running bash
if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
        . "$HOME/.bashrc"

[1]+  Stopped                 less -S .profile
user1@vm1:~$ less -S .bashrc
# for examples

# If not running interactively, don't do anything
[ -z "$PS1" ] && return

# don't put duplicate lines in the history. See bash(
# don't overwrite GNU Midnight Commander's setting of
HISTCONTROL=$HISTCONTROL${HISTCONTROL+:}ignoredups
# ... or force ignoredups and ignorespace
HISTCONTROL=ignoreboth

# append to the history file, don't overwrite it
shopt -s histappend

[2]+  Stopped                 less -S .bashrc
user1@vm1:~$ less -S .bash_history
echo Hello, $LOGNAME!
echo 'echo Hello, $LOGNAME!' >> .profile
cp .profile .profile.bak
tail .profile
ls -altr
history -w
ls -al
cat .profile
echo Hello, $LOGNAME!
echo 'echo Hello, $LOGNAME!' >> .profile
cp .profile .profile.bak
tail .profile
ls -altr

[3]+  Stopped                 less -S .bash_history
user1@vm1:~$ jobs
[1]   Stopped                 less -S .profile
[2]-  Stopped                 less -S .bashrc
[3]+  Stopped                 less -S .bash_history
user1@vm1:~$ fg
user1@vm1:~$ fg
user1@vm1:~$ fg
user1@vm1:~$ fg
-bash: fg: current: no such job
user1@vm1:~$ jobs
user1@vm1:~$

explain

1. Open. profile to view. Notice how I use the - S parameter to let less turn on the - chop-long-lines option to start.
2. Hang less.
3. Open. bashrc to view.
4. Hang less.
5. Open. bash_history to view.
6. Hang less.
7. Print a list of pending programs.
8. Switch to less.
9. Quit it.
10. Switch to the second less.
11. Quit it.
12. Switch to the first less.
13. Quit it.
14. Try switching to the last program. There's no procedure, but you're doing it to make sure there isn't.
15. Print a list of pending programs. This is to ensure that there are no background tasks, by seeing jobs print out empty output.

Additional questions

Open man bash, search JOB CONTROL, enter /, JOB CONTROL, <ENTER> and read it.

Exercise 10: Bash: Program Exit Code (Return Status)

Let's assume you want to copy a directory. You can do this by typing cp-vR/old/dir/path/new/dir/path. After issuing this command, you may want to know how to proceed. Is the directory copied? Or are there some errors because the target directory space is insufficient or something else is wrong?

To understand how it works, you have to understand how the two programs communicate. Let's start by saying that Bash is just another program, so generally speaking, when you issue the above CP command, one program (bash, it's the parent process) calls another program (cp, it's the child process).

In Linux, there is a standard mechanism for obtaining information from child processes to parent processes, which is called Exit status or return code . By using this mechanism, when the child process completes its work, a small number is passed from the child process (or the callee, here cp) to the parent process (or the caller, here bash). When the program does not encounter errors during execution, it returns 0, and if some errors occur, the code is not zero. It's that simple. This exit code in Bash is saved to? Environment variables, as you now know, and can be accessed with $?

Let me repeat what I am saying now:

Bash waits for your input
 Bash parses your input
 Bash starts the program for you and waits for the program to exit
    Program startup
    Programs do what you ask them to do
    The program generates the exit code
    The program exits and returns the exit code to Bash
 Bash assigns this exit code to a variable?

Now you have learned how to print out the exit status of your program.

Do it in this way.

1: ls
2: echo $?
3: ls /no/such/dir
4: echo $?

What will you see?

user1@vm1:~$ ls
hello.txt  ls.out
user1@vm1:~$ echo $?
0
user1@vm1:~$ ls /no/such/dir
ls: cannot access /no/such/dir: No such file or directory
user1@vm1:~$ echo $?
2
user1@vm1:~$

explain

• Print out a catalog. Successful.
• Print out the exit code for ls, which is 0, which means that LS has not encountered any errors.
• Attempts to print out non-existent directories failed, of course.
• Print the exit code for ls/no/like/dir, which is indeed non-zero.

Additional questions

Read the exit code section of man ls.

Exercise 11: Summary

Now that you've tried how to use CLI in Linux, the next step is to open your favorite text editor and make the following tables for yourself. Search for commands and symbols you don't know. Warning! To be effective, you must enter this form manually. Search for these new terms and commands.

Now you will learn how to study something. And remember, don't copy and paste!

term

vim

less

Bash and Bash built-in commands

redirect

Other Procedures You Learned

After filling out the form, write notes for each command later, repeat them, and sleep for a week. Yes, I mean, shake the dust off those pens and paper and do that.

Additional questions

No additional questions. Just learn these commands until you remember them.