1. grep tool
grep is a * * line * * filtering tool; Used to filter rows based on keywords
Syntax and options
Syntax:
# grep [options] 'keyword' file name
Common options:
OPTIONS:
-i: Case insensitive
-v: Finds a row that does not contain the specified content,Reverse selection
-w: Search by word
-o: Print matching keywords
-c: Count the number of rows matched
-n: set number
-r: Traverse directory lookup layer by layer
-A: Displays the matching line and how many lines follow
-B: Displays the matching line and how many lines precede it
-C: Show how many lines before and after the matching line
-l: Only matching file names are listed
-L: List mismatched file names
-e: Use regular matching
-E:Use extended regular matching
^key:Start with keyword
key$:End with keyword
^$:Match blank lines
--color=auto : You can add the color display to the found keyword part
Color display (alias settings):
Temporary settings: # alias grep='grep --color=auto' // Effective only for current terminal and current user Permanent settings: 1)Global (effective for all users) vim /etc/bashrc alias grep='grep --color=auto' source /etc/bashrc 2)Local (for a specific user) vim ~/.bashrc alias grep='grep --color=auto' source ~/.bashrc
For example:
Note: do not directly use the / etc/passwd file, copy it to / tmp for experiment!
# grep -i root passwd Ignore case matching rows containing root # grep -w ftp passwd Exactly match FTP words # grep -w hello passwd Accurately match Hello words; Add your own line containing hello to the file # grep -wo ftp passwd Print the matching keyword FTP # grep -n root passwd Print a good match to the root keyword # grep -ni root passwd Ignore case matching statistics for rows containing the keyword root # grep -nic root passwd Ignore case matching and count the number of rows containing the keyword root # grep -i ^root passwd Ignore case matching lines starting with root # grep bash$ passwd Match lines ending in Bash # grep -n ^$ passwd Match blank lines and print line numbers # grep ^# /etc/vsftpd/vsftpd.conf Match to#Lines beginning with # grep -v ^# /etc/vsftpd/vsftpd.conf Do not match#Lines beginning with # grep -A 5 mail passwd The match contains the mail keyword and its next 5 lines # grep -B 5 mail passwd The match contains the mail keyword and its first 5 lines # grep -C 5 mail passwd The match contains the mail keyword and its first and last 5 lines
2. cut tool
cut is a * * column * * interception tool, which is used to intercept columns
Syntax and options
Syntax:
# cut option file name
Common options:
-c: Split in characters,intercept -d: Custom separator, tab by default\t -f: And-d Used together to specify which area to intercept
For example:
# cut -d: -f1 1.txt Divide with colon and intercept the contents of column 1 # cut -d: -f1,6,7 1.txt Divide with colon and intercept the contents of columns 1, 6 and 7 # cut -c4 1.txt Intercept the 4th character of each line in the file # cut -c1-4 1.txt Intercept 1-4 characters per line in the file # cut -c4-10 1.txt Intercept 4-10 characters per line in the file # cut -c5- 1.txt Intercept all the following characters from the fifth character
practice:
List the operation levels of your current system with gadgets. 5/3
- How to view system operation level
- Command runlevel
- File / etc/inittab
- How to filter run levels
runlevel |cut -c3 runlevel | cut -d ' ' -f2 grep -v '^#' /etc/inittab | cut -d: -f2 grep '^id' /etc/inittab |cut -d: -f2 grep "initdefault:$" /etc/inittab | cut -c4 grep -v ^# /etc/inittab |cut -c4 grep 'id:' /etc/inittab |cut -d: -f2 cut -d':' -f2 /etc/inittab |grep -v ^# cut -c4 /etc/inittab |tail -1 cut -d: -f2 /etc/inittab |tail -1
3. sort tool
sort tool is used for sorting; It takes each line of the file as a unit, compares the ASCII values from the first character back, and finally outputs them in ascending order.
Syntax and options
-u : Remove duplicate lines -r : Sort is descending by default -o : Output the sorting results to a file,Similar redirection symbol> -n : Sort by number. By default, sort by character -t : Separator -k : The first N column -b : Ignore leading spaces. -R : Random sorting, the results of each run are different
Examples
# sort -n -t: -k3 1.txt Sort in ascending order according to the user's uid # sort -nr -t: -k3 1.txt Descending by user uid # sort -n 2.txt Sort by number # sort -nu 2.txt Sort by number and de duplicate # sort -nr 2.txt # sort -nru 2.txt # sort -nru 2.txt # sort -n 2.txt -o 3.txt Sort by number and redirect the results to a file # sort -R 2.txt # sort -u 2.txt
4.uniq tools
uniq is used to remove * * consecutive duplicate * * lines
Common options: -i: ignore case -c: Count the number of repeated lines -d:Show only duplicate lines For example: # uniq 2.txt # uniq -d 2.txt # uniq -dc 2.txt
5.tee tools
tee tool reads and writes from standard input to standard output and file, that is, bidirectional overlay redirection (screen output | text input)
Options: -a Bidirectional additional redirection # echo hello world # echo hello world|tee file1 # cat file1 # echo 999|tee -a file1 # cat file1
6.diff tool
diff tool is used to compare different files line by line
Note: diff describes two files differently by telling us how to change the first file and then match the second file.
Syntax and options
Syntax:
diff [option] File 1 file 2
Common options:
| option | meaning | remarks |
|---|---|---|
| -b | Do not check spaces | |
| -B | Do not check blank lines | |
| -i | Do not check case | |
| -w | Ignore all spaces | |
| –normal | Normal format display (default) | |
| -c | Context format display | |
| -u | Merge format display |
For example:
- Compare the similarities and differences between the two ordinary documents, and prepare the documents:
[root@MissHou ~]# cat file1 aaaa 111 hello world 222 333 bbb [root@MissHou ~]# [root@MissHou ~]# cat file2 aaa hello 111 222 bbb 333 world
1) Normal display
diff Purpose: file1 How to change talent and file2 matching [root@MissHou ~]# diff file1 file2 1c1,2 The first line of the first file needs to be changed(c=change)To match lines 1 to 2 of the second file < aaaa Less than sign"<"Represents the file on the left(file1)Document content --- ---Represents the separator > aaa Greater than sign">"Represents the file on the right(file2)Document content > hello 3d3 Delete line 3 of the first file(d=delete)Before it can match the third line of the second file < hello world 5d4 The first and second lines of the file can only be deleted after the first and fifth lines of the file match < 333 6a6,7 Add line 6 of the first file(a=add)The content can be matched with lines 6 to 7 of the second file > 333 The contents to be added in the second document are 333 and 333 world > world
2) Context format display
[root@MissHou ~]# diff -c file1 file2 The first two lines mainly list the file name and time stamp of the file to be compared; Symbol before file name***express file1，---express file2 *** file1 2019-04-16 16:26:05.748650262 +0800 --- file2 2019-04-16 16:26:30.470646030 +0800 *************** I'm a separator *** 1,6 **** with***Opening representation file1 File, 1,6 Indicates 1 to 6 lines ! aaaa !Indicates that the line needs to be modified to match the second file 111 - hello world -Indicates that the line needs to be deleted to match the second file 222 - 333 -Indicates that the line needs to be deleted to match the second file bbb --- 1,7 ---- with---Opening representation file2 File, 1,7 Indicates lines 1 to 7 ! aaa Indicates that the first file needs to be modified to match the second file ! hello Indicates that the first file needs to be modified to match the second file 111 222 bbb + 333 Indicates that the first file needs to add this line to match the second file + world Indicates that the first file needs to add this line to match the second file
3) Merge format display
[root@MissHou ~]# diff -u file1 file2 The first two lines mainly list the file name and time stamp of the file to be compared; Symbol before file name---express file1，+++express file2 --- file1 2019-04-16 16:26:05.748650262 +0800 +++ file2 2019-04-16 16:26:30.470646030 +0800 @@ -1,6 +1,7 @@ -aaaa +aaa +hello 111 -hello world 222 -333 bbb +333 +world
- Compare the differences between the two directories
By default, the contents of the same files in the two directories will also be compared [root@MissHou tmp]# diff dir1 dir2 diff dir1/file1 dir2/file1 0a1 > hello Only in dir1: file3 Only in dir2: test1 If you only need to compare the differences between the files in the two directories, you don't need to further compare the file contents, you need to add-q option [root@MissHou tmp]# diff -q dir1 dir2 Files dir1/file1 and dir2/file1 differ Only in dir1: file3 Only in dir2: test1
Other tips:
Sometimes we need to take one file as the standard to modify other files, and when there are many modifications, we can complete it by patching.
1)First find out the differences between files, and then output to a file [root@MissHou ~]# diff -uN file1 file2 > file.patch -u:Context mode -N:Treat non-existent files as empty files 2)Patch different contents to the file [root@MissHou ~]# patch file1 file.patch patching file file1 3)Test verification [root@MissHou ~]# diff file1 file2 [root@MissHou ~]#
7. paste tool
The paste tool is used to merge file lines
Common options: -d: Custom spacer, default is tab -s: Serial processing, non parallel
8. tr tools
tr is used for character conversion, replacement and deletion; It is mainly used to delete control characters or convert characters in files
Syntax and options
Syntax:
Usage 1: submit the execution result of the command to tr Processing, where string1 For query, string2 For conversion processing commands|tr 'string1' 'string2' Usage 2: tr The processing content comes from the file, remember to use"<"Standard input tr 'string1' 'string2' < filename Usage 3: Match string1 Perform corresponding operations, such as deleting tr [options] 'string1' < filename
Common options:
-d Delete all input characters in string 1. -s Delete all repeated character sequences and keep only the first one; The recurring string is about to be compressed into one string
Constant matching string:
| character string | meaning | remarks |
|---|---|---|
| a-z or [: lower:] | Match all lowercase letters | All cases and numbers [a-zA-Z0-9] |
| A-Z or [: upper:] | Match all uppercase letters | |
| 0-9 or [: digit:] | Match all numbers | |
| [:alnum:] | Match all letters and numbers | |
| [:alpha:] | Match all letters | |
| [:blank:] | All horizontal blanks | |
| [:punct:] | Match all punctuation marks | |
| [:space:] | All horizontal or vertical spaces | |
| [:cntrl:] | All control characters | \f Ctrl-L line feed \n Ctrl-J wrap |
| \r Ctrl-M enter | ||
| \t Ctrl-I tab |
For example:
[root@MissHou shell01]# cat 3.txt Create this file yourself for testing ROOT:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail:/sbin/nologin uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin boss02:x:516:511::/home/boss02:/bin/bash vip:x:517:517::/home/vip:/bin/bash stu1:x:518:518::/home/stu1:/bin/bash mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin aaaaaaaaaaaaaaaaaaaa bbbbbb111111122222222222233333333cccccccc hello world 888 666 777 999 # tr -d '[:/]' < 3. txt Delete files: and/ # cat 3.txt |tr -d '[:/]' Delete files: and/ # tr '[0-9]' '@' < 3. txt Replace the number in the file with the @ symbol # tr '[a-z]' '[A-Z]' < 3. txt Replace lowercase letters with uppercase letters in the file # tr -s '[a-z]' < 3. txt Match lowercase letters and compress duplicates into one # tr -s '[a-z0-9]' < 3. txt Match lowercase letters and numbers and compress duplicates into one # tr -d '[:digit:]' < 3. txt Delete numbers from file # tr -d '[:blank:]' < 3. txt Delete horizontal blank # tr -d '[:space:]' < 3. txt Delete all horizontal and vertical whitespace
Small trial ox knife
- Use gadgets to intercept the current host IP respectively; Intercept NETMASK; Intercepting broadcast address; Intercept MAC address
# ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4 10.1.1.1:10.1.1.255:255.255.255.0 # ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4|tr ':' '\n' 10.1.1.1 10.1.1.255 255.255.255.0 # ifconfig eth0|grep 'HWaddr'|cut -d: -f2-|cut -d' ' -f4 00:0C:29:25:AE:54 # ifconfig eth0|grep 'HW'|tr -s ' '|cut -d' ' -f5 00:0C:29:B4:9E:4E # ifconfig eth1|grep Bcast|cut -d: -f2|cut -d' ' -f1 # ifconfig eth1|grep Bcast|cut -d: -f2|tr -d '[ a-zA-Z]' # ifconfig eth1|grep Bcast|tr -d '[:a-zA-Z]'|tr ' ' '@'|tr -s '@'|tr '@' '\n'|grep -v ^$ # ifconfig eth0|grep 'Bcast'|tr -d [:alpha:]|tr '[ :]' '\n'|grep -v ^$ # ifconfig eth1|grep HWaddr|cut -d ' ' -f11 # ifconfig eth0|grep HWaddr|tr -s ' '|cut -d' ' -f5 # ifconfig eth1|grep HWaddr|tr -s ' '|cut -d' ' -f5 # ifconfig eth0|grep 'Bcast'|tr -d 'a-zA-Z:'|tr ' ' '\n'|grep -v '^$'
- Save the user name, password and default shell of all ordinary users in the system to a file. It is required to separate the user name, password and default shell with tab key
# grep 'bash$' passwd |grep -v 'root'|cut -d: -f1,2,7|tr ':' '\t' |tee abc.txt