03 Linux file management (directory)

1. Overview of document management

1.Bash Shell manages files

When it comes to Linux file management, the first thing we need to know is, what should we do with files?
In fact, it is nothing more than to, create, copy, move, view, edit, compress, find, delete, and so on
For example, when we want to modify the host name of the system, should we know where the file is before we can make the corresponding modification?

 1 Set alias
 alias www='ls /'  Only in the current bashshell Effective in
 2 Permanently effective (global, local),
	echo "alias wk='ifconfig'" >> /etc/bashrc
 3 unalias wk  
 4 If it is written to the configuration file and cancelled unalias wk+To delete the configuration file
 5 Built in alias: placed in the configuration file of environment variables
	alias ls
 6 /bin/ls -al /root   # ls has built-in aliases

 7 command --help
 8 man command

2. Content summary

System directory structure
File path location
File management command
File type file
Link file ln
File editing vim

2. System directory structure

Almost all computer operating systems use a directory structure to organize files. Since the files stored in the subdirectory and subdirectory are also called "a tree structure", and the files stored in the subdirectory and subdirectory are also called "a tree structure".
Windows: organize files by multiple roots C: D:
Linux: organizing files as a single root/

The following figure shows the directory structure of Centos7

1. Store the directory related to the command

/Bin, the commands used by ordinary users, / bin/ls, /bin/date
/sbin, commands used by the administrator / sbin/service,poweroff,useradd

As long as you see the bin path, you should realize that you are putting executable files

2. Home directory for storing user related data, such as windows. Different users log in to the system and display different desktop backgrounds

/Home, the home directory of ordinary users. The default is / home/username

Don't delete the environment variables. Try deleting them

/Root, the home directory of the super administrator root, and ordinary users have no right to operate

3. System file directory

/usr, equivalent to C:Windows
/usr/local, the directory of software installation, equivalent to C:Program
/usr/bin /, applications used by ordinary users (important)
/usr/sbin, applications used by administrators (important)
/usr/lib, library file Glibc 32bit
/usr/lib64, library file Glibc 64bit

# 1 
df -h #df -h view the usage of files in the system
Size Total capacity of partition
Used Used size
Avail Remaining size
Use% Percentage used
Mounted on Path address
# 2
du -sh * View the space occupied by each file and directory in the current directory
du -sh /usr/
# 3 standards are the basis of automation
 The same software without host is placed in the same path, which is convenient for management and automation
 Now generally installed software is not placed in/usr/local I'm off

# There are a thousand orders in all
ls /usr/bin/ | wc -l
ls /usr/sbin/ | wc -l

# 5. Check which library files the command depends on
ldd /bin/ls

4. Startup directory

/Boot stores files related to system startup, such as kernel and grub (boot loader)

ls /boot
# linux kernel
# Startup mode that can be selected when starting the machine

5. Configuration file directory

/etc is extremely important. All subsequent service configurations are in this directory
/Etc / sysconfig / network script / ifcfg -, network configuration file
/etc/hostname, system host name, configuration file and host name are very important. Some special services depend on host name. If there is no host name, it will report an error and can't get up; Modified to restart: reboot
/etc/resolv.conf, dns client configuration file and domain name resolution server are generally not configured, because the network card will be overwritten after it is configured, and the priority of the network card is high
/etc/hosts, local domain name resolution configuration file, domain name resolution, find your own hosts first, and then go to domain name resolution

# 1 
/etc/hosts corresponding windows C:windows/system32/drivers/etc/hosts,Hacker phishing website
# 2 test modification
yum install httpd -y
systemctl stop firewalld
echo "lqz NB" >/var/www/html/index.html
systemctl start httpd

6. Variable directory and temporary directory

/VaR, which stores some change files, such as log files under / var/log / and login logs
/var/tmp, temporary file generated by the process
/tmp, system temporary directory (similar to public toilet), anyone can use it

# 1. Check the login log
cat /var/log/secure #View login time
# 2 temporary files generated by the process (360 will clean up the garbage)

7. Equipment catalog file

/dev, which stores device files, such as hard disk, hard disk partition, optical drive, etc
/dev/sd hard disk device
/dev/null, black hole device, only in and out. Similar to garbage collection bin
/dev/random, a device that generates random numbers
/dev/zero, which can generate data far and continuously, is similar to the ATM, which can withdraw money anytime and anywhere

# 1 sda sdb sdc sda1 sdb4
linux Medium disk file sd,The first hard disk is called a,The second one is b,sda1 Represents the first partition of the first disk, sdb4: The fourth partition of the second hard disk (the service can insert many hard disks)
# 2 /dev/null 
ls >/dev/null
# 3 /dev/random generates random numbers
echo $RANDOM
echo lqz_$RANDOM
 Create random users in batches and set passwords in batches
# 4. Keep getting data
dd if=/dev/zero of=/opt/test.txt bs=1M count=1024
'''
dd: Copy a file with a block of specified size, and perform the specified conversion at the same time.
if=File name: enter a file name. The default is standard. That is, specify the source file.< if=input file >
of=File name: the name of the output file. The default is standard output. That is, specify the destination file.< of=output file >
 bs=bytes: Set read in at the same time/The output block size is bytes Bytes.
 count=blocks: Copy only blocks Blocks, block size equal to ibs Number of bytes specified.
'''

ll /opt/test.txt
ll -h /opt/test.txt

8. Virtual file system (if the corresponding process stops, the corresponding directory under / proc will be deleted)

/proc, reflecting the real-time status of the current process of the system: process
PS: similar to the dashboard of a car, you can see whether the car is faulty or short of oil.

ls /proc # You can see many id numbers, pid numbers, process numbers and unique
ls process id Folder with number
 If the process is shut down, id No. folder is gone
id No. is not unique every time it is started. Only one process is unique, systemd Is a process with process number 1. All processes are derived from it

9 others

#1 media: provides the mounting point of the device and media files
# linux has a new drive letter, which needs to be mounted manually
# Mount the data in the CD to the media directory
mount  /dev/cdrom /media/

# 2 mnt: provide mounting point of equipment (ditto)

# 3 opt: a third-party tool. The third-party software is installed by default (mysql...)

# run: the file with pid and log at the end
ls /run
cat sshd.pid  # The pid number of the process running and put it in the file
ps aux |grep sshd

# Function of. lock file, locking mechanism
# Suppose you execute now
yum install tree
# Open another window and execute the same command
yum install tree
'''
Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum
    Memory :  71 M RSS (470 MB VSZ)
    Started: Tue Aug 18 00:26:31 2020 - 00:24 ago
    State  : Sleeping, pid: 6191
'''
cat /run/yum.pid

10.PS: in Linux 7 system,

/bin, /sbin, /lib, /lib64 are all linked to the / usr / directory in the form of soft links

/bin –> /usr/bin
/sbin –> /usr/sbin
lib -> usr/lib
lib64 -> usr/lib64

3. File path location

Before we start to operate the file, we first need to locate the file, such as:
Where do you want to create what file? Where do you want to copy the file? Or where is the file you want to delete?
What is location: for example, / etc/hostname, the whole file contains the file name and the location of the file. We call this path, and the path is a way to locate the file. Example: as shown in the figure below, the path of message is?

Ask FQ: is / home / Oldboy / file and / home/oldgirl/file the same file?

1. There is one under each directory And What's your catalog for?

One point represents the current directory, and two points represent the upper directory of the current directory.
Assuming that the current directory is under / usr /, its upper (/) directory uses/ The lower level (local) directory of / usr / is represented by/ Local means.

2. What is the absolute path and what is the relative path?

Absolute path: only the path starting from /, such as / home/alice/file
Relative path: relative to the current directory, such as a.txt/ a.txt ../ Bob / A.mp3 [add: in the directory / home/alice]
Summary: the so-called (.) And (..) A directory is actually a representation of a relative path.

3. Examples of absolute path and relative path

#Absolute path
[root@bgx /]# useradd alice
[root@bgx /]# touch /home/alice/file1
[root@bgx /]# touch ~/file2
[root@bgx /]# touch ~alice/file3

#Relative path
[root@bgx /]# mkdir abc
[root@bgx /]# touch ../file3
[root@bgx /]# touch file4
[root@bgx /]# touch abc/file5

cd change directory, common usage

# cd absolute path cd /etc/hostname
# CD relative path CD test / ABC CD cd ..


# cd      #Switch directories, for example: cd /etc
# cd -    #Switch back to the last directory
# cd ~    #Switch back to the current user's home directory. Note: is root different from ordinary users?
# cd .    #Represents the current directory. Generally, cp /etc/hostname is used when copying, moving, etc/
# cd ..   #Switch back to the parent directory of the current directory

4. System document management

1. Document management: create / copy / move / delete

establish

1. File creation command touch

# touch file                    #If there is none, it will be created; if there is one, it will be modified
# touch file2 file3
# touch /home/od/file4 file5
# touch file{a,b,c}             #{} set, equivalent touch a b c
# touch file{1..10}
# touch file{a..z}

2. Directory creation command mkdir

# Options: - v show details - p create directory recursively
# mkdir dir1
# mkdir /home/od/dir1 /home/od/dir2
# mkdir -v /home/od/{dir3,dir4} 
# mkdir -pv /home/od/dir5/dir6
# mkdir -pv /home/{od/{diu,but},boy}

3. Display the directory structure in a tree command tree

# Options: - L: displays the hierarchy of the directory tree
# tree /home/od/    #Displays the structure under the current directory
/home/od/
├── but
├── dir1
├── dir2
├── dir3
├── dir4
├── dir5
│   └── dir6
└── diu

cp replication

#Options: - v: detailed display of operations performed by the command - r: recursive processing of directories and subdirectories - p: retention of attributes of source files or directories

# cp file /tmp/file_copy
# cp name /tmp/name         #Do not modify name
# cp file /tmp/             #Do not modify name
# cp -p file /tmp/file_p    #-p keep the attributes of the original file or directory
# cp -r  /etc/ /tmp/        #To copy a directory, you need to use the - r parameter to copy recursively
# cp -rv /etc/hosts /etc/hostname /tmp  #Copy multiple files to a directory
# cp -rv /etc/{hosts,hosts.bak}
# cp -rv /etc/hosts{,-org}

mv Mobile

# mv file file1             #Change the name when moving in place
# mv file1 /tmp/            #Move files to tmp directory
# mv /tmp/file1 ./          #Move files from tmp directory to current directory
# mv dir/ /tmp/             #Move directory to / tmp directory

# touch file{1..3}
# mv file1 file2 file3 /opt/    #Move multiple files or to the same directory

# mkdir dir{1..3}
# mv dir1/ dir2/ dir3/ /opt     #Move multiple directories to the same directory

rm delete

#Options: - r: recursion - f: forced deletion - v: detailed procedure
# rm  file.txt      #If the file alias exists, rm will delete it by default
# rm -f file.txt    #Delete files without reminding


# rm -r dir/        #You will be prompted if you delete the directory recursively
# rm -rf dir/       #Forcibly delete the directory without reminding (use with caution)


#1.rm delete example
# mkdir /home/dir10
# touch /home/dir10/{file2,file3,.file4}
# rm -f /home/dir10 / / / excluding hidden files 
# ls /home/dir10/ -a
. .. .file4


#2.rm delete example 2
# touch file{1..10}
# touch {1..10}.pdf
# rm -rf file 
# rm -rf .pdf

2. Document management: cat TAC less more head tail tail grep...)

#------cat
# cp /etc/passwd ./pass
# cat pass      #Normal file viewing mode
# cat -n pass   #-n shows how many lines there are in the file
# cat -A pass   #Check the special symbols of the file, such as the tab key in the file
# tac pass      #View files in reverse order
cat >> test2.txt <<EOF
ads
adf
EOF

#------less,more
# less /etc/services    #Use the cursor to flip up and down, the space to turn the page, and q exit
# more /etc/services    #Use enter to scroll up and down, space to turn pages, q exit (percentage)

#------head
# head pass     #View the content of the header, the first ten lines by default
# head -n5 pass #View the first 5 lines and specify with - n
# ps aux | head -5 # Just look at the first five processes

#------tail
# tail pass  # View the end of the file. The default is 10 lines
# tail -20 /var/log/secure  # View the last 20 lines of the file
# tail -f /var/log/messages #-f dynamically view the changes at the end of the file
# tailf /var/log/messages   #View changes at the end of the file
# ps aux | tail -2

#------grep filter file content
# grep "^root" pass     #Match lines starting with root
# grep "bash$" pass     #Match lines ending in bash
# grep -i "ftp" pass    #Ignore case matching
# grep  -Ei "sync$|ftp" pass    #The matching file contains the end of sync or ftp string
# grep -n -A 2 "Failed" /var/log/secure #Match the Failed string in the / var/log/secure file and print its next 2 lines
# grep -n -B 2 "Failed" /var/log/secure #Match the Failed string in the / var/log/secure file and print its last 2 lines
# grep -n -C 2 "Failed" /var/log/secure #Match the Failed string in the / var/log/secure file and print its upper and lower lines

# Turn up, turn down
control+b: Turn down
control+f: Upturning

3. Document management: downloading files online (wget, curl), uploading and downloading files (rz, sz)

wget and curl download files online

yum install wget -y
#Option: - O: specify download address
# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

# crul: browse resources on the network and -o save them locally
#Options: - o: specify download address
# curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

rzsz upload and download files

# yum install lrzsz -y  #The command cannot be executed without installing the software

# rz            #Only files can be uploaded (no more than 4g)
# sz /path/file #Only files can be downloaded

4. File management: file or command search (locate, which, whereis, find)

File search

# locate /etc/sh       #Search all files starting with sh in the etc directory
# locate -i /etc/sh    #Search the etc directory for all files starting with sh, ignoring case

Command lookup

# which ls  #Find the absolute path of ls command

# whereis ls / / find the path of the command, help manual, etc
# whereis -b ls / / only the path of the command is displayed, only binary

# For some commands related to the kernel, which whereis cannot be queried. You need to use type query
# type -a ls        #View the absolute path of the command (including aliases)
# type -a for

5. File management: character processing commands (sort, uniq, cut, sed, awk, wc,)

Sort sort

In some cases, you need to sort the data corresponding to an unordered text file. At this time, you need to use sort.

sort [OPTION]... [FILE]...
# -r: Reverse order - n: sort by number - t: specify the separator (default space) - k: specify the column and characters (specify 1,1,3.1,3.3)

#1. First create a file and write unordered contents
[root@lqz ~]# cat >> file.txt <<EOF
b:3
c:2
a:4
e:5
d:1
f:11
EOF

#2. Use sort to sort the output content
[root@lqz ~]# sort file.txt
a:4
b:3
c:2
d:1
e:5
f:11

#The results are not sorted by numbers, but by letters.
#You can use - t to specify the separator and - k to specify the columns to be sorted.
# -t specifies the separator and - k specifies the column, sorted by the second column
[root@lqz ~]# sort -t ":" -k2 file.txt
d:1
f:11 #Why is the second line 11? Shouldn't it be in order?
c:2
b:3
a:4
e:5

#According to the sorting method, you can only see the first character. The first character of 11 is 1. Sorting according to characters is indeed smaller than 2. 
#If you want to sort by number, you need to use the - n parameter to sort by number.
[root@lqz ~]# sort -t ":" -n -k2 file.txt
d:1
c:2
b:3
a:4
e:5
f:11


#Test cases, download files http://fj.xuliangwei.com/public/ip.txt , sort the file
[root@lqz ~]# sort -t. -k3.1,3.1nr -k4.1,4.3nr ip.txt

uniq de duplication
If there are multiple identical lines in the file, and you want to delete duplicate lines and count the total number of occurrences of identical lines at the same time, you can use the uniq command to solve this problem (but it must be used with sort, and the adjacent lines can be de duplicated, so it must be used with sort).

uniq [OPTION]... [INPUT [OUTPUT]]
#Option: - c calculate duplicate rows

#1. Create a file Txt file:
[root@lqz ~]# cat >>file2.txt <<EOF
abc
123
abc
123
EOF
#2.uniq needs to be used together with sort. First use sort sorting to make repeated contents continuous
[root@lqz ~]# cat file.txt |sort
123
123
abc
abc
#3. Use uniq to remove adjacent duplicate rows
[root@lqz ~]# cat file.txt |sort|uniq
123
abc
#4.-c parameter can count the number of repetitions of each line in the file
[root@lqz ~]# cat file.txt |sort|uniq -c
      2 123
      2 abc

cut intercept field

cut OPTION... [FILE]...
#Options: - d specify the separator - F number, which column to take - f3,6 three columns and 6 columns - c take by character (spaces also count)
#echo "Im lqz, is QQ 306334678" >file.txt   
#Filter out the files lqz and 306334678

#Several ideas to realize the above problems
# cut -d " " -f2,5 file.txt
# cut -d " " -f2,5 file.txt |sed 's#,##g'
# sed 's#,# #g' file.txt | awk -F " " '{print $2 " " $5}'
# awk  '{print $2,$5}' file.txt |awk -F ',' '{print $1,$2}'
# awk -F  "[, ]" '{print $2,$6}' file.txt
# awk -F '[, ]+' '{print $2,$5}' file.txt

# awk fetch column
awk '{print $2,$5}' file.txt
# -F specifies the separator
awk '{print $2,$5}' file.txt | awk -F "," '{print $1,$2}'

# sed replace sed 's###g'
# sed 's###g 'fixed writing
# sed 's#,##g 'replace comma with empty
cut -d " " -f2,5 file.txt | sed 's#,##g'

wc statistical line number

wc [OPTION]... [FILE]...
#Options: - l display file lines - c Display File Bytes - w display file words

# wc -l /etc/fstab      #Count the number of lines in the / etc/fstab file
# wc -l /etc/services   #Count the line number of / etc/services file
# ls |wc -l             #Count the number of files and folders under the current path

#Extension method
# grep -n "." /etc/services  | tail -1
# awk '{print NR $0}' /etc/services | tail -1
# cat -n /etc/services  | tail -1

Exercise: analyze the following logs and count the number of visits to each domain name.

[root@student tmp]# cat web.log 
http://www.lqz.com/index.html
http://www.lqz.com/1.html
http://post.lqz.com/index.html
http://mp3.lqz.com/index.html
http://www.lqz.com/3.html
http://post.lqz.com/2.html

# awk -F '/' '{print $3}' web.log|sort -rn|uniq –c
# cut -d / -f3 web.log|sort -rn|uniq –c

Exercise: using awk to extract the IP address diagram of the system

5. System file attributes

When we use ls -l column to display all the files in the directory, they are usually displayed in long format. In fact, the long format display is the file details we see under Windows. We also call it file attribute, and the attributes of the whole file are divided into ten columns.

[root@lqz ~]# ls -l ks.cfg
[root@lqz ~]# ll -h ks.cfg
-rw-------. 1 root root 4434 May 30 13:58 ks.cfg

#
-rw-------. ①:The first character is the file type, and the others are permissions
1           ②:Number of hard links
root        ③:Which user does the file belong to
root        ④:Which group does the file belong to
4434        ⑤:file size
May30 13:58 ⑥⑦⑧:Time and date of last modification
ks.cfg      ⑨:File or directory name

6. System file type

1. Usually we use color or suffix name to distinguish file types, but it is often not very accurate, so we can use ls -l to display the attributes of a file in a long format and use the first character in the first column to further judge the specific type of the file.

[root@lqz ~]# ll -d /etc/hosts /tmp /bin/ls  /dev/sda /dev/tty1 /etc/grub2.cfg /dev/log /run/dmeventd-client
-rwxr-xr-x.  1 root root 117656 Jun 30  2016 /bin/ls
srw-rw-rw-.  1 root root      0 Jan 20 10:35 /dev/log
brw-rw----.  1 root disk   8, 0 Jan 20 10:36 /dev/sda
crw--w----.  1 root tty    4, 1 Jan 20 10:36 /dev/tty1
lrwxrwxrwx.  1 root root     22 Jan 13 11:31 /etc/grub2.cfg -> ../boot/grub2/grub.cfg
-rw-r--r--.  1 root root    199 Jan 20 11:03 /etc/hosts
prw-------.  1 root root      0 Jan 20 10:36 /run/dmeventd-client
drwxrwxrwt. 61 root root   8192 Jan 21 13:01 /tmp

#explain
-   #Ordinary files (text, binary, compressed, pictures, logs, etc.) 
d   #Catalog file
b   #Device file (block device) storage device hard disk / dev/sda1, /dev/sda2
c   #Device file (character device), terminal / dev/tty1, /dev/zero
s   #Socket file, interprocess communication (socket)
p   #Pipeline file
l   #Link file

2. However, in some cases, we can't judge the type of ls -l file through the type of ls -l file. For example, a file may be an ordinary file, a compressed file, or a command file. At this time, we need to use file to judge the type of this file more accurately.

[root@lqz ~]# file /etc/hosts
/etc/hosts: ASCII text

[root@lqz ~]# file /bin/ls
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=aa7ff68f13de25936a098016243ce57c3c982e06, stripped

[root@lqz ~]# file /dev/sda
/dev/sda: block special

[root@lqz ~]# file /dev/tty1
/dev/tty1: character special

[root@lqz ~]# file /etc/grub2.cfg
/etc/grub2.cfg: broken symbolic link to `../boot/grub2/grub.cfg'

[root@lqz ~]# file /home
/home: directory

[root@lqz ~]# file /run/dmeventd-client
/run/dmeventd-client: fifo (named pipe)

[root@lqz ~]# ll but
-rw-r--r-- 1 root root 42125 Apr  1 12:26 but
[root@lqz ~]# file but
but: Zip archive data, at least v1.0 to extract

PS: the Linux file extension does not mean anything, just for us to better identify the type of the file.

7. System link file

Files have file names and data, which are divided into two parts on Linux: user data and metadata.
User data, that is, file data Block. Data Block is the place where the real content of the file is recorded. We call it Block
Metadata refers to the additional attributes of the file, such as file size, creation time, owner and other information. We call it Inode
In Linux, inode is a part of file metadata, but it does not contain the file name. Inode number is the index node number)
The file name is only for the convenience of people's memory and use. The system or program looks for the correct file data block through inode number. Figure 1 It shows the process that the program obtains the file content through the file name.

Linux system introduces two kinds of links: hard link and soft link (also known as symbolic link, i.e. soft link or symbolic link)

ll -i view inode

1. What is soft link

Soft link is equivalent to the shortcut of Windows. The soft link file will point the inode to the block of the source file. When we access this soft link file, we actually access the source file itself. So when we create multiple soft links to a file, in fact, multiple inodes point to the same block. When we delete a soft link file, we only delete an inode point, which will not affect the source file. However, if we delete the source file, all soft link files will become invalid.

2. What is hard link

If an inode number corresponds to multiple file names, these files are called hard links. In other words, hard link means that the same file uses multiple aliases. As shown in the figure below, hard link is an alias of file, and they have a common inode.

3. Soft link practice

#File soft link example
touch /root/file
ln -s /root/file /tmp/file_bak
ll /tmp/file_bak  //File under root is linked to / tmp and renamed as file_bak
  

#Directory soft link example
mkdir /soft/nginx1.1 -p
ln -s /soft/nginx1.1/ /soft/nginx
ll /soft/nginx   //View link points

#Soft link usage scenario
1.Software upgrade
2.Enterprise code release 
3.Inconvenient directory movement

4. Hard link practice

#File hard link example
ln  /root/file /tmp/file_hard
ll /tmp/file_hard

#PS: hard links cannot be created in directories. Hard link files can be deleted with rm command

5. The difference between hard link and soft link
1) The LN command creates a hard link and the ln -s command creates a soft link.
2) Directories cannot create hard links, and hard links cannot cross partition systems.
3) Directory soft links are particularly common, and soft links support cross partition systems.
4) The hard link file is the same as the inode of the source file, and the soft link file is different from the inode of the source file.
5) Deleting soft link files has no impact on the source files and hard link files.
6) Deleting the hard link file of the file has no impact on the source file and the linked file.
7) Deleting the source file of the linked file has no effect on the hard link and will lead to the failure of the soft link.
8) Delete the source file and its hard link file, and the whole file will be truly deleted.

Keywords: Linux shell

Added by codeDV on Thu, 10 Feb 2022 23:43:59 +0200