File system and log analysis

summary

inode and block

Hard link and soft link

Recover accidentally deleted files

Analysis log file

inode and block overview

• File data includes meta information and actual data
• Files are stored on the hard disk. The minimum storage unit of the hard disk is "sector". Each sector stores 512 bytes
• Block
  1) A block(4k) composed of 8 consecutive sectors
  2) Is the smallest unit of file access
• Inode (index node)
  1) The Chinese translation is "index node", also known as i node
  2) Used to store file source information

1, File system

Firstly, files must be stored on the hard disk, but in the linux system, files and hard disk are subdivided into other concepts. Let's start to understand these in detail.

1. sector

• Sector is the smallest storage unit of hard disk. Each sector can store 512 bytes

Fast (block)

• When the operating system reads the hard disk, it continuously reads multiple sectors at one time and reads one or more fast sectors at one time,
• A fast block is generally composed of eight consecutive sectors, so a block is 4k in size, which is the smallest unit of file access.

3. File data

• File data includes actual data and meta information (such as file creation, creation date, file size, file permission, etc.).
• The file data is stored in the "block", and the area where the file meta information is stored is called inode

4. Inode (inode or i-node)

• A file must occupy one inode
• inode does not contain a file name. The file name is stored in the directory. Everything in the linux system is a file, so the directory is also a file.
• Each inod has a number, and the operating system uses the inode number to identify different files. The linux system does not use the file name, but uses the inode number to identify files. For the system, the file name is only a nickname for the inode number, which is easy to identify. The file name and inode number are corresponding, and each inode number corresponds to a file name.

5. Summary in related concepts

Steps for users to access files (in the system)
Step 1: the user accesses a file in the linux system
Part II: the system searches the corresponding inode number according to the file name
Part III: get inode information through inode number
Part IV: according to the inode information, check whether the user has access to the file. If so, point to the corresponding data block and read the data

inode contains meta information of the file

• File name not included
• Bytes of file
• User ID of the file owner (excluding the file name)
• Group ID of the file
• Read, write and execute permissions of files
• Timestamp of the file

Use the stat command to view the inodex information of a file

Example:

stat aa.txt

inode size

Inodes also consume hard disk space, so when formatting, the operating system automatically divides the hard disk into two areas. One is the database area to store file data, and the other is the inode area to store the information contained in the inode. The size of each inode is generally 128 bytes or 256 bytes.

• The total number of inode s is determined when the file system is formatted
• The df - i command can view the total number of inodes corresponding to each hard disk partition and the number of inodes that have been used.

3. Query of inode number

View file name correspondence  inode    There are two ways to number
ls  -i        file name     #Poor ordinary file
stat          file name     # Check common files and directories (recommended)

4. Special role of inode

Because the inode number and file name are different, the linux system has the following unique phenomena

• The file name contains special characters and may not be deleted normally. In this case, deleting inode directly can delete the file
• Moving a file or renaming a file only changes the file name without affecting the inode number
• After opening a folder, the system can use the inode number to identify the song file, regardless of the file name.
• After the file data is modified, a new inode number will be generated

5. Delete files by inode number

1,find ./ -inum 52305140 -exec rm  -i   { } \;
2,find ./ -inum 50464299 -delete

Three, analog inode node exhaustion fault handling

Summary of steps
1,use fdisk Create partition/dev/sdb1，Partition size 30 M that will do
fdisk /dev/sdb   n→p→enter→+30M→w  #Create partition
mkfs.ext4 /dev/sdb1   #For centos 7 system, the file type of node exhaustion fault handling can be ext3 or ext4
mkdir /test
mount /dev/sdb1 /mnt
df -i
2,simulation inode Node exhaustion fault
for ((i=1; i<=7680; i++));do touch /test/file$i;done  
perhaps   touch {1..7680}.txt
df -i
df -hT
3,Delete file recovery
rm -rf /test/*
df -i
df -hT

Illustration:
1. Create the partition / dev/sdb1 using fdisk and mount it

2. Analog inode point depletion fault

Summary: each file corresponds to an inode number. When the inode number is used up, even if there is still a lot of space on the disk, the file cannot be created.

Recover deleted files (ext format)

• extundelete is an open source Linux Data Recovery tool that supports ext3 and ext4 file systems. (ext4 can only be recovered in CentOS 6 version)

Operation steps
1,use fdisk Create partition/dev/sdb1，format ext3 file system
fdisk /dev/sdb  
mkfs.ext3 /dev/sdb1
mkdir /test
mount /dev/sdb1 /test
df -hT

2,Install dependent packages
yum -y install e2fsprogs-devel e2fsprogs-libs

3,Compile and install extundelete
cd /test  Switch to test In the directory
wget http://Nchc.dl.sourceforge.net/project/extend/extend/0.2.4/extend-0.2.4.tar.bz2 # network download installation package
tar jxvf extundelete-0.2.4.tar.bz2   #Unzip the tar package
cd extundelete-0.2.4/     #Switch to the extracted directory
./configure --prefix=/usr/local/extundelete && make && make install  #Specify the installation directory to start the installation   
ln -s /usr/local/extundelete/bin/* /usr/bin/    #Create a soft connection for the system to recognize the command

4,Simulate deletion and restore
cd /test
echo a>a
echo a>b
echo a>c
echo a>d
ls
extundelete /dev/sdb1 --inode 2			#Check which files exist in the file system / dev/sdb1. The i node starts from 2, and 2 represents the initial directory of the file system.

rm -rf a b
extundelete /dev/sdb1 --inode 2	
cd ~
umount /test
extundelete /dev/sdb1 --restore-all		#Recover everything under the / dev/sdb1 file system
#A RECOVERED_FILES / directory will appear in the current directory, which saves the recovered files
ls RECOVERED_FILES/

Illustration:
1. I directly use the previous sdb1, unmount it first, and then reformat it.



2. Install dependent packages using yum or up2date

3. Download the installation package and install

There is a problem with this picture. The service is not installed ##. / configure -- prefix = / usr / local / extendelete & & make & & make install

4. Simulate deletion and restore

5, xfs type file backup and recovery

• By default, the CentOS 7 system uses xfs files. xfs files can be backed up and restored using xfsdump and xfsrestore tools.
• There are two backup levels of xfsdump: 0 for full backup and 1-9 for incremental backup. The default backup level of xfsdump is 0.

xfsdump The command format is:
xfsdump -f Backup storage location path or device file to be backed up

xfsdump Use restrictions:
1.Only mounted file systems can be backed up
2.Must use root You can only operate with your permission
3.Only backup XFS file system
4.The data after backup can only make xfsrestore analysis
5.You cannot back up two with the same UUID File system (available) blkid Command view)

Common options for commands	effect
-f	Specify backup file directory
-L	Specify the label session label
M	Specify device label media label
-s	Backup a single file, - s cannot be directly followed by a path

Detailed steps

1,use fdisk Create partition/dev/sdb1，format xfs file system
fdisk /dev/sdb
partprobe /dev/sdb   #Reread the partition table and use it when the disk query is not available
mkfs.xfs [-f] /dev/sdb1
mkdir /data
mount /dev/sdb1 /data/
cd /data
cp /etc/passwd ./
mkdir test
touch test/a

2,use xfsdump Command to back up the entire partition
rpm -qa | grep xfsdump
yum install -y xfsdump
xfsdump -f /opt/dump_sdb1 /dev/sdb1 [-L dump_sdb1 -M sdb1]

3,Analog data loss and use xfsrestore Command restore file
cd /data/
rm -rf ./*
ls

xfsrestore -f /opt/dump_sdb1 /data/

1. Illustration steps





2. Back up files, delete and restore them

6, System log

1. Log file
1) Log function

• It is used to record various events during the operation of the system and program
• By reading the log, it is helpful to diagnose and solve system faults

2) Classification of log files

• Kernel and system log
  1) It is managed uniformly by the system service rsyslog, and the log format is basically similar
  2) Main configuration file / etc/rsyslog.conf
• User log
  1) Record relevant information of system user login and logout
  2) Main configuration file / var/log/secure
• Program log
  1) Log files independently managed by various applications have different recording formats

3) Default save location
Log files are placed in the directory / var/log / by default.

2. Log file analysis

1) Kernel and system log
It is uniformly managed by the system service rsyslog

• Software package: rsyslog-7.4.7-16.el7.x86_64,
• Main program: / sbin/rsyslogd
• Configuration file: / etc/rsyslog.cont

2) Log message priority
Priority of Linux system kernel log messages (the lower the number level, the higher the priority, and the more important the message):
Level number priority description
0 EMERG can cause the host system to become unavailable.
1 ALERT: problems that must be solved immediately.
2. Crit (serious) is a serious situation.
3 ERR (error) operation error.
4 WARNING is an important event that may affect the system function and needs to be reminded to the user.
5 note is an event that does not affect normal functions but requires attention.
6 INFO general information.
7 DEBUG debug

3) View the rsyslog.conf configuration file

vim /etc/rsyslog.conf		#View the rsyslog.conf configuration file
*.info;mail.none;authpriv.none;cron.none         /var/log/messages

*.info		#Information indicating all levels above info level is written to the corresponding log file
mail.none	#Indicates that the information of an event is not written to the log file (e.g. mail here)

4) Format of log

#Record format of public log / var/log/messages file
 Timestamp: the date and time when the message was sent.
Host name: the name of the computer that generated the message.
Subsystem Name: the name of the application that sent the message.
Message: the specific content of the message.

5) User log

• Record relevant information of system user login and logout
  /var/log/secure: Records security event information related to user authentication.
  /var/log/lastlog: record the latest login events of each user. Binary format
  /var/log/wtmp: record the login, logoff, system startup and shutdown events of each user. Binary format
  /var/run/btmp: log failed and incorrect login attempts and authentication events. Binary format
• Analysis tools
  users, who,w ,last,lastb
  The last command is used to query the user records that have successfully logged in to the system
  The lastb command is used to query the user record of login failure

6) Program log

• Log files independently managed by various applications have different recording formats

    1,Web Services: Nar/log/httpd/
             1) access_log //Record customer access events   
             2) error_log //Log error events
  

 2,Agency services:/var/log/squid/
                  1)access.log,cache.log

• Analysis tools
  1) Text viewing, grep filtering and retrieval, and viewing in Webmin Management Suite
  2) awk, sed and other text filtering, formatting and editing tools
  3) Webalizer, Awstats and other special log analysis tools

7) Summary of common logs

Some common log files:
#Kernel and public message logs:
/var/log/messages: record Linux Kernel messages and public log information of various applications, including startup IO Error, network error, program failure, etc. For applications or services that do not use a separate log file, you can generally obtain relevant event recording information from the log file.

#Scheduled task log:
/var/log/cron: record crond Event information generated by the scheduled task.

#System boot log:
/var/log/dmesg: record Linux Various event information during system boot.

#Mail system log:
/var/log/maillog: Record email activity entering or sending out the system.

#User login log:
/var/log/secure: Record security event information related to user authentication.
/var/log/lastlog: Record the latest login events of each user. Binary format
/var/log/wtmp: Record the login, logout, system startup and shutdown events of each user. Binary format
/var/run/btmp: Log failed and incorrect login attempts and authentication events. Binary format

3. Log management

• Timely backup and archive

• Extend log retention

• Control log access
  1) The log may contain various sensitive information, such as account, password, etc

• Centralized management log
  *Send the log files of the server to the unified log file server
  *It is convenient for the unified collection, sorting and analysis of log information
  *Prevent accidental loss, malicious tampering or deletion of log information