Author: Panda

Transferred from: http://siye1982.github.io/2016/02/25/linux-list/

Although most of my work is Java related development, I am exposed to Linux every day, especially after using Mac. I work in the command line environment with black background every day I don't have a good memory. I can't remember many useful linux commands very well. Now I'll summarize them gradually for later viewing.

basic operation

Linux shutdown, restart

#Shut down
shutdown -h now

#Restart
shutdown -r now

View system and CPU Information

#View system kernel information
uname -a

#View system kernel version
cat /proc/version

#View current user environment variables
env

cat /proc/cpuinfo

#See how many logical CPUs are available, including cpu models
cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c

#Check how many CPUs are there, and how many cores are each
cat /proc/cpuinfo | grep physical | uniq -c

#Check whether the current CPU is running in 32bit or 64bit mode. If it is running in 32bit mode, it does not mean that the CPU does not support 64bit
getconf LONG_BIT

#The result is greater than 0, indicating that 64 bit calculation is supported lm refers to long # mode, while supporting lm is 64bit
cat /proc/cpuinfo | grep flags | grep ' lm ' | wc -l

Establish soft connection

ln -s /usr/local/jdk1.8/ jdk

rpm correlation

#Check whether the software is installed through rpm
rpm -qa | grep Software name

sshkey

#Create sshkey
ssh-keygen -t rsa -C your_email@example.com

#id_ rsa. Copy the contents of pub to the "home / username /. Of the server to be controlled ssh/authorized_ Keys ， if not, create a new one (. SSH permission is 700, authorized_keys permission is 600)

Command rename

#In the of each user bash_ Add rename configuration in profile
alias ll='ls -alF'

Synchronize server time

sudo ntpdate -u ntp.api.bz

Background run command

#Run in the background and have nohup Out output
nohup xxx &

#Run in the background without outputting any logs
nohup xxx > /dev/null &

#Run in the background and output the error information to the log as a standard
nohup xxx >out.log 2>&1 &

Force active user exit

#Command to complete the force active user exit Where TTY represents the terminal name
pkill -kill -t [TTY]

View command path

which <command>

View the maximum fd number of all open processes

ulimit -n

Configure dns

vim /etc/resolv.conf

nslookup to view the domain name routing table

nslookup google.com

last, list of recent login information

#5 recently logged in accounts
last -n 5

Set fixed ip

ifconfig em1  192.168.5.177 netmask 255.255.255.0

View environment variables loaded in process

#You can also go to the cd / proc directory to view the things loaded in the process memory
ps eww -p  XXXXX(Process number)

View the process tree to find the server process

ps auwxf

View process startup path

cd /proc/xxx(Process number)
ls -all
#cwd corresponds to the startup path

Add users and configure sudo permissions

#New user
useradd user name
passwd user name

#Add sudo permission
vim /etc/sudoers
#Modification of documents inside
# root    ALL=(ALL)       ALL
#User name: ALL=(ALL) ALL

Force the shutdown of all processes whose process name contains xxx

ps aux|grep xxx | grep -v grep | awk '{print $2}' | xargs kill -9

Disk, file and directory related operations

vim operation

#In normal mode, g represents the global, x represents the searched content, and y represents the replaced content
:%s/x/y/g

#In normal mode
0  #Move the cursor to the beginning of the line (number 0)
$  #Move cursor to end of line
shift + g #Skip to the end of the file
gg #Jump to file header

#Display line number
:set nu

#Remove line number
:set nonu

#Search
/xxx(Search content)  #Search from the beginning and press n to find the next one
?xxx(Search content)  #Retrieve from tail

Open a read-only file and save it after modification (you can save it without switching users)

#In normal mode
:w !sudo tee %

View basic information of disk, file and directory

#View disk mounting
mount

#View disk partition information
df

#View directory and subdirectory size
du -H -h

#Check the space occupied by each file and folder in the current directory without recursion
du -sh *

wc command

#See how many lines there are in the file
wc -l filename

#See how many word s there are in the file
wc -w filename

#What is the longest line in the file
wc -L filename

#Count bytes
wc -c

Common compression and decompression commands

Compression command

tar czvf xxx.tar Compressed directory

zip -r xxx.zip Compressed directory

Decompression command

tar zxvf xxx.tar

#Extract to the specified folder
tar zxvf xxx.tar -C /xxx/yyy/

unzip xxx.zip

Change the user and user group to which the file belongs

chown eagleye.eagleye xxx.log

cp, scp, mkdir

#copy
cp xxx.log

#Copy and force overwrite of files with the same name
cp -f xxx.log

#Copy folder
cp -r xxx(Source folder) yyy(Destination folder)

#Remote replication
scp -P ssh port username@10.10.10.101:/home/username/xxx /home/xxx

#Cascade create directory
mkdir -p /xxx/yyy/zzz

#When creating folders in batch, java and resources folders will be created under test and main
mkdir -p src/{test,main}/{java,resources}

Compare two files

diff -u 1.txt 2.txt

The number of bytes of log output, which can be used as a performance test

#If you do a performance test, you can output "." to the log every time you execute it In this way, the number of bytes in the log is the actual number of performance test runs, and the real-time rate can be seen
tail -f xxx.log | pv -bt

View, remove special characters

#View special characters
cat -v xxx.sh

#Remove special characters
sed -i 's/^M//g’ env.sh ， remove the special characters of the file, such as ^ M: you need to enter: ctrl+v+enter

Handle the problem of special characters in files caused by system reasons

#It can be converted to the file format under the system
cat file.sh > file.sh_bak

#First set file Copy the contents of the file in SH, then run it, then paste the contents, and finally ctrl + d # save and exit
cat > file1.sh

#In vim, set the file code and file format as follows
:set fileencodings=utf-8 ，then w (Save it) and it can be converted into utf8 Format,
:set fileformat=unix

#Using dos2unix to format files under mac
find . -name "*.sh" | xargs dos2unix

tee, output to the screen while redirecting

awk '{print $0}' xxx.log | tee test.log

Search correlation

Grep (learn a Linux command every day (5): grep)

#Reverse matching to find content that does not contain xxx
grep -v xxx

#Exclude all blank lines
grep -v '^/pre>

#If the result {2 is returned, the second line is empty
grep -n "^$" 111.txt    

#Query rows starting with abc
grep -n "^abc" 111.txt 

#Also list the lines on which the word appears in the article
grep 'xxx' -n xxx.log

#Count the number of occurrences of the string
grep 'xxx' -c xxx.log

#When comparing, do not care about the difference between case and case
grep 'xxx' -i xxx.log

Awk (one Linux command per day (4): awk)

#Take ':' as the separator. If there is a user in the fifth field, this line will be output
awk -F ':' '{if ($5 ~ /user/) print $0}' /etc/passwd 

#Count the number of occurrences of a character (string) (Chinese is invalid) in a single file
awk -v RS='character' 'END {print --NR}' xxx.txt

Find search command (learn a Linux command every day (20): find)

#Find the suffix in the directory mysql files
find /home/eagleye -name '*.mysql' -print

#Start from the / usr directory to find the files that have been fetched from the memory in the last three days.
find /usr -atime 3 –print

#I will start from the / usr directory to find the files that have been modified in the last 5 days.
find /usr -ctime 5 –print

#I will start from the / doc directory to find the file of jacky whose file name starts with j.   
find /doc -user jacky -name 'j*' –print

#Start from the / doc directory and look for files with file names beginning with ja or ma.
find /doc \( -name 'ja*' -o- -name 'ma*' \) –print

#You will start from the / doc directory to find the file with bak at the end of the file name and delete it- The exec option means to execute, rm means to delete the command, {} means the file name, "\;" Is the end of the specified command.  
find /doc -name '*bak' -exec rm {} \;

Network related

See what processes use this port

lsof -i:por

Get local ip address

/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:"

iptables

#View iptables status
service iptables status

#To block an ip
iptables -I INPUT -s ***.***.***.*** -j DROP

#To unseal an IP, use the following command:
iptables -D INPUT -s ***.***.***.*** -j DROP

remarks: parameter-I Yes, it means Insert(Add),-D express Delete(Delete). Followed by rules, INPUT Indicates inbound,***.***.***.***It means to shut down IP，DROP Indicates abandoning the connection.

#Open access to port 9090
/sbin/iptables -I INPUT -p tcp --dport 9090 -j ACCEPT 

#Firewall on, off and restart
/etc/init.d/iptables status
/etc/init.d/iptables start
/etc/init.d/iptables stop
/etc/init.d/iptables restart

nc command, tcp debugging tool

#When sending a TCP request to an endpoint, the content of data is sent to the opposite end
nc 192.168.0.11 8000 < data.txt

#nc can be used as a server to listen to a port number and store the contents of a request in received_ In data
nc -l 8000 > received_data

#The upper side only listens once. If it listens for multiple times, the - k parameter can be added
nc -lk 8000

Tcpdump (learn a Linux command every day (72): tcpdump)

#dump the tcp packet of the local port 12301
tcpdump -i em1 tcp port 12301 -s 1500 -w abc.pcap

Track network routing path

#traceroute uses udp mode by default. If it is - I, it will be changed to icmp mode
traceroute -I www.163.com

#Track from ttl 3rd hop
traceroute -M 3 www.163.com  

#Plus port tracking
traceroute -p 8080 192.168.10.11ss
#Displays all ports opened locally
ss -l 

#Displays the specific open socket of each process
ss -pl 

#Display all tcp} socket s
ss -t -a 

#Show all UDP # Socekt
ss -u -a 

#Displays all established SMTP connections
ss -o state established '( dport = :smtp or sport = :smtp )'  

#Displays all established HTTP connections
ss -o state established '( dport = :http or sport = :http )'  

#Find all processes connected to the X server
ss -x src /tmp/.X11-unix/*  

#List current socket statistics
ss -s 

Explanation: netstat Is traversal/proc Each of the following PID catalogue ss Direct reading/proc/net The following statistics. therefore ss It consumes more resources and time than netstat Much less

netstat

#Output the number of connections per ip and the total number of connections in each state
netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":");if(n<=2)++S[array[(1)]];else++S[array[(4)]];++s[$NF];++N} END {for(a in S){printf("%-20s %s\n", a, S[a]);++I}printf("%-20s %s\n","TOTAL_IP",I);for(a in s) printf("%-20s %s\n",a, s[a]);printf("%-20s %s\n","TOTAL_LINK",N);}'

#Count all connection status
# CLOSED: no connection is active or in progress
# LISTEN: the server is waiting for an incoming call
# SYN_RECV: a connection request has arrived, waiting for confirmation
# SYN_SENT: the application has started. Open a connection
# ESTABLISHED: normal data transmission status
# FIN_WAIT1: the application says it's finished
# FIN_WAIT2: the other side has agreed to release
# ITMED_WAIT: wait for all packets to die
# CLOSING: both sides try to close at the same time
# TIME_WAIT: the state of actively closing one end of the connection before waiting for feedback from the other end
# LAST_ACK: wait for all packets to die
netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'

#Find more time_wait connection
netstat -n|grep TIME_WAIT|awk '{print $5}'|sort|uniq -c|sort -rn|head -n20

Monitoring linux performance commands

top

Press the uppercase F or O key, and then press a-z to sort the processes according to the corresponding columns, and then enter. The uppercase R key can reverse the current sort. Learn a Linux command every day (48): top

PID process id
PPID Parent process id
RUSER Real user name
UID User of the process owner id
USER User name of the process owner
GROUP Group name of the process owner
TTY The name of the terminal that started the process. Processes that are not started from the terminal are displayed as ?
PR priority
NI nice Value. Negative values indicate high priority and positive values indicate low priority
P Last used CPU，Only in many CPU Meaningful in the environment
%CPU From last update to now CPU Time occupancy percentage
TIME Used by the process CPU Total time in seconds
TIME+ Used by the process CPU Total time, unit 1/100 second
%MEM Percentage of physical memory used by the process
VIRT Total amount of virtual memory used by the process, in kb. VIRT=SWAP+RES
SWAP The size, unit, of the virtual memory used by the process kb. 
RES The size of physical memory used by the process and not swapped out, unit: kb. RES=CODE+DATA
CODE Physical memory occupied by executable code, unit kb
DATA Parts other than executable code(Data segment+Stack)Size of physical memory occupied, unit kb
SHR Shared memory size in kb
nFLT Number of page errors
nDRT The number of pages that have been modified since the last write.
S Process status. D=A state of uninterrupted sleep,R=function,S=sleep,T=track/stop it,Z=Zombie process
COMMAND Command name/command line
WCHAN If the process is sleeping, the system function name in sleep is displayed
Flags Task flag, reference sched.h

dmesg, view the system log

dmesg

iostat, disk IO condition monitoring

iostat -xz 1

# r/s, w/s, rkB/s, wkB/s: respectively represents the number of reads and writes per second and the amount of data read and written per second (kilobytes). Excessive reading and writing may cause performance problems.
# await: average wait time of IO operation, in milliseconds. This is the time consumed when the application interacts with the disk, including IO waiting and actual operation time. If this value is too large, the hardware device may encounter a bottleneck or failure.
# avgqu-sz: the average number of requests sent to the device. If this value is greater than 1, the hardware device may be saturated (some front-end hardware devices support parallel writing).
#Utilization rate of equipment: util%. This value indicates the busy degree of the equipment. The empirical value is that if it exceeds 60, the IO performance may be affected (refer to the average waiting time of IO operation). If it reaches 100%, the hardware device has been saturated.
#If the data of logical devices is displayed, the device utilization does not mean that the actual hardware devices at the back end have been saturated. It is worth noting that even if the IO performance is not ideal, it does not necessarily mean that the application performance will be poor. Strategies such as pre read and write cache can be used to improve the application performance.

free, memory usage

free -m

eg:

     total       used       free     shared    buffers     cached
Mem:          1002        769        232          0         62        421
-/+ buffers/cache:          286        715
Swap:          1153          0       1153

Part I Mem line:

total Total memory: 1002M
used Memory used: 769M
free Free memory: 232M
shared It has been abandoned,Always 0
buffers Buffer Cache memory: 62M
cached Page Cache memory:421M

Relationship: total(1002M) = used(769M) + free(232M)

Part 2 (- / + buffers/cache):

(-buffers/cache) used Number of memory: 286 M (Refers to the first part of the Mem In line used – buffers – cached)
(+buffers/cache) free Number of memory: 715M (Refers to the first part of the Mem In line free + buffers + cached)

It can be seen that - buffers/cache reflects the memory actually eaten by the program, while + buffers/cache reflects the total amount of memory that can be misappropriated

The third part refers to the switching partition

sar to view the network throughput status

# Here you can view the throughput of network devices with the sar command. When troubleshooting performance problems, you can judge whether the network equipment is saturated by the throughput of the network equipment.

sar -n DEV 1

#
# The sar command is used here to view the TCP connection status, including:
# active/s: the number of locally initiated TCP connections per second, i.e. TCP connections created through connect call;
# passive/s: the number of remote initiated TCP connections per second, that is, the TCP connections created through the accept call;
# retrans/s: number of TCP retransmissions per second;
# The number of TCP connections can be used to determine whether the performance problem is due to the establishment of too many connections, and further determine whether the connection is actively initiated or passively accepted. TCP retransmission may be caused by poor network environment or excessive server pressure, resulting in packet loss
sar -n TCP,ETCP 1

vmstat, monitor CPU utilization, memory usage, virtual memory interaction, IO read and write at a given time

#2 means to collect status information every 2 seconds, and 1 means to collect only once (ignore or collect all the time)

vmstat 2 1

eg:
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 3499840 315836 3819660 0 0 0 1 2 0 0 0 100 0
0 0 0 3499584 315836 3819660 0 0 0 0 88 158 0 0 100 0
0 0 0 3499708 315836 3819660 0 0 0 2 86 162 0 0 100 0
0 0 0 3499708 315836 3819660 0 0 0 10 81 151 0 0 100 0
1 0 0 3499732 315836 3819660 0 0 0 2 83 154 0 0 100 0

• r indicates the running queue (that is, how many processes are actually allocated to the CPU). At present, the CPU of the server I tested is relatively idle, and there are no programs running. When this value exceeds the number of CPUs, there will be a CPU bottleneck. This is also related to the load of the top. Generally, the load is higher when it exceeds 3, higher when it exceeds 5, and abnormal when it exceeds 10. The state of the server is very dangerous. The load of top is similar to the running queue per second. If the running queue is too large, it indicates that your CPU is very busy, which generally leads to high CPU utilization.

• b represents a blocked process. I won't say much about it. Process blocking, you know.

• If the size of swpd virtual memory used is greater than 0, it indicates that your machine is out of physical memory. If it is not the cause of program memory leakage, it is time to upgrade memory or migrate memory consuming tasks to other machines.

• Free is the size of free physical memory. My machine memory is 8G in total, with 3415M remaining.

• buff Linux/Unix system is used to store the cache of contents, permissions, etc. in the directory. My local machine takes about more than 300 M

• cache cache is directly used to remember the files we open and buffer the files. My machine occupies about more than 300 m (here is the wisdom of Linux/Unix. Using part of the free physical memory as the cache of files and directories is to improve the performance of program execution. When the program uses memory, buffer/cached will be used quickly.)

• The amount of virtual memory that si reads from the disk every second. If this value is greater than 0, it means that the physical memory is not enough or the memory is leaked. Find the memory consuming process and solve it. My machine has plenty of memory and everything is normal.

• so the size of virtual memory written to disk per second. If this value is greater than 0, the same as above.

• The number of blocks received by the bi block device per second. The block device here refers to all disks and other block devices on the system. The default block size is 1024byte. There is no IO operation on my machine, so it has always been 0. However, I have seen it on the machine that processes and copies a large amount of data (2-3T). It can reach 140000/s, and the disk write speed is almost 140M per second

• bo block the number of blocks sent by the device per second. For example, when we read a file, bo must be greater than 0. bi and bo are generally close to 0, or IO is too frequent and needs to be adjusted.

• in the number of CPU interrupts per second, including time interrupts

• cs the number of context switches per second. For example, when we call a system function, we need to switch the context, thread and process context. The smaller the value, the better. If it is too large, we should consider reducing the number of threads or processes, such as in web servers such as apache and nginx, When we do performance test, we usually conduct thousands or even tens of thousands of concurrent tests. The process of selecting Web server can be lowered by the peak value of process or thread until cs reaches a relatively small value, and the number of processes and threads is a more appropriate value. System call is the same. Every time we call the system function, our code will enter the kernel space, resulting in context switching. This is very resource consuming. We should also try to avoid calling the system function frequently. Too many context switches means that most of your CPU is wasted on context switching, resulting in less time for the CPU to do serious things, and it is not advisable to make full use of the CPU.

• The CPU time of us users. I used to work on a server that encrypts and decrypts frequently. I can see that us is close to 100 and the R running queue reaches 80 (the machine is doing stress testing and its performance is poor).

• sy system CPU time. If it is too high, it indicates that the system call time is long, such as frequent IO operations.

• ID idle CPU time. Generally speaking, id + us + sy = 100. Generally, I think id is the idle CPU utilization rate, us is the user CPU utilization rate, and sy is the system CPU utilization rate.

• wt wait IO CPU time.