Linux environment intrusion emergency and troubleshooting

1, Account security

1. User information file / etc/passwd

# Format: account:password:UID:GID:GECOS:directory:shell
# User name: Password: user ID: group ID: user description: Home Directory: shell after login
# View logged in users:
cat /etc/passwd | grep /bin/bash
# View users with UID=0
awk -F: '$3==0{print $1}' /etc/passwd
# Users who view sudo permissions
more /etc/sudoers | grep -v "^#\|^$" | grep "ALL=(ALL)"

Note: only local login is allowed without password, and remote login is not allowed
2. Shadow file: / etc/shadow

# User name: encryption password: date of last password modification: time interval between two password modifications: password validity: warning days until password modification expires: Grace days after password Expiration: account expiration time:
# Reserved root: $6 $ogs1pqhl2p3zetre $x7o7bzoouhqvsemsgsyn5ud4 kMHx6qgbTqwNVC5oOAouXvcjQSt. Ft7ql1WpkopY0UV9ajBwUt1DpYxTCVvI/:16809:0:99999:7:::

3. View the current login user and login duration

who     # View all users currently logged in to the system (tty local login, pts remote login)
w       # Displays the users who have logged in to the system and the instructions being executed
uptime  # Check the login time, number of users and load status

4. Check user login information
View the users and information who have successfully logged in recently

# Displaying logged in indicates that the user is still logged in
# pts indicates remote login from SSH
# tty means log in from the console, that is, log in next to the server
# last

To view the users and information who failed to log in recently:

# SSH means logging in remotely from SSH
# tty means logging into sudo lastb from the console

Display the latest login information of all users:


When checking the server, the hacker is not online. You can use the last command to check when the hacker logs in. Some hackers will delete or empty the / var/log/wtmp file when logging in, so we can't use the last command to obtain useful information.
Before hacking, you must use chatr + A to lock the / var/log/wtmp file to avoid being deleted by hackers
5. sudo user list


2, Intrusion detection:

# Query privileged user (uid is 0):
awk -F: '$3==0{print $1}' /etc/passwd
# Query the account information of remote login:
awk '/\$1|\$6/{print $1}' /etc/shadow
# Whether other accounts except root account have sudo permission. If it is not necessary for management, sudo permission should be deleted for ordinary account:
more /etc/sudoers | grep -v "^#\|^$" | grep "ALL=(ALL)" 
# Disable or delete redundant and suspicious accounts
usermod -L user    # Disable the account, the account cannot log in, and the second column of / etc/shadow is! start
userdel user       # Delete user
userdel -r user    # The user user will be deleted and the user directory under the / home directory will be deleted together

Pass bash_history file to view the system commands executed by the account:
Open / home under each account directory bash_history to view the historical commands executed by ordinary accounts.
Add login IP address, command execution time and other information for historical commands:

# 1. Save 10000 commands:
sed -i 's/^HISTSIZE=1000/HISTSIZE=10000/g' /etc/profile 
# 2. Add the following line number configuration information at the end of the / etc/profile file:
USER_IP=`who -u am i 2>/dev/null | awk '{print $NF}' | sed -e 's/[()]//g'`
if [ "$USER_IP" = "" ]
export HISTTIMEFORMAT="%F %T $USER_IP `whoami` "
shopt -s histappend
export PROMPT_COMMAND="history -a"
# 3. Make configuration effective
source /etc/profile

Note: clear history operation command: history -c
This operation does not clear the records saved in the file, so it needs to be deleted manually bash_ Records in profile file

3, Check the port connection:

netstat -antlp | more

Use the ps command to analyze the process and get the corresponding pid number:

ps aux | grep 6666 

View the process file path corresponding to pid:

# $pid is the corresponding pid number ls -l /proc/$PID/exe or file /proc/$PID/exe
 Analysis process:
# Check the process lsof -p 6071 according to the pid number
# View the file lsof -c sshd opened by the process through the service name
# Viewing process by port number: lsof - I: 22

View the start time of the process:

ps -p 6071 -o lstart

Forcibly stop the process according to pid:

kill -9 6071

Note: if no suspicious file is found, the file may be deleted. This suspicious process has been saved to memory. It is a memory process. At this time, you need to find the PID and kill it

4, Check startup items:

Schematic diagram of system operation level:

Run levelmeaning
0Shut down
1Single user mode, which can be imagined as the security mode of windows, is mainly used for system repair
2Incomplete command line mode without NFS service
3The complete command line mode is the standard character interface
4System retention
5Graphic mode

View run level commands:


Boot profile:


When the Linux system is started, some scripts will be run to configure the environment - rc script. After the kernel initializes and loads all modules, the kernel starts a daemon called init or init d. This daemon starts running / etc / init Some scripts in D / rc. These scripts include commands to start the services required to run the Linux system
There are two ways to execute scripts at startup:

In / etc / RC Add a startup script between the exit 0 statements of local. Scripts must have executable permissions
Use update RC D command to add startup execution script

1. Edit and modify / etc / RC local

2,update-rc.d: This command is used to install or remove System-V style initialization script connections. The script is stored in / etc / init D / directory. Of course, you can create connection files in this directory to connect to script files stored in other places.
This command can specify the execution sequence number of the script. The value range of the sequence number is 0-99. The larger the sequence number, the later the execution.
When we need to start our own script, we just need to leave the executable script in / etc / init D directory, and then in / etc / RC d/rc_. D file to establish a soft link

update-rc.d Script name or service <remove|defaults|disable|enable>
#1. In / etc / init D directory to create a script linking files to the backdoor:
ln -s /home/b4yi/kali-6666.elf /etc/init.d/backdoor 
#2. Use update RC The D command adds the connection file backdoor to the startup script
sudo update-rc.d backdoor defaults 99

Execute upon startup.

Intrusion detection:

more /etc/rc.local/etc/rc.d/rc[0~6].dls -l /etc/rc.d/rc3.d/

Planned task Troubleshooting:
Several paths using cron should be noted:

crontab -l  # Lists the timer settings for the current user
crontab -r  # Delete the cron task of the current user

The above command actually lists the contents of / var/spool/cron/crontabs/root:

/etc/crontab Only allowed root User modification
/var/spool/cron/Each user's crontab Tasks, each named after the Creator
/etc/cron.d/Write the file to this directory in the format and/etc/crontab identical
 Put the script in/etc/cron.hourly/,/etc/cron.daily/,/etc/cron.weekly/,/etc/cron.monthly/In the directory, let it every hour/day/week/Once a month


more /etc/cron.daily/*  View all files in the directory

Intrusion detection:
Focus on whether there are malicious scripts in the following directories;

/var/spool/cron/* /etc/crontab/etc/cron.d/*/etc/cron.daily/* /etc/cron.hourly/* /etc/cron.monthly/*/etc/cron.weekly//etc/anacrontab/var/spool/anacron/*

Intrusion detection:
Query installed services:
Services installed by RPM package:

chkconfig  --list  View the service self start status, and you can see all the RPM Package installed services
ps aux | grep crond View the startup items of the current service system at levels 3 and 5 
Chinese environment
chkconfig --list | grep "3:Enable\|5:Enable" 
English environment
chkconfig --list | grep "3:on\|5:on"

Services installed in source package:

Check the service installation location, usually in/user/local/
service httpd start
 search/etc/rc.d/init.d/  Check to see if it exists

Abnormal file check:
There are three ways to find modified files:

By name
 Based on file size
 Find by time

Find files by name

find / -name a.Test
# If the file name is incomplete, the wildcard * can be used to complete it
# If case insensitive, you can replace - name with - iname

Find by file size:

find / -size +1000M
# +1000M represents files larger than 1000M, and - 10M represents files smaller than 10M

Find by time:

# -Access time of atime file
# -mtime file content modification time
# -ctime file status modification time (file permission, owner / group, file size, etc. of course, ctime will change with the change of file content)
# Note: when the system process / script accesses the file, atime/mtime/ctime will also be modified. It is not necessarily manual modification that will be recorded 
# Find files modified within the last day:
find / -mtime -1 -ls  | more
# Find files modified 50 days ago:
find ./ -mtime +50 -ls

Find by owner and group:

-user Find by owner
-group Find by group
-nouser Find files that do not belong to a master
-nogroup Find files that do not belong to a group 
# View the file whose owner is root and find- user root -type f
# -type f means to find the file, - type d means to find the directory
# Note: if there is no master or group file or directory in the system, it is also easy to cause potential safety hazards. It is recommended to delete it.

Sort by CPU utilization from high to low:

ps -ef --sort -pcpu

Sort by memory usage from high to low:

ps -ef --sort -pmem

1. View sensitive directories, such as files in / tmp directory, and pay attention to hidden folders. Folders named "..." have hidden properties.
2. Get the creation time of WEBSHELL and remote control Trojan horse. How to find the files created within the same time range? You can use the find command to find, such as find /opt -iname "*" - atime 1 -type f to find the file accessed by / opt the next day.
3. For suspicious files, you can use stat to create and modify the time.

5, System log check:

Default log storage location: / var/log/
Required logs: secure, history
Check the log configuration: more / etc / rsyslog conf

/var/log/wtmp Log in, log out, data exchange, shutdown and restart records
/var/log/lastlog The file records the last login information of the user, which is available lastlog Command to view.
/var/log/secure A file that records data accessed by logging in to the system, such as pop3/ssh/telnet/ftp And so on will be recorded.
/var/log/cron Log information related to scheduled tasks
/var/log/message Information and error log after system startup
/var/log/apache2/access.log apache access log

Log analysis skills:

1,How many positioning IP In the blasting host root Account number:    
grep "Failed password for root" /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more 
What are the positioning IP During blasting:
grep "Failed password" /var/log/secure|grep -E -o "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9]
[0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"|uniq -c 
What is a user name dictionary?
grep "Failed password" /var/log/secure|perl -e 'while($_=<>){ /for(.*?) from/; print "$1\n";}'|uniq -c|sort -nr 
2,Successful login IP What are:   
grep "Accepted " /var/log/secure | awk '{print $11}' | sort | uniq -c | sort -nr | more 
Date of successful login, user name IP: 
grep "Accepted " /var/log/secure | awk '{print $1,$2,$3,$9,$11}'  
3,Add a user kali journal:
Jul 10 00:12:15 localhost useradd[2382]: new group: name=kali, GID=1001
Jul 10 00:12:15 localhost useradd[2382]: new user: name=kali, UID=1001, GID=1001, home=/home/kali, shell=/bin/bash
Jul 10 00:12:58 localhost passwd: pam_unix(passwd:chauthtok): password changed for kali
#grep "useradd" /var/log/secure  
4,delete user kali journal:
Jul 10 00:14:17 localhost userdel[2393]: delete user 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed group 'kali' owned by 'kali'
Jul 10 00:14:17 localhost userdel[2393]: removed shadow group 'kali' owned by 'kali'# grep "userdel" /var/log/secure 
5,su Switch users: Jul 10 00:38:13 localhost su: pam_unix(su-l:session): session opened for user good by root(uid=0) 
sudo Authorized execution:
sudo -l
Jul 10 00:43:09 localhost sudo:    good : TTY=pts/4 ; PWD=/home/good ; USER=root ; COMMAND=/sbin/shutdown -r now

More share WeChat search Security info official account
Add group leader to "secure communication" wechat group

Keywords: Linux ssh server

Added by pkallberg21 on Mon, 21 Feb 2022 10:52:15 +0200