Squid proxy server
Squid mainly provides cache acceleration and application layer filtering control functions
1. Working mechanism of agency
- Instead of the client requesting data from the website, you can hide the user's real IP address
- The obtained Web page data (static Web elements) is saved in the cache and sent to the client, so that it can respond quickly the next time the same data is requested
2. Type of squid agent
- Traditional proxy: it is applicable to the Internet. You need to specify the address and port of the proxy server on the client
- Transparent proxy: the client does not need to specify the address and port of the proxy server, but redirects the Web access to the proxy server through the default route and firewall policy
- Reverse proxy: if the requested resources are cached in the Squid reverse proxy server, the requested resources will be returned directly to the client; Otherwise, the reverse proxy server will request resources from the background WEB server, and then return the requested response to the client. At the same time, it will also cache the response locally for use by the next requester
3. Benefits of using agents
- Improve Web access speed
- Hide the real IP address of the client
Install Squid service
The server address squid 192.168.88.60 apache1 192.168.88.70 Client( windows10) 192.168.88.100
1. Close the firewall and security mechanism
systemctl stop firewalld systemctl disable firewalld setenforce 0
2. Compile and install Squid
mount /dev/sr0 /mnt #Install environment dependent packages yum -y install gcc gcc-c++ make tar zxvf squid-3.5.28.tar.gz -C /opt/ cd /opt/squid-3.5.28 ./configure --prefix=/usr/local/squid \ #Point to the installation directory path --sysconfdir=/etc \ #Specify the configuration file path --enable-arp-acl \ #MAC address control to prevent clients from using ip spoofing --enable-linux-netfilter \ #Using kernel filtering --enable-linux-tproxy \ #Support transparent mode --enable-async-io=100 \ #Asynchronous IO to improve storage performance --enable-err-language="Simplify_Chinese" \ #Display language of error message --enable-underscore \ #Allow underscores in URLs --enable-poll \ #Turn off the default use poll mode --enable-gnuregex #Using gnu regular expressions Compile and install added modules cd /opt/squid-3.5.28 ./configure --prefix=/usr/local/squid \ --sysconfdir=/etc \ --enable-arp-acl \ --enable-linux-netfilter \ --enable-linux-tproxy \ --enable-async-io=100 \ --enable-err-language="Simplify_Chinese" \ --enable-underscore \ --enable-poll \ --enable-gnuregex make && make install ln -s /usr/local/squid/sbin/* /usr/local/sbin/ #Optimization path useradd -M -s /sbin/nologin squid #Create program user chown -R squid:squid /usr/local/squid/var/
3. Modify Squid's configuration file
vim /etc/squid.conf #The operation mechanism of this file is to match from top to bottom, and the matching is not completed from bottom to top ... #Line 56 insert http_access allow all #Put on HTTP_ Before access deny all, allow any client to use the proxy service http_access deny all http_port 3128 #Used to specify the address and port that the proxy service listens to (the default port number is 3128) #Line 61 inserts cache management users and groups cache_effective_user squid #Add, specify the program user, which is used to set the account of initialization and runtime cache. Otherwise, the startup will not succeed cache_effective_group squid #Add, specify account basic group coredump_dir /usr/local/squid/var/cache/squid #Specify cache file directory
4.Squid operation control
#Check whether the configuration file syntax is correct squid -k parse #Start Squid squid -z #-The z option is used to initialize the cache directory squid #Start squid service netstat -anpt | grep "squid"
5. Write Squid startup control service script
vim /etc/init.d/squid
#!/bin/bash
#chkconfig: 2345 90 25 #2345 is the default startup level, 90: startup level, 90th startup (0-100) 25: stop level
PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"
case "$1" in
start)
netstat -natp | grep squid &> /dev/null
if [ $? -eq 0 ]
then
echo "squid is running"
else
echo "Starting squid..."
$CMD
fi
;;
stop)
$CMD -k kill &> /dev/null
rm -rf $PID &> /dev/null
;;
status)
[ -f $PID ] &> /dev/null
if [ $? -eq 0 ]
then
netstat -natp | grep squid
else
echo "squid is not running"
fi
;;
restart)
$0 stop &> /dev/null
echo "Closing squid..."
$0 start &> /dev/null
echo "Starting squid..."
;;
reload)
$CMD -k reconfigure
;;
check)
$CMD -k parse
;;
*)
echo "Usage: $0{start|stop|status|reload|check|restart}"
;;
esac
chmod +x /etc/init.d/squid
chkconfig --add squid #Added to the system to identify
chkconfig --level 35 squid on #3.5 automatically start the service at the operation level
Building traditional proxy server
1. squid server (192.168.88.60)
vim /etc/squid.conf ...... http_access allow all http_access deny all http_port 3128 cache_effective_user squid cache_effective_group squid 63 Row insertion cache_mem 64 MB #Specify the size of the memory space used by the cache function to maintain the frequently accessed WEB objects. The capacity is preferably a multiple of 4, and the unit is MB. It is recommended to set it to 1 / 4 of the physical memory reply_body_max_size 10 MB #The maximum file size that users are allowed to download, in bytes. When downloading a Web object that exceeds the specified size, a prompt of "request or access too large" will appear on the error page of the browser. The default setting is 0, which means no restriction maximum_object_size 4096 KB #The maximum object size allowed to be saved to the cache space, in KB. Files exceeding the minimum limit will not be cached, but will be forwarded directly to the user service squid restart systemctl restart squid
2. Modify firewall rules
iptables -F iptables -I INPUT -p tcp --dport 3128 -j ACCEPT iptables -L INPUT
3.apache1 installation of http service
systemctl stop firewalld.service setenforce 0 yum -y install httpd systemctl start httpd netstat -natp | grep 80
4. Configure the proxy service of the client browser (ip:192.168.88.100 gw: 192.168.88.2)
4. Apache 1 viewing logs
tail -f /var/log/httpd/access_log
Build transparent proxy server
host IP address Services, installation packages, tools Squid-Server ens33: 192.168.88.60 ens36: 192.168.100.1 squid-3.5.28.tar.gz Web1 192.168.88.70 httpd client Windows 192.168.88.100 -
1.Web1(192.168.88.70)
① Install the http service and restart the service
yum -y install httpd systemctl restart httpd.service
2.Squid server configuration (ens33, ens36)
① Power off, add a network card, change to host only mode, power on, ifconfig, view the name of the new network card, and edit the network card configuration file
cd /etc/sysconfig/network-scripts/ cp ifcfg-ens33 ifcfg-ens36 vim ifcfg-ens36 systemctl restart network
② Change profile support transparent mode
vim /etc/squid.conf ...... http_access allow all http_access deny all http_port 192.168.88.100:3128 transparent service squid reload #If it has been started before and overloaded, it can also be restarted. If it has not been started, start
③ Turn on routing forwarding
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf sysctl -p
Note that the virtual machine can ping the web server of the network segment in the host mode only. Even if route forwarding is not enabled, the client cannot Ping the same server. It is necessary to add a static route
Add static route on web server
route add -net 192.168.88.0/24 gw 192.168.88.60
④ Add firewall rules (client access does not need to specify an intermediate server, but is forwarded through the firewall)
Modify firewall rules (from 136 network segments in the future): 80/443 Port redirected to port 3128, 443: https) iptables -F iptables -t nat -F iptables -t nat -I PREROUTING -i ens36 -s 192.168.88.0/24 -p tcp --dport 80 -j REDIRECT --to 3128 #This is the Internet interface ens36 iptables -t nat -I PREROUTING -i ens36 -s 192.168.88.0/24 -p tcp --dport 443 -j REDIRECT --to 3128 iptables -I INPUT -p tcp --dport 3128 -j ACCEPT #If you restart, you need to reconfigure this rule #REDIRECT: REDIRECT
3. Client test (192.168.88.100)
View new records of Squid access log
tail -f /usr/local/squid/var/logs/access.log
Squid proxy server detects that the client accesses the target website
Check the new records in the Web access log, which shows that the external network port of the proxy server replaces the access of the client
tail -f /var/log/httpd/access_log
ACL access control
1. In the configuration file squid In conf, ACL access control is realized through the following two steps
(1) Use acl configuration items to define the conditions to be controlled;
(2) Via http_ The access configuration item controls "allow" or "deny" access to the defined list.
1. Define access control list
Format:
acl list name list type list content
List name: the name is user-defined, which is equivalent to giving ACL a name (a bit similar to shell script variable)
List type: the predefined value of Squid must be used to correspond to different types of control conditions
List content: refers to the specific object to be controlled. The corresponding contents of different types of lists are also different. There can be multiple values (separated by spaces and the relationship of "or")
Mode 1 vim /etc/squid.conf ...... acl host src 192.168.136.10/24 #The source address is 192.168.136.10, which is added separately acl MYLAN src 192.168.0.0/24 #Segment based acl destionhost dst 192.168.190.20/32 #The destination address is 192.168.184.20 acl MC20 maxconn 20 #Maximum concurrent connections 20 acl PORT port 21 #Target port 21 acl DMBLOCK dstdomain .qq.com #Target domain, matching all sites in the domain acl BURL url_regex -i ^rtsp:// ^emule:// # URLs starting with rtsp: / /, eMule: /, - i means case is ignored The structure of url regular expression, which controls the filtering ^ rtsp (Protocol) based on the specified protocol acl PURL urlpath_regex -i \.mp3$ \.mp4$ \.rmvb$ #Based on access to the end of the file mp3,. mp4,. URL path at the end of rmvb acl WORKTIME time MTWHF 08:30-17:30 #Based on the access time control, it is 8:30 ~ 17:30 from Monday to Friday Insert in the first clause: http_access deny host Mode 2: mkdir /etc/squid #Enable object list management vim dest.list 192.168.190.5 192.168.190.20 192.168.190.30 vim /etc/squid.conf #Call the contents of the specified file list #The called list is the content in the form of a file. You need to define the file to the list first acl List name of the file definition dst (Destination address, file path) For example: acl destionhost dst "/etc/squid/dest.list" http_access deny destionhost #Note that if it is a rejection list, it needs to be placed in http_access allow all http_port 3128 systemctl restart squid
Squid log analysis
1. Install image processing software package
#You need to use the online source. If an error occurs, you can modify the dns and gateway in the network card configuration file back to the original yum install -y pcre-devel gd gd-devel mkdir /usr/local/sarg tar zxvf sarg-2.3.7.tar.gz -C /opt/ cd /opt/sarg-2.3.7 ./configure --prefix=/usr/local/sarg --sysconfdir=/etc/sarg --enable-extraprotection make && make install
2. Modify the configuration file
vim /etc/sarg/sarg.conf 7 Line uncomment access_log /usr/local/squid/var/logs/access.log #Specify access log file 25 Line uncomment title "Squid User Access Reports" #Page title 120 Line uncomment output_dir /var/www/html/squid-reports #Report output directory 178 Line uncomment user_ip no #Display with user name 184 Line uncomment, modify topuser_sort_field connect reverse #In top sorting, the specified connection times are arranged in descending order, and the ascending order is normal 190 Line uncomment, modify user_sort_field connect reverse #For user access records, the number of connections is sorted in descending order 206 Line uncomment, modify exclude_hosts /usr/local/sarg/noreport #Specifies files that are not included in the sorted site list 257 Line uncomment overwrite_report no #Overwrite logs with the same name and date 289 Line uncomment, modify mail_utility mailq.postfix #Send mail report command 434 Line uncomment, modify charset UTF-8 #Specifies the character set UTF-8 518 Line uncomment weekdays 0-6 #Week cycle of top ranking 525 Line uncomment hours 0-23 #Time period of top ranking 633 Line uncomment www_document_root /var/www/html #Specify page root
Add is not included in the site file, and the added domain name will not be displayed in the sorting
touch /usr/local/sarg/noreport ln -s /usr/local/sarg/bin/sarg /usr/local/bin/ sarg --help
function
sarg #Start a record
verification
yum install httpd -y systemctl start httpd
3. Add scheduled tasks, execute and generate reports every day
vim /usr/local/sarg/report.sh #/bin/bash #Get current date TODAY=$(date +%d/%m/%Y) #Get one week ago today YESTERDAY=$(date -d "1 day ago" +%d/%m/%Y) /usr/local/sarg/bin/sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/squid-reports -z -d $YESTERDAY-$TODAY &> /dev/null exit 0
chmod +x /usr/local/sarg/report.sh crontab -e 0 0 * * * /usr/local/sarg/report.sh
Squid reverse proxy
If the requested resource is cached in the Squid reverse proxy server, the requested resource is directly returned to the client; Otherwise, the reverse proxy server will request resources from the background Web server, and then return the requested response to the client. At the same time, it will also cache the response locally for the next requester to use.
(it is recommended not to continue based on the above experiment, which may cause the squid server to fail to start at port 80)
Working mechanism:
● cache web page objects to reduce duplicate requests
● assign Internet request rotation training or weight to intranet Web server
● acting as the user's request, avoiding the user's direct access to the Web server and improving security
1. Squid server (192.168.88.60) network card nat mode
Because httpd in the experiment will occupy port 80 of the server, it must be closed
Add a web2 server 192.168.88.80
Turn on the firewall
systemctl start firewalld
Local httpd shutdown
systemctl stop httpd
iptables -F #Clear firewall rules iptables -t nat -F #The emptying rule can't do internal and external forwarding, so it didn't pass 3 a.m. and then the nat mode succeeded iptables -I INPUT -p tcp --dport 3128 -j ACCEPT vim /etc/squid.conf ...... 60 Row modification, inserting http_port 192.168.190.11:80 accel vhost vport cache_peer 192.168.190.12 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web1 cache_peer 192.168.190.13 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web2 cache_peer_domain web1 web2 www.gg.com systemctl restart squid netstat -natp | grep 80
http_ Port 80 access Vhost vport #squid changes from a cache to a web server using the acceleration mode. At this time, squid listens to requests at port 80 and binds to the request port (vhost vport) of the web server. At this time, squid does not need to forward requests. Instead, squid directly takes data from the cache or directly requests data from the bound port.
parent :Represents the parent node 80: HTTP_PORT 0: ICP_PORT no-query: Do not query, directly obtain data originserver: Specify source server round-robin: appoint squid The request is distributed to one of the parent nodes by polling max_conn: Specify the maximum number of connections weight: Specify weights name: Set alias
2.web1 and web2 (install httpd service and configure web page content)
systemctl stop firewalld.service setenforce 0 yum install -y httpd systemctl start httpd web1(192.168.88.70) echo "this is web1" >> /var/www/html/index.html web2(192.168.88.80) echo "this is web2" >> /var/www/html/index.html
3. Domain name mapping of client (192.168.88.100)
modify C:\Windows\System32\drivers\etc\hosts file 192.168.88.60 www.gg.com
4.squid server
Delete the previous acl reject source address