squid proxy server

Cache agent overview

Working mechanism of Web proxy:
Cache web page objects to reduce duplicate requests

Basic types of agents

Traditional agent: it is applicable to the Internet, and the server needs to be clearly specified
Transparent proxy: the client does not need to specify the address and port of the proxy server, but redirects Web access to the proxy server through default routing and firewall policies

Benefits of using agents

Improve Web access speed
Hide the real IP address of the client

Deploy squid proxy service

Install squid service

systemctl stop firewalld
systemctl disable firewalld
setenforce 0

1.Compile and install Squid(Upload the required package to/opt (under directory)
yum -y install gcc gcc-c++ make
tar zxvf squid-3.5.28.tar.gz -C /opt/
cd /opt/squid-3.5.28
./configure --prefix=/usr/local/squid \     #Specify the installation directory path
--sysconfdir=/etc \                         #Specify profile path
--enable-arp-acl \                          #MAC address control to prevent clients from using IP spoofing
--enable-linux-netfilter \                  #Using kernel filtering
--enable-linux-tproxy \                     #Support transparent mode
--enable-async-io=100 \                     #Asynchronous IO to improve storage performance
--enable-err-language="Simplify_ Chinese" \ #Display language of error message
--enable-underscore \                       #Allow underscores in URL s
--disable-poll \                            #Turn off the default use poll mode
--enable-epoll \                            #Enable epoll mode to improve performance
--enable-gnuregex                           #Using GNU regular expressions

-------------------------------------------------------------------------------------
./configure --prefix=/usr/local/squid \
--sysconfdir=/etc \
--enable-arp-acl \
--enable-linux-netfilter \
--enable-linux-tproxy \
--enable-async-io=100 \
--enable-err-language="Simplify_Chinese" \
--enable-underscore \
--disable-poll \
--enable-epoll \
--enable-gnurege
---------------------------------------------------------------------------------------

make && make install

ln -s /usr/local/squid/sbin/* /usr/local/sbin/

useradd -M -s /sbin/nologin squid

chown -R squid:squid /usr/local/squid/var/       #This directory is used to store cache files


2.modify Squid Configuration file for
vim /etc/squid.conf
......
-----56 that 's ok--insert------
http_access allow all    #Put on HTTP_ Before access deny all, any client is allowed to use the proxy service to control the top-down matching of rules
http_access deny all
http_port 3128           #Used to specify the address and port that the proxy service listens to (the default port number is 3128)
-----61 that 's ok--insert------
cache_effective_user squid     #Add, specify the program user, which is used to set the account of initialization and runtime cache. Otherwise, the startup will not succeed
cache_effective_group squid    #Add, specify account basic group
coredump_dir /usr/local/squid/var/cache/squid   #Specify cache file directory


3. Squid Operation control of
#Check whether the configuration file syntax is correct
squid -k parse    #Check configuration file
squid -k rec      #Reload profile
squid -zX         #Initialize cache directory

#Start Squid. When Squid service is started for the first time, the cache directory will be initialized automatically
squid -z   #-The z option is used to initialize the cache directory
squid      #Start squid service

netstat -anpt | grep squid

4.establish Squid Service script
vim /etc/init.d/squid
#!/bin/bash
#chkconfig: 2345 90 25
PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"

case "$1" in
   start)
     netstat -natp | grep squid &> /dev/null
     if [ $? -eq 0 ]
     then
       echo "squid is running"
       else
       echo "Starting squid..."
       $CMD
     fi
   ;;
   stop)
     $CMD -k kill &> /dev/null
     rm -rf $PID &> /dev/null
   ;;
   status)
     [ -f $PID ] &> /dev/null
        if [ $? -eq 0 ]
          then
            netstat -natp | grep squid
          else
            echo "squid is not running"
        fi
   ;;
   restart)
      $0 stop &> /dev/null
      echo "Closing squid..."
      $0 start &> /dev/null
      echo "Starting squid..."
   ;;
   reload)
      $CMD -k reconfigure
   ;;
   check)
      $CMD -k parse
   ;;
   *)
      echo "Usage: $0{start|stop|status|reload|check|restart}"
   ;;
esac

#2345 is the default self startup level. If yes - means that no self startup is available at any level; 90 is the start priority, 25 is the stop priority, and the priority range is 0-100. The higher the number, the higher the priority
 The lower.
chmod +x /etc/init.d/squid
chkconfig --add squid
chkconfig --level 35 squid on

Configure legacy proxy services

environment

squid Server: 192.168.121.15
web1 Server: 192.168.121.16
 Client: 192.168.121.200

vim /etc/squid.conf
......
http_access allow all
http_access deny all
http_port 3128
cache_effective_user squid
cache_leffective_group squid
-------63 that 's ok--insert---------
cache_mem 64 MB
#Specify the memory space used by the cache function to maintain frequently accessed WEB objects. The capacity is preferably a multiple of 4, in MB. It is recommended to set it to 1 / 4 of the physical memory
reply_body_max_size 10 MB
#The maximum file size that users are allowed to download, in bytes. When downloading a Web object of more than the specified size, a prompt of "request or access too large" will appear on the error page of the browser. The default setting is 0, which means no restriction
maximum_object_size 4096 KB
#The maximum object size allowed to be saved to the cache space, in kilobytes. Files exceeding the size limit will not be cached, but will be forwarded directly to the user

service squid restart
systemctl restart squid

#Firewall rules also need to be modified in the production environment
iptables -F
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

#Proxy configuration for client
 Open browser, tools-->Internet option-->connect -->LAN settings-->Turn on proxy server (address: Squid The server IP Address, port: 3128)
After clearing the cache, use the browser to access http://192.168.121.16 (web server address)

#View new records of Squid access log
tail -f /usr/local/squid/var/logs/access.log

#View new records in the Web access log
tail -f /var/log/httpd/access_log
 Enter in the browser Web The server IP Address access, view Web The server access log shows that the proxy server accesses for the client.

Configure transparent proxy service

Environment construction

squid Server: Intranet ens33:192.168.121.15  Extranet ens37:12.0.0.1
Web Server: 192.168.121.16
 Client: 12.0.0.12

Squid server configuration

1. First, you need to add a network card on the squid server: ens37 as the gateway from the external network client
2. Modify the profile of squid server

vim /etc/squid.conf
http_access allow all
http_access deny all
--60 that 's ok--Modify and add intranet services IP Address, and support transparent proxy options transparent
http_port 12.0.0.1:3128 transparent

systemctl restart squid

#Enable routing forwarding to realize address forwarding of different network segments in the machine
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
sysctl -p

#Modify firewall rules
iptables -F
iptables -t nat -F
iptables -t nat -I PREROUTING -i ens33 -s 12.0.0.0/24 -p tcp --dport 80 -j REDIRECT --to 3128  #http protocol for forwarding
iptables -t nat -I PREROUTING -i ens33 -s 12.0.0.0/24 -p tcp --dport 443 -j REDIRECT --to 3128  #https protocol for forwarding
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

Web server

route add -net 12.0.0.0/24 gw 192.168.121.15  #Make a static route, otherwise the data can be sent out but can't come back

tail -f /usr/local/squid/var/logs/access.log  #The Web node server can monitor the flow of data
 Or you can squid For monitoring on the server, use the following command
tail -f /usr/local/squid/var/logs/access.log

test

ACL access control

In profile squid.conf In, ACL Access control is implemented in the following two steps:
(1)use acl The configuration item defines the conditions to be controlled;
(2)adopt http_access The configuration item controls "allow" or "deny" access to the defined list.

1.Define access control lists
 format:
acl List name list type list content ...

Method 1:
vim /etc/squid.conf
.......
acl localhost src 12.0.0.12              #The source address is 12.0.0.12
acl MYLAN src 12.0.0.0/24           #Client network segment
acl destinationhost dst 192.168.121.15       #The destination address is 192.168.121.15
acl MC20 maxconn 20                             #Maximum concurrent connections 20
acl PORT port 21                                #Target port 21
acl DMBLOCK dstdomain .qq.com                   #Target domain, matching all sites in the domain
acl BURL url_regex -i ^rtsp:// ^Module: / / # with RTSP: / / URL starting with module: / /, - i means case is ignored
acl PURL urlpath_regex -i \.mp3$ \.mp4$ \.rmvb$   #With mp3,. mp4,. URL path at the end of rmvb
acl WORKTIME time MTWHF 08:30-17:30             #The time is 8:30 ~ 17:30 from Monday to Friday, "MTWHF" is the English initials of each week

--------------------------------------------------------------------------------------
Method 2:
#Start object list management
mkdir /etc/squid
vim /etc/squid/dest.list
192.168.121.15      #Squid server IP
12.0.0.0/24     #Any required network segment or IP address

vim /etc/squid.conf
......
acl destinationhost dst "/etc/squid/dest.list"    #Call the contents of the list in the specified file
http_access deny (or allow) destinationhost      #Note that if it is a rejection list, it needs to be placed in http_ access allow all

systemctl restart squid


Open the proxy service and visit the browser Web The server http://192.168.121.16, access denied.

Squid log analysis

squid server

#Install image processing package
yum install -y pcre-devel gd gd-devel

mkdir /usr/local/sarg

#Add zxvf sarg-2.3.7 tar. Upload GZ compressed package to / opt directory
tar zxvf sarg-2.3.7.tar.gz -C /opt/

cd /opt/sarg-2.3.7
./configure --prefix=/usr/local/sarg \
--sysconfdir=/etc/sarg \     #The configuration file directory is / usr/loca/etc by default
--enable-extraprotection       #Additional safety protection

--------------------------------------------------------------------------------
./configure --prefix=/usr/local/sarg --sysconfdir=/etc/sarg --enable-extraprotection

make && make install
--------------------------------------------------------------------------------

vim /etc/sarg/sarg.conf
--7 that 's ok--note off
access_log /usr/local/squid/var/logs/access.log  #Specify access log file
--25 that 's ok--note off
title "Squid User Access Reports"     #Page title
--120 that 's ok--Uncomment, modify
output_dir /var/www/html/sarg      #Report output directory
--178 that 's ok--note off
user_ip no           #Display with user name
--184 that 's ok--Uncomment, modify
topuser_sort_field connect reverse     #In top sorting, the specified connection times are arranged in descending order, and the ascending order is normal
--190 that 's ok--Uncomment, modify
user_sort_field connect reverse      #For user access records, the number of connections is sorted in descending order
--206 that 's ok--Uncomment, modify
exclude_hosts /usr/local/sarg/noreport    #Specifies files that are not included in the sorted site list
--257 that 's ok--note off
overwrite_report no         #Overwrite logs with the same name and date
--289 that 's ok--Uncomment, modify
mail_utility mailq.postfix       #Send mail report command
--434 that 's ok--Uncomment, modify
charset UTF-8          #Specifies the character set UTF-8
--518 that 's ok--note off
weekdays 0-6          #Week cycle of top ranking
--525 that 's ok--note off
hours 0-23           #Time period of top ranking
--633 that 's ok--note off
www_document_root /var/www/html      #Specify page root

#Add is not included in the site file, and the added domain name will not be displayed in the sorting
touch /usr/local/sarg/noreport

ln -s /usr/local/sarg/bin/sarg /usr/local/bin/

sarg --help   #get help 

#function
sarg     #Start a record

#verification
yum install httpd -y
systemctl start httpd

stay squid Use browser access on the server http://192.168.200.60/sarg, check the Sarg report page.

#Add scheduled tasks to perform daily report generation
vim /usr/local/sarg/report.sh
#/bin/bash
#Get current date
TODAY=$(date +%d/%m/%Y)
#Get one week ago today
YESTERDAY=$(date -d "1 day ago" +%d/%m/%Y)
/usr/local/sarg/bin/sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/sarg -z -d $YESTERDAY-$TODAY &> /dev/null
exit 0

chmod +x /usr/local/sarg/report.sh 

crontab -e
0 0 * * * /usr/local/sarg/report.sh

Reverse proxy

If the requested resource is cached in the Squid reverse proxy server, the requested resource is directly returned to the client; Otherwise, the reverse proxy server will request resources from the background Web server, and then return the requested response to the client. At the same time, it will also cache the response locally for use by the next requester.

Working mechanism:

● cache web page objects to reduce duplicate requests
● assign the Internet request to the intranet Web server in rotation or by weight
● proxy user requests to prevent users from directly accessing the Web server and improve security

Environment configuration

squid Server: 192.168.121.15    CentOS7	
Web1 Server: 192.168.121.16      CentOS7	
Web2 Server: 192.168.121.17      CentOS7	
client Server: 192.168.121.200    Windows

vim /etc/squid.conf
------ 60 that 's ok--Modify, insert-------
http_port 192.168.121.15:80 accel vhost vport
cache_peer 192.168.121.16 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web1
cache_peer 192.168.121.17 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web2
cache_peer_domain web1 web2 www.song.com
#Means to www.song.com COM, squid to 192 168.121.16 and 192.168.121.17

-------------------------------------------------------------------------------------
http_port 80 accel vhost vport
#Squid has changed from a cache to a Web server reverse proxy acceleration mode. At this time, squid listens to requests on port 80 and binds to the request port (vhost vport) of webserver. At this time, squid does not need to forward requests. Instead, it directly needs to take data from the cache or directly request data from the bound port.
accel :Reverse proxy acceleration mode
vhost:Support domain name or host name to represent proxy node
vport :support IP And port to represent the proxy node

parent :Represents the parent node, the parent node, the parent node, the parent node, the parent node, and the parent node
80:Agent internal web Port 80 of the server
0 :Not used icp，It means only one squid The server
no-query :Get data directly without query
originserver :Specify source server
round-robin :appoint squid The request is distributed to one of the parent nodes by polling
max_ conn :Specify the maximum number of connections
weight :Specify weights
name :Set alias
----------------------------------------------------------------------------------------

systemctl stop httpd
service squid reload

#Backend node server settings
yum install -y httpd
systemctl start httpd

#Node 1(web1):
echo "this is test01" >> /var/www/html/index.html
#Node 2(web2):
echo "this is test02" >> /var/www/html/index.html

#Domain name mapping configuration of the client
 modify C: \Windows\System32\drivers\etc\hosts file
192.168.121.15 www.song.com
#Proxy configuration for client
 Open browser, tools-->Internet option-->connect-->LAN settings-->Turn on proxy server(address: Squid The server IP Address, port: 80)

Browser access http://www.song.com

summary

squid servers are positioned for cache acceleration

Obtained from the back-end web server when caching
Acceleration is for client access

Three modes of squid agent:

Traditional agent:
The client needs to point to the squid proxy server, and the client can perceive the existence of the squid proxy server
Transparent proxy:
The client does not need to be configured, but can be accessed directly. The server completes the transparent proxy with the help of firewall rules and static routing
Reverse proxy
As a reverse proxy function similar to Nginx server, but it does not need a home page to complete the reverse proxy based on IP, port and weight

For squid's own management / functions:

ACL: mainly used for http_ Permission and denial management of access (access based on http protocol)
sarg: log analysis function. Access can be specified in days_ The contents of the log are output to a web page (with the help of httpd) for display

What is reverse proxy? What is the difference between forward agent and forward agent?

Reverse proxy hides the real server when we request www.baidu.com COM, just like dialing 10086, there may be thousands of servers behind us, but you don't know or need to know which one. You just need to know who the reverse proxy server is, www.baidu.com COM is our reverse proxy server. The reverse proxy server will help us forward requests to the real server. Nginx is a very good reverse proxy server for load balancing.

The difference between forward proxy and reverse proxy is that the object of forward proxy is the client, and the object of reverse proxy is the server

Keywords: MySQL

Added by jcbarr on Sat, 22 Jan 2022 19:34:37 +0200

Programming VIP