Squid proxy application

Catalog

One 、squid proxy server

Two 、 Overview of caching agents

1、Web The working mechanism of agency

2、 Working mechanism of forward agent  

3、 Concept and function of proxy server

3.1 The concept of proxy server

3.2 The main role of proxy server

3.3Squid、Varinsh and Nginx What's the difference? , How do you choose ？

4、squid Type of agent

3、 ... and 、Squid The installation is introduced

1、 install Squid service

1.1 Turn off firewall

1.2 Compilation and installation squid

2、 modify Squid Configuration file for

3、Squid The operation control of

4、 establish Squid Service script

Four 、 Build a traditional proxy server

1、 proxy server 192.168.52.110

2、web The server 192.168.52.100

3、 Agent configuration for the client

Proxy the client

After configuring the agent , More visits web The server  

see web Server access log

see Squid Hits in the cache

5、 ... and 、 Build a transparent proxy server

 1、 To configure squid The server

1.1 Add network card

1.2 edit Squid Server profile

2、Web The server 12.0.0.100 To configure  

2.1 install apache

2.2 modify web The network card of the server

3、 The client 192.168.52.30 Set up

3.1 Configure the IP With gateway

 3.2 Shut down the proxy server

3.3IE Browser access web The server

3.4squid Server view access log

3.5 see Web New record of access log

6、 ... and 、 ACL Access control

         Define access control list

Edit profile , Define control access lists  

Restart the service

Use the client to access web The server

7、 ... and 、Squid Log analysis

8、 ... and 、Squid Reverse proxy

1、 To configure squid The server

1.1 Modify the configuration file

1.2 Clear firewall rules

1.3 Turn on squid service

2、web Server configuration

2.1web1 Server configuration

2.2web2 Server configuration

3、 The domain name mapping configuration of the client

         3.1 modify C:\Windows\System32\drivers\etc\hosts file

3.2 The browser does not open proxy access http://www.my.com

3.3 Check the log

---

One 、squid proxy server

Squid It mainly provides cache acceleration 、 Application layer filtering control function .

Two 、 Overview of caching agents

1、Web The working mechanism of agency

Cache web page objects , Reduce duplicate requests

2、 Working mechanism of forward agent  

1、 Instead of the client requesting data from the website , This can hide the user's real IP Address .

2、 Will get the web page data （ static state Web Elements ） Save to cache and send to client , So that the next time you request the same data, you can respond quickly . 

3、 Concept and function of proxy server

3.1 The concept of proxy server

A proxy server is one that is located between the client and the original （ resources ） Servers between servers , To get content from the original server , The client sends a request to the proxy server and specifies the target original server , Then the proxy server forwards the request to the original server and returns the obtained content to the client .

Cache proxy for Web crucial , Especially for large and high loads Web Site .Nginx Caching can be used as an important means of performance optimization , Can greatly reduce the back-end server load . Usually for static resources , Less frequently updated resources , Such as images ,css or js Wait for caching , So every time you refresh your browser , Don't ask again , It's read from the cache , This reduces the pressure on the server .

3.2 The main role of proxy server

The resource acquisition ： Instead of the client, we can get resources from the original server ;

To speed up access ： The proxy server may be closer to the original server , So as to play a certain role in accelerating ：

Caching ： The proxy server holds the resources obtained from the original server , So as to achieve the client fast access ;

Hide the real address ： Proxy server instead of client to get the original server resources , So as to hide the real information of the client .

3.3Squid、Varinsh and Nginx What's the difference? , How do you choose ？

Squid、Varinsh and Nginx It's all proxy servers

What is a proxy server ： Be able to access the public network for users , And can cache the accessed data to the local server , The next time a user accesses the same resource , The proxy server responds directly to the user locally , When there is no local , I will visit the public network instead of you , I accept your request , I'll find it in my own local cache first , If my local cache has , I will reply you directly from my local cache if I don't find the cached data you want to access in my local cache , Then the proxy server will access the public network instead of you .

difference ：

• Nginx It was a reverse agent /web The server , Using plug-ins can do this sideline, but it doesn't support many features , Only static files can be cached ;

• From these functions .varnish and squid It's professional cache service , and nginx These are the third-party modules that complete ;

• varnish Its own technical advantage is higher than squid, It uses visual page caching technology ; In the use of memory ,Varnisht Than Squid Have an advantage , Performance is better than Squid high . And powerful passage Varnish Management port , You can use regular expressions quickly 、 Batch clears part of the cache. It is a memory cache , Top speed , But memory caching also limits its capacity , Caching pages and images is generally good ; Only reverse proxy is supported , Function less , Small cache capacity ,CPU/IO/ Memory consumption , The process will lose cached data if it exits unexpectedly .

• squid The advantage is that the whole is huge cache Technical data , And a lot of application production environments ; It can support both forward proxy and reverse proxy , Cached data can also support persistence , There are many modes

Selection in work ： Want to do cache service , We must choose a major cache service , Preference squid perhaps varnish.

4、squid Type of agent

• Traditional agency ： Apply to Internet, You need to specify the address and port of the proxy server on the client .

• Transparent proxy ： The client does not need to specify the address and port of the proxy server , But through the default route 、 The firewall strategy will Web Access redirection to the proxy server for processing .

• Reverse proxy ： If squid The requested resource is cached in the reverse proxy server , The requested resource is returned directly to the client ; Otherwise, the reverse proxy server will go to the background WEB Server requests resources , Then return the requested response to the client , At the same time, the response will be stored locally , For the next requester .

3、 ... and 、Squid The installation is introduced

1、 install Squid service

1.1 Turn off firewall

systemctl stop firewalld
systemctl disable firewalld
setenforce 0

1.2 Compilation and installation squid

yum -y install gcc gcc-c++ make                     # Install dependent environment 
tar zxvf squid-3.5.28.tar.gz -C /opt/               # Extract the installation package to the specified directory opt in 
cd /opt/squid-3.5.28

./configure --prefix=/usr/local/squid \				# Specify the installation directory path 
--sysconfdir=/etc \									# Specify the profile path 
--enable-arp-acl \									#MAC Address control , Prevent clients from using IP cheating 
--enable-linux-netfilter \							# Use kernel filtering 
--enable-linux-tproxy \								# Support transparent mode 
--enable-async-io=100 \								# asynchronous IO, Improve storage performance 
--enable-err-language="Simplify_Chinese" \			# Display language of error message 
--enable-underscore \								# allow URL There is an underline in 
--disable-poll \									# Turn off the default use  poll  Pattern 
--enable-epoll \									# Turn on epoll Mode improves performance 
--enable-gnuregex	                                # Use GNU Regular expressions 
-------------------------------------------------------------------------
make && make install

ln -s /usr/local/squid/sbin/* /usr/local/sbin/   # Create soft links so that the system can directly recognize squid command 
useradd -M -s /sbin/nologin squid
chown -R squid:squid /usr/local/squid/var/		 # This directory is used to store cache files 

2、 modify Squid Configuration file for

vim /etc/squid.conf
......
--56 That's ok -- Insert 
http_access allow all	# Put it in  http_access deny all  Before , Allow any client to use the proxy service , Control rules match from top to bottom 
http_access deny all
http_port 3128					# Used to specify the address and port on which the proxy service listens ( The default port number is  3128)
--61 That's ok -- Insert 
cache_effective_user squid		# add to , Specify the program user , Used to set initialization 、 Accounts cached at runtime , Otherwise, the startup is not successful 
cache_effective_group squid		# add to , Specify account basic group 
coredump_dir /usr/local/squid/var/cache/squid		# Specify cache file directory 

3、Squid The operation control of

# Check that the configuration file syntax is correct 
squid -k parse
# start-up  Squid, First boot  Squid  The service , The cache directory is automatically initialized 
squid -z 					#-z  Option to initialize the cache directory 
squid						# start-up  squid  service 

netstat -anpt | grep "squid"

 4、 establish Squid Service script

vim /etc/init.d/squid
#!/bin/bash
#chkconfig: 2345 90 25
PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"

case "$1" in
   start)
     netstat -natp | grep squid &> /dev/null
     if [ $? -eq 0 ]
     then
       echo "squid is running"
     else
       echo " Starting  squid..."
       $CMD
     fi
   ;;
   stop)
     $CMD -k kill &> /dev/null
     rm -rf $PID &> /dev/null
   ;;
   status)
     [ -f $PID ] &> /dev/null
        if [ $? -eq 0 ]
          then
            netstat -natp | grep squid
          else
            echo "squid is not running"
        fi
   ;;
   restart)
      $0 stop &> /dev/null
      echo " Shutting down  squid..."
      $0 start &> /dev/null
      echo " Starting  squid..."
   ;;
   reload)
      $CMD -k reconfigure
   ;;
   check)
      $CMD -k parse
   ;;
   *)
      echo " usage ：$0{start|stop|status|reload|check|restart}"
   ;;
esac
-------------------------------------------------------------------------
#2345 Is the default self start level , " - Represents that no level is self starting ;90 Is the start priority ,25 Is the stop priority , The priority range is 0－100, The greater the number , The lower the priority .
chmod +x /etc/init.d/squid
chkconfig --add squid
chkconfig --level 35 squid on
chkconfig --list squid

Four 、 Build a traditional proxy server

Squid proxy server ：192.168.52.110/24

web The server ：192.168.52.100/24

The client ：192.168.52.30/24

1、 proxy server 192.168.52.110

 Close the protective wall and selinux
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
----------------------------------------
vim /etc/squid.conf
--63 That's ok -- Insert 
cache_mem 64 MB
# Specifies the amount of memory space used by the cache function , It's easy to keep a more frequent WEB object , The capacity should preferably be 4 Multiple , Unit is MB, It is recommended to set it to... Of physical memory 1/4

reply_body_max_size 100 MB 
# The maximum file size that users are allowed to download , In bytes , When downloading more than the specified size Web Object time , The error page of the browser will appear “ Request or access is too large ” The default setting for the prompt 0 It means that there is no restriction 

maximum_object_size 4096 KB
# Maximum object size allowed to be saved to cache space , With KB In units of , Files that exceed the size limit will not be cached , Instead, it is forwarded directly to the user 
----------------------------------------------------------------------------------
service squid restart     # Restart the service 


# Firewall rules also need to be modified in the production environment 
iptables -F
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

# see  Squid  Hits in the cache 
tail -f /usr/local/squid/var/logs/access.log
        TCP_MEM_HIT/200

# see  Web  New record of access log 
tail -f /var/log/httpd/access_log
 Type in the browser Web The server IP Address access , see Web Server access log , It shows that the proxy server accesses... For the client .

Turn off firewall

Edit profile

Restart the service

  Modify firewall rules

2、web The server 192.168.52.100

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
yum install httpd -y
systemctl start httpd

install apache Webpage

Start the service

3、 Agent configuration for the client

Before acting First use the browser to access and view web Server access log

see web Server access log

cd /etc/httpd/logs/
ls
tail -f access_log

Proxy the client

Open the browser （ Use as much as possible IE browser ）, Tools -->Internet Options --> Connect --> LAN settings --> Turn on the proxy server （ Address ：Squid The server IP Address , port ：3128） 

After configuring the agent , More visits web The server  

see web Server access log

see Squid Hits in the cache

tail -f /usr/local/squid/var/logs/access.log 

5、 ... and 、 Build a transparent proxy server

Squid The server ： Intranet ens33：192.168.52.30, Extranet ens36：12.0.0.254

Web The server ：12.0.0.100

The client ：192.168.52.30

 1、 To configure squid The server

1.1 Add network card

Set up ——> add to ——> network adapter ——> determine

ifconfig  # Check the NIC name 
cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 ifcfg-ens36
vim ifcfg-ens36
------------------------------
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens36
DEVICE=ens36
ONBOOT=yes
IPADDR=12.0.0.254
NETMASK=255.255.255.0
------------------------------------
vim ifcfg-ens33
----------------------------
 Comment out DNS And Gateway
----------------------
systemctl restart network

1.2 edit Squid Server profile

vim /etc/squid.conf
......
http_access allow all
http_access deny all
--60 That's ok -- Modify and add intranet services IP Address , And support transparent proxy options  transparent
http_port 192.168.52.110:3128 transparent

# restart squid service 
systemctl restart squid

# Turn on route forwarding , Realize the address forwarding of different network segments in the machine 
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
sysctl -p

# Modify firewall rules 
iptables -F
iptables -t nat -F
iptables -t nat -I PREROUTING -i ens33 -s 192.168.52.0/24 -p tcp --dport 80 -j REDIRECT --to 3128	# For forwarding http agreement 
iptables -t nat -I PREROUTING -i ens33 -s 192.168.52.0/24 -p tcp --dport 443 -j REDIRECT --to 3128	# For forwarding https agreement 
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

 

2、Web The server 12.0.0.100 To configure  

2.1 install apache

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
yum install httpd -y
systemctl start httpd

install apache Webpage

Start the service

2.2 modify web The network card of the server

vim /etc/sysconfig/network-scripts/ifcfg-ens33
--------------------------------------------------
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=a3484a07-0b78-4735-9302-e04ba6617e8b
DEVICE=ens33
ONBOOT=yes
IPADDR=12.0.0.100
NETMASK=255.255.255.0
GATEWAY=12.0.0.254
DNS1=8.8.8.8
--------------------------------------------------------
systemctl restart network
ifconfig

3、 The client 192.168.52.30 Set up

3.1 Configure the IP With gateway

 3.2 Shut down the proxy server

3.3IE Browser access web The server

3.4squid Server view access log

tail -f /usr/local/squid/var/logs/access.log

3.5 see Web New record of access log

It shows that the external network port of the proxy server replaces the client in accessing

tail -f /var/log/httpd/access_log 

6、 ... and 、 ACL Access control

In profile squid.conf in ,ACL Access control is implemented in the following two steps ：

（1） Use acl Configuration items define the conditions that need to be controlled ;

（2） adopt http_access The configuration item does “ allow ” or “ Refuse ” Access control .

1、 Define access control list

 Format ：
acl  List name   List the type   List content  …

vim /etc/squid.conf
......
acl localhost src 192.168.52.110/32 				# The source address is  192.168.52.110
acl MYLAN src 192.168.52.0/24 192.168.1.0/24		# Client network segment 
acl destinationhost dst 192.168.80.13/32			# The target address is  192.168.52.120
acl MC20 maxconn 20									# Maximum concurrent connection  20
acl PORT port 21									# Target port  21
acl DMBLOCK dstdomain .qq.com						# Target domain , Match all sites in the domain 
acl BURL url_regex -i ^rtsp:// ^emule://			# With  rtsp://、emule://  Initial goal  URL Address ,-i Indicates ignore case 
acl PURL urlpath_regex -i \.mp3$ \.mp4$ \.rmvb$		# With  .mp3、.mp4、.rmvb  End goal  URL  route 
acl WORKTIME time MTWHF 08:30-17:30					# It's Monday to Friday  8:30~17:30,“MTWHF” For the English initials of each week 

Edit profile , Define control access lists  

acl src 192.168.52.30/32
http_access deny myhost

Restart the service

systemctl restart squid

Use the client to access web The server

7、 ... and 、Squid Log analysis

# Install the image processing software package 
yum install -y pcre-devel gd gd-devel

mkdir /usr/local/sarg
tar zxvf sarg-2.3.7.tar.gz -C /opt/

cd /opt/sarg-2.3.7
./configure --prefix=/usr/local/sarg \
--sysconfdir=/etc/sarg \							# Profile directory , The default is /usr/local/etc
--enable-extraprotection							# Additional safety protection 

----------------------------------------------------------------------------------------------------------
./configure --prefix=/usr/local/sarg --sysconfdir=/etc/sarg --enable-extraprotection
make && make install

vim /etc/sarg/sarg.conf
--7 That's ok -- uncomment 
access_log /usr/local/squid/var/logs/access.log		# Specify access log file 
--25 That's ok -- uncomment 
title "Squid User Access Reports"					# Webpage title 
--120 That's ok -- uncomment , modify 
output_dir /var/www/html/sarg						# Report output directory 
--178 That's ok -- uncomment 
user_ip no											# Use the user name to display 
--184 That's ok -- uncomment , modify 
topuser_sort_field connect reverse					#top Sorting , Specify the number of connections in descending order , The ascending order is normal
--190 That's ok -- uncomment , modify 
user_sort_field connect reverse						# For user access records , The number of connections is sorted in descending order 
--206 That's ok -- uncomment , modify 
exclude_hosts /usr/local/sarg/noreport				# Specify files that are not included in the sorted site list 
--257 That's ok -- uncomment 
overwrite_report no									# Whether to overwrite logs with the same name and date 
--289 That's ok -- uncomment , modify 
mail_utility mailq.postfix							# Send mail report command 
--434 That's ok -- uncomment , modify 
charset UTF-8										# Specify character set UTF-8
--518 That's ok -- uncomment 
weekdays 0-6										#top Week period of ranking 
--525 That's ok -- uncomment 
hours 0-23											#top The time period of ranking 
--633 That's ok -- uncomment 
www_document_root /var/www/html						# Specify the web root 


# Add not included in the site file , The added domain name will not be displayed in the sort 
touch /usr/local/sarg/noreport

ln -s /usr/local/sarg/bin/sarg /usr/local/bin/
sarg --help

# verification 
yum install httpd -y
systemctl start httpd

# function 
sarg				# Start recording once 


 Browser access  http://192.168.80.10/sarg , see sarg Report page .

# Add scheduled tasks , Perform daily report generation 
vim /usr/local/sarg/report.sh
#/bin/bash
#Get current date
TODAY=$(date +%d/%m/%Y)
#Get one week ago today
YESTERDAY=$(date -d "1 day ago" +%d/%m/%Y)
/usr/local/sarg/bin/sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/sarg -z -d $YESTERDAY-$TODAY &> /dev/null
find ./ -type d -a -name "$(env LANG=en_US.UTF-8 date -d "30 day ago" +%Y%b%d)-$(env LANG=en_US.UTF-8 date -d "29 day ago" +%Y%b%d)" | xargs rm -rf
exit 0


chmod +x /usr/local/sarg/report.sh

crontab -e
0 0 * * * /usr/local/sarg/report.sh

8、 ... and 、Squid Reverse proxy

If Squid The requested resource is cached in the reverse proxy server , The requested resource is returned directly to the client ; Otherwise, the reverse proxy server will go to the background Web Server requests resources , Then return the requested response to the client , At the same time, the response will be stored locally , For the next requester .

Working mechanism ：

• Cache web page objects , Reduce duplicate requests
• Assign the Internet request for rotation training or assign it to the intranet according to the weight Web The server
• Proxy user requests , Avoid direct user access to Web The server , Improve safety

squid The server ：192.168.52.110
web1 The server ：192.168.52.120
web2 The server ：192.168.52.130
 The client ：192.168.52.30

1、 To configure squid The server

1.1 Modify the configuration file

vim /etc/squid.conf
......
--60 That's ok -- modify , Insert 
http_port 192.168.52.110:80 accel vhost vport
cache_peer 192.168.52.130 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web1
cache_peer 192.168.52.130 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web2
cache_peer_domain web1 web2 www.my.com

# Said to www.my.com Request ,squid towards 192.168.52.130 and 192.168.52.120 Of 80 Request from port 

------------------------------------------------------------------------
http_port 80 accel vhost vport 
 ##squid From a cache to a Web Server reverse proxy acceleration mode , This is the time squid stay 80 Port listening requests , At the same time with web server Request port (vhost vport) binding , It's time for the request squid,squid There is no need to forward requests , Instead, you can take data directly from the cache 、 Or request data directly from the bound port .
 accel ： Reverse proxy acceleration mode .
 vhost ： Support domain name or host name to represent proxy node .
 vport ： Support IP And port to represent the proxy node .
 ​
 parent ： Represents the parent node , Up and down relationship , Non horizontal relationship .
 80 ： Agent internal web Server's 80 port .
 0 ： Not used icp（ Telecom operators ）, It means just one squid The server .
 no-query ： No query operation , Get data directly .
 originserver ： Specify the source server .
 round-robin ： Appoint  squid  The request is distributed to one of the parent nodes by polling .
 max_conn ： Specify the maximum number of connections .
 weight ：  Assign weights .
 name ： Set alias .

1.2 Clear firewall rules

iptables -F
iptables -t nat -F

1.3 Turn on squid service

systemctl stop httpd       # prevent  httpd  The service uses  80  The port number and  squid  The listening port configured by the reverse proxy conflicts 
netstat -natp | grep 80
systemctl restart squid

2、web Server configuration

2.1web1 Server configuration

Turn off firewall

systemctl stop firewalld.service 
setenforce 0

  install apache And configure the homepage

yum install -y httpd
systemctl start httpd
echo "this is test01" >> /var/www/html/index.html
systemctl restart httpd

2.2web2 Server configuration

Turn off firewall

systemctl stop firewalld.service 
setenforce 0

  install apache And configure the homepage

yum install -y httpd
systemctl start httpd
echo "this is test02" >> /var/www/html/index.html
systemctl restart httpd

3、 The domain name mapping configuration of the client

3.1 modify C:\Windows\System32\drivers\etc\hosts file

192.168.52.110 www.my.com

3.2 The browser does not open proxy access http://www.my.com

3.3 Check the log

tail -f /usr/local/squid/var/logs/access.log