Istio traffic hijacking process

Opening

The article about Istio traffic hijacking can be found in a very detailed article in the servicemesher community at present. You can refer to: Sidecar injection and transparent traffic hijacking in Istio . In particular, the "traffic hijacking diagram" collated by the blogger can clearly see the hijacking process. Here I use that picture to explain a text version of the picture. Before starting the text version, if you don't know the iptables command very well, it's recommended to focus on the following two articles to explain the concept and usage of the command in a simple way:

  1. iptables concept -Describe iptables related concepts in an easy to understand way
  2. iptables Guide -iptables command usage guide

Here is a message flow chart of iptables (copyright belongs to the original blogger)

When the client accesses the web service of the server, the client sends the message to the network card, and the tcp/ip protocol stack belongs to a part of the kernel. Therefore, the information of the client will be transmitted to the web service in the user space through the TCP protocol of the kernel. At this time, the destination of the client message is the socket (IP: Port) monitored by the web service. When the web service needs to respond to the client When the client requests, the destination of the response message sent by the web service is the client. At this time, the IP and Port monitored by the web service become the origin. --Quoted from zsythink

The above description is very important, which is one of the bases to understand sidecar's traffic hijacking.

Let's analyze the command istio iptables executed yesterday when istio init was started

nsenter -t 8533 -n iptables -t nat -S

# default
# Set the policy to receive packets (ACCEPT) for the preceding / input / output / posting chain
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT

# Customize the rule chain of 4 istio s
-N ISTIO_INBOUND
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-N ISTIO_REDIRECT

# Enter the pre routing chain tcp protocol and all requests are directed to ISTIO_INBOUND custom chain for rule matching
-A PREROUTING -p tcp -j ISTIO_INBOUND
# All tcp protocol requests entering the OUTPUT chain are directed to ISTIO_OUTPUT custom chain for rule matching
-A OUTPUT -p tcp -j ISTIO_OUTPUT

# entrance
# The tcp protocol request with the request port of 22 / 15090 / 15021 / 15020 stops executing the subsequent Rules in the current chain and executes the next chain
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
# All requests with tcp protocol and port not 22 / 15090 / 15021 / 15020 are directed to ISTIO_IN_REDIRECT
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
# Redirect all the tcp protocol request traffic redirected to this port to 15006 (envoy inlet traffic port)
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006

# exit
#  The traffic whose source IP address is localhost and packet exit is "lo" is returned to the next chain execution in its call point (1)
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
#  The destination is not localhost, the packet exit is "lo", which is the traffic of istio proxy users forwarded to ISTIO_REDIRECT (2)
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
#  The packet exit is "lo", and the traffic of non istio proxy users is returned to the next chain execution in its call point (1)
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
#  Istio proxy users' traffic is returned to the next chain execution in its call point
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
#  The destination is not localhost, the packet exit is "lo", which means the traffic of istio proxy user group is forwarded to ISTIO_REDIRECT(2)
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
#  Packet exit is "lo" and non istio proxy user group traffic is returned to the next chain execution in its call point (1)
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
#  Traffic to the istio proxy user group is returned to the next chain execution in its call point (1)
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
#  All traffic destined for localhost is returned to the next chain execution in its call point (1)
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
#  All other traffic that does not meet the above rules is forwarded to ISTIO_REDIRECT  (2)
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
#  Redirect all the tcp protocol request traffic redirected to this port to 15001 (envoy outlet traffic port)
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001

-M = -- match. Istio proxy runs as the user identity. Uid owner 1337 is the user ID / GID owner 1337 is the user group, that is, the user space where sidecar is located. This is also the default user used by the istio proxy container.

Pay attention to the parentheses in the text

(1) The representative traffic will directly execute the next interception chain. In this paper, the next interception chain is the post routing chain

(2) Delegate traffic will be redirected to the envoy outlet traffic port

According to the above rules, summarize:

ISTIO_INBOUND chain: all traffic entering Pod but not specified port (such as 22) is redirected to 15006 port (envoy inlet traffic port) for interception.

ISTIO_OUTPUT chain: all outgoing pods sent by istio proxy user space and not to localhost are redirected to port 15001 (envoy export traffic port), and all other traffic is directly released to the next post routing chain without being intercepted by envoy.

In fact, after careful consideration, we can see that traffic interception mainly occurs in two places:

  1. When the user requests to arrive at Pod, the corresponding traffic will be blocked to sidecar for processing, and sidecar requests the business service
  2. When the business service responds to the user's request, the response is blocked to sidecar again, and the sidecar responds to the user

Look at the rules of iptables nat table again

nsenter -t 8533 -n iptables -t nat -L -v


Chain PREROUTING (policy ACCEPT 3435 packets, 206K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 3435  206K ISTIO_INBOUND  tcp  --  any    any     anywhere             anywhere      (1)       

Chain INPUT (policy ACCEPT 3435 packets, 206K bytes)                                  (5)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 599 packets, 54757 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   22  1320 ISTIO_OUTPUT  tcp  --  any    any     anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT 599 packets, 54757 bytes)                            (8)
 pkts bytes target     prot opt in     out     source               destination         

Chain ISTIO_INBOUND (1 references)                                                    (2) 
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:22
    1    60 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15090
 3434  206K RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15021
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15020
    0     0 ISTIO_IN_REDIRECT  tcp  --  any    any     anywhere             anywhere  (3)        

Chain ISTIO_IN_REDIRECT (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REDIRECT   tcp  --  any    any     anywhere             anywhere             redir ports 15006                                                                     (4)

Chain ISTIO_OUTPUT (1 references)                                                     (6)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  any    lo      127.0.0.6            anywhere            
    0     0 ISTIO_IN_REDIRECT  all  --  any    lo      anywhere            !localhost            owner UID match 1337
    0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner UID match 1337
   22  1320 RETURN     all  --  any    any     anywhere             anywhere             owner UID match 1337
    0     0 ISTIO_IN_REDIRECT  all  --  any    lo      anywhere            !localhost            owner GID match 1337
    0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner GID match 1337
    0     0 RETURN     all  --  any    any     anywhere             anywhere             owner GID match 1337
    0     0 RETURN     all  --  any    any     anywhere             localhost           
    0     0 ISTIO_REDIRECT  all  --  any    any     anywhere             anywhere            

Chain ISTIO_REDIRECT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REDIRECT   tcp  --  any    any     anywhere             anywhere             redir ports 15001

Let's take a closer look at the following picture (the copyright belongs to the original blogger) and observe the rules of iptables chain synchronously. The analysis here focuses on one-to-one interpretation of red numbers:

  1. The productpage service sends a TCP connection request to the reviews service
  2. The request enters the Pod kernel space where the reviews service is located, is intercepted by netfilter, passes through the preouting chain, and then forwarded to ISTIO_INBOUND chain
  3. At istio_ The inbound chain is defined by this rule - A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT intercepts and forwards to istio again_ IN_ Redirect chain
  4. ISTIO_IN_REDIRECT chain is directly redirected to the 15006 inlet traffic port monitored by envoy
  5. After a series of inbound traffic governance actions, the envoy sends a TCP connection request to review service. This step belongs to outbound traffic for envoy and will be intercepted by netfilter and forwarded to the outbound traffic OUTPUT chain
  6. OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
  7. The destination is localhost, which can't be matched to the forwarding rule chain. RETURN directly to the next chain, i.e. posting chain
  8. The request from sidecar arrives at the views service 9080 port
  9. After processing the business logic, the reviews service responds to sidecar. This step belongs to the export traffic for the reviews service, which is intercepted by netfilter again and forwarded to the export traffic OUTPUT chain
  10. OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
  11. Traffic that sends non localhost requests and is istio proxy user space is forwarded to ISTIO_REDIRECT chain
  12. ISTIO_REDIRECT chain is directly redirected to the 15001 outlet traffic port monitored by envoy
  13. After a series of export traffic governance actions, envoy continues to send response data, which will be intercepted by netfilter and forwarded to the export traffic OUTPUT chain
  14. OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
  15. The traffic is directly returned to the next chain, i.e. post routing chain

In view of the above, I still have two questions. Please give me some advice:

  • The above understanding did not write point 16. The 16 points in the blogger's figure will be added to istio again_ Redirect chain, we can see istio_ There is only one rewrite port forwarding rule in the redirect chain, so it will enter a dead cycle? Or I don't understand
  • Whether the envoy forwarding traffic is a newly established tcp connection request by itself or by modifying the request message address. Because of the limited understanding of c + +, we can't look up the source code to find out

From the whole flow interception process, you can see that such a long path will definitely lose forwarding performance in a large concurrency scenario. At present, there are some frameworks in the industry trying to shorten the interception path. Let's wait and see.

reference

https://www.servicemesher.com/blog/sidecar-injection-iptables-and-traffic-routing/

https://www.frozentux.net/iptables-tutorial/cn/iptables-tutorial-cn-1.1.19.html#REDIRECTTARGET

http://www.zsythink.net/archives/1199

Keywords: Programming iptables network socket

Added by river001 on Fri, 29 May 2020 09:27:35 +0300