Opening
The article about Istio traffic hijacking can be found in a very detailed article in the servicemesher community at present. You can refer to: Sidecar injection and transparent traffic hijacking in Istio . In particular, the "traffic hijacking diagram" collated by the blogger can clearly see the hijacking process. Here I use that picture to explain a text version of the picture. Before starting the text version, if you don't know the iptables command very well, it's recommended to focus on the following two articles to explain the concept and usage of the command in a simple way:
- iptables concept -Describe iptables related concepts in an easy to understand way
- iptables Guide -iptables command usage guide
Here is a message flow chart of iptables (copyright belongs to the original blogger)
When the client accesses the web service of the server, the client sends the message to the network card, and the tcp/ip protocol stack belongs to a part of the kernel. Therefore, the information of the client will be transmitted to the web service in the user space through the TCP protocol of the kernel. At this time, the destination of the client message is the socket (IP: Port) monitored by the web service. When the web service needs to respond to the client When the client requests, the destination of the response message sent by the web service is the client. At this time, the IP and Port monitored by the web service become the origin. --Quoted from zsythink
The above description is very important, which is one of the bases to understand sidecar's traffic hijacking.
Let's analyze the command istio iptables executed yesterday when istio init was started
nsenter -t 8533 -n iptables -t nat -S # default # Set the policy to receive packets (ACCEPT) for the preceding / input / output / posting chain -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT # Customize the rule chain of 4 istio s -N ISTIO_INBOUND -N ISTIO_IN_REDIRECT -N ISTIO_OUTPUT -N ISTIO_REDIRECT # Enter the pre routing chain tcp protocol and all requests are directed to ISTIO_INBOUND custom chain for rule matching -A PREROUTING -p tcp -j ISTIO_INBOUND # All tcp protocol requests entering the OUTPUT chain are directed to ISTIO_OUTPUT custom chain for rule matching -A OUTPUT -p tcp -j ISTIO_OUTPUT # entrance # The tcp protocol request with the request port of 22 / 15090 / 15021 / 15020 stops executing the subsequent Rules in the current chain and executes the next chain -A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN -A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN -A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN -A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN # All requests with tcp protocol and port not 22 / 15090 / 15021 / 15020 are directed to ISTIO_IN_REDIRECT -A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT # Redirect all the tcp protocol request traffic redirected to this port to 15006 (envoy inlet traffic port) -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006 # exit # The traffic whose source IP address is localhost and packet exit is "lo" is returned to the next chain execution in its call point (1) -A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN # The destination is not localhost, the packet exit is "lo", which is the traffic of istio proxy users forwarded to ISTIO_REDIRECT (2) -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT # The packet exit is "lo", and the traffic of non istio proxy users is returned to the next chain execution in its call point (1) -A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN # Istio proxy users' traffic is returned to the next chain execution in its call point -A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN # The destination is not localhost, the packet exit is "lo", which means the traffic of istio proxy user group is forwarded to ISTIO_REDIRECT(2) -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT # Packet exit is "lo" and non istio proxy user group traffic is returned to the next chain execution in its call point (1) -A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN # Traffic to the istio proxy user group is returned to the next chain execution in its call point (1) -A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN # All traffic destined for localhost is returned to the next chain execution in its call point (1) -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN # All other traffic that does not meet the above rules is forwarded to ISTIO_REDIRECT (2) -A ISTIO_OUTPUT -j ISTIO_REDIRECT # Redirect all the tcp protocol request traffic redirected to this port to 15001 (envoy outlet traffic port) -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-M = -- match. Istio proxy runs as the user identity. Uid owner 1337 is the user ID / GID owner 1337 is the user group, that is, the user space where sidecar is located. This is also the default user used by the istio proxy container.
Pay attention to the parentheses in the text
(1) The representative traffic will directly execute the next interception chain. In this paper, the next interception chain is the post routing chain
(2) Delegate traffic will be redirected to the envoy outlet traffic port
According to the above rules, summarize:
ISTIO_INBOUND chain: all traffic entering Pod but not specified port (such as 22) is redirected to 15006 port (envoy inlet traffic port) for interception.
ISTIO_OUTPUT chain: all outgoing pods sent by istio proxy user space and not to localhost are redirected to port 15001 (envoy export traffic port), and all other traffic is directly released to the next post routing chain without being intercepted by envoy.
In fact, after careful consideration, we can see that traffic interception mainly occurs in two places:
- When the user requests to arrive at Pod, the corresponding traffic will be blocked to sidecar for processing, and sidecar requests the business service
- When the business service responds to the user's request, the response is blocked to sidecar again, and the sidecar responds to the user
Look at the rules of iptables nat table again
nsenter -t 8533 -n iptables -t nat -L -v Chain PREROUTING (policy ACCEPT 3435 packets, 206K bytes) pkts bytes target prot opt in out source destination 3435 206K ISTIO_INBOUND tcp -- any any anywhere anywhere (1) Chain INPUT (policy ACCEPT 3435 packets, 206K bytes) (5) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 599 packets, 54757 bytes) pkts bytes target prot opt in out source destination 22 1320 ISTIO_OUTPUT tcp -- any any anywhere anywhere Chain POSTROUTING (policy ACCEPT 599 packets, 54757 bytes) (8) pkts bytes target prot opt in out source destination Chain ISTIO_INBOUND (1 references) (2) pkts bytes target prot opt in out source destination 0 0 RETURN tcp -- any any anywhere anywhere tcp dpt:22 1 60 RETURN tcp -- any any anywhere anywhere tcp dpt:15090 3434 206K RETURN tcp -- any any anywhere anywhere tcp dpt:15021 0 0 RETURN tcp -- any any anywhere anywhere tcp dpt:15020 0 0 ISTIO_IN_REDIRECT tcp -- any any anywhere anywhere (3) Chain ISTIO_IN_REDIRECT (3 references) pkts bytes target prot opt in out source destination 0 0 REDIRECT tcp -- any any anywhere anywhere redir ports 15006 (4) Chain ISTIO_OUTPUT (1 references) (6) pkts bytes target prot opt in out source destination 0 0 RETURN all -- any lo 127.0.0.6 anywhere 0 0 ISTIO_IN_REDIRECT all -- any lo anywhere !localhost owner UID match 1337 0 0 RETURN all -- any lo anywhere anywhere ! owner UID match 1337 22 1320 RETURN all -- any any anywhere anywhere owner UID match 1337 0 0 ISTIO_IN_REDIRECT all -- any lo anywhere !localhost owner GID match 1337 0 0 RETURN all -- any lo anywhere anywhere ! owner GID match 1337 0 0 RETURN all -- any any anywhere anywhere owner GID match 1337 0 0 RETURN all -- any any anywhere localhost 0 0 ISTIO_REDIRECT all -- any any anywhere anywhere Chain ISTIO_REDIRECT (1 references) pkts bytes target prot opt in out source destination 0 0 REDIRECT tcp -- any any anywhere anywhere redir ports 15001
Let's take a closer look at the following picture (the copyright belongs to the original blogger) and observe the rules of iptables chain synchronously. The analysis here focuses on one-to-one interpretation of red numbers:
- The productpage service sends a TCP connection request to the reviews service
- The request enters the Pod kernel space where the reviews service is located, is intercepted by netfilter, passes through the preouting chain, and then forwarded to ISTIO_INBOUND chain
- At istio_ The inbound chain is defined by this rule - A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT intercepts and forwards to istio again_ IN_ Redirect chain
- ISTIO_IN_REDIRECT chain is directly redirected to the 15006 inlet traffic port monitored by envoy
- After a series of inbound traffic governance actions, the envoy sends a TCP connection request to review service. This step belongs to outbound traffic for envoy and will be intercepted by netfilter and forwarded to the outbound traffic OUTPUT chain
- OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
- The destination is localhost, which can't be matched to the forwarding rule chain. RETURN directly to the next chain, i.e. posting chain
- The request from sidecar arrives at the views service 9080 port
- After processing the business logic, the reviews service responds to sidecar. This step belongs to the export traffic for the reviews service, which is intercepted by netfilter again and forwarded to the export traffic OUTPUT chain
- OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
- Traffic that sends non localhost requests and is istio proxy user space is forwarded to ISTIO_REDIRECT chain
- ISTIO_REDIRECT chain is directly redirected to the 15001 outlet traffic port monitored by envoy
- After a series of export traffic governance actions, envoy continues to send response data, which will be intercepted by netfilter and forwarded to the export traffic OUTPUT chain
- OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain
- The traffic is directly returned to the next chain, i.e. post routing chain
In view of the above, I still have two questions. Please give me some advice:
- The above understanding did not write point 16. The 16 points in the blogger's figure will be added to istio again_ Redirect chain, we can see istio_ There is only one rewrite port forwarding rule in the redirect chain, so it will enter a dead cycle? Or I don't understand
- Whether the envoy forwarding traffic is a newly established tcp connection request by itself or by modifying the request message address. Because of the limited understanding of c + +, we can't look up the source code to find out
From the whole flow interception process, you can see that such a long path will definitely lose forwarding performance in a large concurrency scenario. At present, there are some frameworks in the industry trying to shorten the interception path. Let's wait and see.
reference
https://www.servicemesher.com/blog/sidecar-injection-iptables-and-traffic-routing/
https://www.frozentux.net/iptables-tutorial/cn/iptables-tutorial-cn-1.1.19.html#REDIRECTTARGET