By Kasun Talwatta Translator | Luga Lee Planning | Luga Lee

Service grid projects such as Istio have introduced many functions and advantages into our architecture, including more secure management of traffic between cluster micro services, service discovery, request routing and reliable communication between services.

Although Istio is a neutral platform, it has become one of the more popular service grids used with Kubernetes. Despite this popularity, it may be complex for novices in service grid to understand Istio's network and core mechanisms, such as:

1. Envoy Sidecar proxy injection

2. How does Sidecar intercept and route traffic

3. Distribute traffic management configuration

4. How do traffic rules take effect on the data plane

In the first article in a series of blogs that explain these mechanisms by analyzing Istio's architecture and implementation mechanisms, we will introduce Istio's network basics, data plane and control plane, network and Sidecar injection using Envoy agent. Using the demo environment, we will be able to see how Istio injects Init and Sidecar containers and how these containers are configured in the Pod template.

Istio network foundation

Istio's overview has been widely introduced in the official documents, but before we go further, we will focus on the key components.

Istio consists of two main parts: data plane and control plane.

Data plane: the data plane or data layer consists of a set of proxy services, which are represented as side car containers in each Kubernetes Pod, using an extended Envoy proxy server. These side cars mediate and control all network communication between microservices, and also collect and report useful telemetry data.

Control plane: the control plane or control layer consists of a binary file named istiod, which is responsible for converting advanced routing rules and flow control behaviors into Envoy specific configurations, and then propagating them to Sidecar at run time. In addition, the control plane provides security measures, realizes powerful service to service and end-user authentication through built-in identity and credential management, and implements security policies according to service identity.

Istio environment demo

Before continuing, let's create a local sandbox environment. This will ensure that we have deployed the Istio service grid in Kubernetes and run a sample application in the grid.

The required tools are as follows:

• minikube
• istioctl (Installed with curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.11.4 sh)

Steps to deploy Istio Service Grid:

1. Use the hyperkit driver to create 1.22.0 locally Version 2 of Kubernetes cluster. If you are using a non Mac OS X machine, you need to install virtualbox.

[administrator@JavaLangOutOfMemory ~ ] % minikube start --memory=4096 --cpus=2 --disk-size='20gb' --kubernetes-version=1.22.2 --driver=hyperkit -p istio-demo

2. After the cluster is fully started, execute the following command to set Istio.

# Deploy Istio operator
[administrator@JavaLangOutOfMemory ~ ] % istioctl operator init

# Inject operator configuration
[administrator@JavaLangOutOfMemory ~ ] % cat << EOF | kubectl apply -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-control-plane
  namespace: istio-system
spec:
  profile: minimal
  meshConfig:
    accessLogFile: /dev/stdout
    enableAutoMtls: true
    defaultConfig:
      proxyMetadata:
        # Enable basic DNS proxying
        ISTIO_META_DNS_CAPTURE: 'true'
        # Enable automatic address allocation
        ISTIO_META_DNS_AUTO_ALLOCATE: 'true'
EOF

3. Deploy the sample application.

# Create apps namespace
[administrator@JavaLangOutOfMemory ~ ] % kubectl create ns apps
# Label apps namespace for sidecar auto injection
[administrator@JavaLangOutOfMemory ~ ] % kubectl label ns apps istio-injection=enabled

# Deploy a unprivileged sleep application
[administrator@JavaLangOutOfMemory ~ ] % cat << EOF | kubectl apply -n apps -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - name: http
    port: 80
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: secret-volume
          mountPath: /etc/sleep/tls
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true
EOF

4. Verify that the istio init and istio proxy containers are ready and running.

[administrator@JavaLangOutOfMemory ~ ] % kubectl get po -l app=sleep -n apps -o jsonpath='{range .items[*]}{range @.status.containerStatuses[*]}{.name},{"ready="}{.ready},{"started="}{.started}{"\n"}{end}{range @.status.initContainerStatuses[*]}{.name},{"ready="}{.ready},{"terminated="}{.state.terminated.reason}{end}' | sort

The results are shown as follows:

istio-init,ready=true,terminated=Completed
istio-proxy,ready=true,started=true

Istio Sidecar container and Envoy agent

Sidecar injection is one of the key functions in Istio, which simplifies the process of adding and running additional containers as part of the Pod template. As part of this injection process, two additional containers are also provided:

1. Istio Init – this container configures iptables in the application Pod so that the Envoy agent (running as a separate container) can intercept inbound and outbound traffic. Before any other container starts, Kubernetes runs it as an Init container to initialize the network in the Pod. Note that allowing istio Init to manipulate iptables in kernel space does require upgrading Kubernetes permissions. Once the task is successfully completed, the container will automatically terminate. The Pod will not be ready until then. Note that in order to eliminate any security issues and operational challenges when deploying this container, istio introduces the CNI plug-in, so it can integrate directly with the underlying Kubernetes CNI without operating iptables.

2. Istio proxy – packaged as an extended version of upstream Envoy proxy. For a list of supported extensions, see the official documentation.

In depth inspection of Sidecar list

Let's take a look at the YAML listings of these two containers in our previously deployed application Pod.

[administrator@JavaLangOutOfMemory ~ ] % kubectl get po -l app=sleep -n apps -o yaml

We'll look at excerpts from the istio init and istio proxy containers.

istio-init Container:

initContainers:
- name: istio-init
  image: docker.io/istio/proxyv2:1.11.4
  imagePullPolicy: IfNotPresent
  args:
  - istio-iptables
  - -p
  - "15001"
  - -z
  - "15006"
  - -u
  - "1337"
  - -m
  - REDIRECT
  - -i
  - '*'
  - -x
  - ""
  - -b
  - '*'
  - -d
  - 15090,15021,15020
  env:
  - name: ISTIO_META_DNS_AUTO_ALLOCATE
    value: "true"
  - name: ISTIO_META_DNS_CAPTURE
    value: "true"
  resources:
    limits:
      cpu: "2"
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      add:
      - NET_ADMIN
      - NET_RAW
      drop:
      - ALL
    privileged: false
    readOnlyRootFilesystem: false
    runAsGroup: 0
    runAsNonRoot: false
    runAsUser: 0

istio-proxy Container:

containers:
- name: istio-proxy
  image: docker.io/istio/proxyv2:1.11.4
  imagePullPolicy: IfNotPresent
  args:
  - proxy
  - sidecar
  - --domain
  - $(POD_NAMESPACE).svc.cluster.local
  - --proxyLogLevel=warning
  - --proxyComponentLogLevel=misc:error
  - --log_output_level=default:info
  - --concurrency
  - "2"
  ports:
  - name: http-envoy-prom
    containerPort: 15090
    protocol: TCP
  readinessProbe:
    httpGet:
      path: /healthz/ready
      port: 15021
      scheme: HTTP
    failureThreshold: 30
    initialDelaySeconds: 1
    periodSeconds: 2
    successThreshold: 1
    timeoutSeconds: 3
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    privileged: false
    readOnlyRootFilesystem: true
    runAsGroup: 1337
    runAsNonRoot: true
    runAsUser: 1337
  env:
  - name: PROXY_CONFIG
    value: |
      {"proxyMetadata":{"ISTIO_META_DNS_AUTO_ALLOCATE":"true","ISTIO_META_DNS_CAPTURE":"true"}}
  - name: ISTIO_META_DNS_AUTO_ALLOCATE
    value: "true"
  - name: ISTIO_META_DNS_CAPTURE
    value: "true"
  ...

There are some interesting key points to note in these additional parameters:

Both containers are served by the same image: docker io/istio/proxyv2:1.11. What does this mean and how does it work? Istio iptables and proxy (under args) commands are loaded into the pilot agent binary in the image. Therefore, if we run the pilot agent binary in the istio proxy container, we will see the following:

kubectl exec $(kubectl get po -l app=sleep -n apps -o jsonpath="{.items[0]. metadata.name}") -n apps -c istio-proxy --pilot-agent

Istio Pilot agent runs in the sidecar or gateway container and bootstraps Envoy.

Usage:
  pilot-agent [command]

Available Commands:
  completion           generate the autocompletion script for the specified shell
  help                 Help about any command
  istio-clean-iptables Clean up iptables rules for Istio Sidecar
  istio-iptables       Set up iptables rules for Istio Sidecar
  proxy                XDS proxy agent
  request              Makes an HTTP request to the Envoy admin API
  version              Prints out build version information
  wait                 Waits until the Envoy proxy is ready

To minimize the attack surface, the securityContext section in the Istio init container (which is part of the PodSecurityContext object) indicates that the container runs with root permission (runAsUser: 0), but except NET_ADMIN and NET_RAW function. These capabilities provide runtime permissions for the Istio init init container to override the iptables of the application Pod. This is described in more detail in the Istio documentation.

allowPrivilegeEscalation: false
    capabilities:
      add:
      - NET_ADMIN
      - NET_RAW
      drop:
      - ALL
    privileged: false
    readOnlyRootFilesystem: false
    runAsGroup: 0
    runAsNonRoot: false
    runAsUser: 0

On the other hand, the istio proxy container runs with the restricted privileges of 1337 users. Since this is reserved, the UID (user ID) of the application workload must be different and must not conflict with 1337. 1337 UID has been arbitrarily selected. Istio team bypasses traffic and redirects to istio proxy container. We can also see that 1337 is used as a parameter of istio iptables when initializing iptables. Since this container runs actively with the application workload, istio also ensures that if it is threatened, it can only have read-only access to the root file system.

allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    privileged: false
    readOnlyRootFilesystem: true
    runAsGroup: 1337
    runAsNonRoot: true
    runAsUser: 1337

The istio proxy container runs with the ready probe shown below. Kubelet in Kubernetes uses this ready probe to determine whether istio proxy is ready to accept traffic. Kubelet will recognize that the Pod is ready only when the istio proxy container and all corresponding application containers are running and the health probe has been successfully executed. If the / health / ready handler of the server path (defined in the pilot agent source code) returns a successful return code, kubelet will assume that the container is active and healthy. The failureThreshold configuration specifies the number of consecutive times this ready probe can fail before the container is marked as not ready.

readinessProbe:
    httpGet:
      path: /healthz/ready
      port: 15021
      scheme: HTTP
    initialDelaySeconds: 1
    failureThreshold: 30
    periodSeconds: 2
    successThreshold: 1
    timeoutSeconds: 3

Sidecar injection analysis

Istio injects the Sidecar agent into the application workload in two different ways: manual and automatic. Both methods follow the same injection principle. Given "some" application workloads (which can be defined as higher-level Kubernetes resources, such as deployment, Statefulset, daemon set and even Pod), Kubernetes is allowed to inject Sidecar container templates and configuration parameters (isto Sidecar injector configmap) using Sidecar.

Manual side car injection in Istio

Of the two methods, this is the easiest to understand. Manual injection is done by the istioctl command using the Kube inject parameter. We can use any of the following formats for injection:

[administrator@JavaLangOutOfMemory ~ ] % istioctl kube-inject -f application.yaml | kubectl apply -f -

or

[administrator@JavaLangOutOfMemory ~ ] % kubectl apply -f <(istioctl kube-inject -f application.yaml)

When istioctl Kube inject is used to inject Sidecar, it will use the intra cluster configuration written as istio Sidecar injector kubernetes configmap by default. There are many flags that we can specify to customize this behavior:

--injectConfigFile string    Injection configuration filename. Cannot be used with --injectConfigMapName
--meshConfigFile string      Mesh configuration filename. Takes precedence over --meshConfigMapName if set
--meshConfigMapName string   ConfigMap name for Istio mesh configuration, key should be "mesh" (default "istio")
--injectConfigMapNam string  ConfigMap name for Istio sidecar injection, key should be "config" (default "istio-sidecar-injector")

Please note that, - injectConfigMapNam is a hidden flag in istioctl Kube inject, which allows us to override Sidecar injection configuration in the cluster. Alternatively, you can use the configured local copy and the above flag to complete the injection:

[administrator@JavaLangOutOfMemory ~ ] % kubectl -n istio-system get configmap istio-sidecar-injector -o=jsonpath='{.data.config}' > inject-config.yaml
[administrator@JavaLangOutOfMemory ~ ] % kubectl -n istio-system get configmap istio-sidecar-injector -o=jsonpath='{.data.values}' > inject-values.yaml
[administrator@JavaLangOutOfMemory ~ ] % kubectl -n istio-system get configmap istio -o=jsonpath='{.data.mesh}' > mesh-config.yaml
[administrator@JavaLangOutOfMemory ~ ] % istioctl kube-inject \
    --injectConfigFile inject-config.yaml \
    --meshConfigFile mesh-config.yaml \
    --valuesFile inject-values.yaml \
    --filename application.yaml \
    | kubectl apply -f -

During manual injection, care must be taken not to destroy Sidecar, especially when using custom configuration.

Automatic side car injection in Istio

This is considered to be the de facto method of injecting side cars into Istio. This involves fewer configuration steps than the manual method; However, it depends on whether the underlying Kubernetes distribution enables support for admission controllers. To this end, Istio uses a variant webhook admission controller.

The following is the process of Kubernetes mutation admission controller in Sidecar injection:

1. First, the Istio sidecar injector mutating configuration injected during Istio installation (as shown below) sends a webhook request with all Pod information to the Istio controller.

2. Next, the controller modifies the Pod specification at runtime and introduces Init and Sidecar container agent into the actual Pod specification.

3. Then, the controller returns the modified object to Webhook for object verification.

4. Finally, after verification, the modified Pod specification is deployed with all Sidecar containers.

For the complete configuration, see kubectl get mutatingwebhookconfiguration istio sidecar injector - O yaml. For brevity, only two of the four webhook configurations are given in the following excerpt:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: istio-sidecar-injector
webhooks:
- admissionReviewVersions:
  - v1beta1
  - v1
  clientConfig:
    caBundle: cert
    service:
      name: istiod
      namespace: istio-system
      path: /inject
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: namespace.sidecar-injector.istio.io
  namespaceSelector:
    matchExpressions:
    - key: istio-injection
      operator: In
      values:
      - enabled
  objectSelector:
    matchExpressions:
    - key: sidecar.istio.io/inject
      operator: NotIn
      values:
      - "false"
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10
- admissionReviewVersions:
  - v1beta1
  - v1
  clientConfig:
    caBundle: cert
    service:
      name: istiod
      namespace: istio-system
      path: /inject
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: namespace.sidecar-injector.istio.io
  namespaceSelector:
    matchExpressions:
    - key: istio-injection
      operator: In
      values:
      - enabled
  objectSelector:
    matchExpressions:
    - key: sidecar.istio.io/inject
      operator: NotIn
      values:
      - "false"
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10
- admissionReviewVersions:
  - v1beta1
  - v1
  clientConfig:
    caBundle: cert
    service:
      name: istiod
      namespace: istio-system
      path: /inject
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: object.sidecar-injector.istio.io
  namespaceSelector:
    matchExpressions:
    - key: istio-injection
      operator: DoesNotExist
    - key: istio.io/rev
      operator: DoesNotExist
  objectSelector:
    matchExpressions:
    - key: sidecar.istio.io/inject
      operator: In
      values:
      - "true"
    - key: istio.io/rev
      operator: DoesNotExist
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

This configuration tells the Kubernetes mutation controller to safely send requests to the / inject endpoint of the istiod service on the HTTPS port. Before calling mutating webhook, Kubernetes checks whether the requesting user has the right to make the request. In Istio, webhook is implemented as part of the istiod binary.

Injection can be triggered using namespace level tags (istio injection = enabled) or as annotations at the object level (sidecar.istio.io/inject="true"). Each webhook configuration defines the matching rules for these triggers in the namespaceSelector and objectSelector. When injecting tags defined based on the namespace level, any Deployment objects (Deployment, StatefulSet, DaemonSet) created in the namespace will be mutated using the Sidecar agent. The following is a summary of the matching rules:

When the pod list is injected, the pod object can also be changed directly (if the namespace has no label). Pod list must have a label sidecar istio. io/inject="true". For example:

apiVersion: v1
kind: Pod
metadata:
  name: sleep
  namespace: apps
  labels:
    app: sleep
    sidecar.istio.io/inject: "true"
...

So far, we have learned about Istio's basic network knowledge, data plane and control plane, network and Sidecar injection using Envoy agent, how Istio uses demonstration to inject Init and Sidecar containers, and the configuration environment of these containers in Pod template. In the next blog, we will analyze how iptables are configured and managed.

Solo on Slack Contact us on Istio for more information about Istio and our products. We also offer many webinars and seminars about Istio, so please feel free to register for more information.

Original text: https://www.solo.io/blog/istios-networking-in-depth/?utm_source=thenewstack&utm_medium=website&utm_campaign=platform