The Kubernetes Auditing function provides a security related, chronological record set that records the activities triggered by each user, applications using the Kubernetes API, and the control surface itself.

The audit function enables the Cluster Administrator to answer the following questions:

• What happened?
• When did it happen?
• Who triggered it?
• On which object (s) does the activity occur?
• Where did you observe it?
• Where did it trigger?
• What is the follow-up behavior of the activity

Kubernetes without audit policy

When enabled, audit records begin their lifecycle within the Kube apiserver component. Each request generates an audit event at each stage of its execution, and then preprocesses it according to specific policies and writes it to the back end. This policy determines the content of the record, and the back end will keep the record. Current backend implementations include log files and webhooks.

Audit strategy

Audit records were originally generated in kube-apiserver Inside. Each request will generate an audit event at different execution stages; These audit events are preprocessed and written to the back end according to specific policies. The policy determines what to record and the backend used to store records. The current backend supports log files and webhook.

Each request can be recorded in its associated stage. The defined phases are:

• RequestReceived - this phase corresponds to the event generated after the audit processor receives the request and before delegating to other processors.
• ResponseStarted - the event generated after the header of the response message is sent and before the body of the response message is sent. Only long-running requests (such as watch) generate this stage.
• ResponseComplete - when the response message body is complete and no more data needs to be transmitted.
• Panic - generated when panic occurs.

The audit logging function increases the memory consumption of the API server because some context required for audit needs to be stored for each request. In addition, memory consumption depends on the configuration of audit logging.

The audit policy defines rules about what events should be recorded and what data should be included. The audit policy object structure is defined in audit.k8s.io API Group When events are processed, they are compared with the rule list in order. The first matching rule sets the Audit Level of the event. The defined audit levels are:

• None - logs that meet this rule will not be logged.
• Metadata - records the metadata of the request (requested user, timestamp, resource, verb, etc.), but does not record the message body of the request or response.
• Request - records the metadata of the event and the message body of the request, but does not record the message body of the response. This does not apply to non resource type requests.
• RequestResponse - records the metadata of the event and the message body of the request and response. This does not apply to non resource type requests.

You can use the -- audit policy file flag to pass the file containing the policy to Kube apiserver. If this flag is not set, no event is logged. Note that the rules field must be provided in the audit policy file. Policies without (0) rules will be considered illegal configuration.

The following is an example of an audit policy file:

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

You can use the minimum audit policy file to record all requests at the Metadata level:

# Generate logs for all requests at the Metadata level
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata

K8sMeetup

Architecture - Audit Strategy in Kubernetes

 

Ubernets has an audit policy enabled

We can use Webhooks to send audit logs to files or remote web APIs.

In this article, we will force Kube API server to send audit logs to files.

K8sMeetup

Enable audit policy in Kubernetes (for audit log files)

1. Create an audit policy YAML file: go to the Kubernetes cluster and create audit policy using the following rules YAML:

apiVersion: audit.k8s.io/v1
kind: Policy
rules: 
  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    omitStages:
      - "RequestReceived"

Update Kube API server manifest file:

- kube-apiserver
    - --advertise-address=10.156.0.6
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/audit.log 
 ---
 - mountPath: /etc/kubernetes/audit-policy.yaml
   name: audit
   readOnly: true
 — mountPath: /var/log/audit.log
   name: audit-log
   readOnly: false
---
volumes:
  - name: audit
    hostPath:
     path: /etc/kubernetes/audit-policy.yaml
     type: File
  - name: audit-log
    hostPath:
      path: /var/log/audit.log
      type: FileOrCreate

reference resources: https://kubernetes.io/zh/docs/tasks/debug-application-cluster/audit/

https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/

https://kubernetes.io/zh/docs/reference/config-api/apiserver-audit.v1/#audit-k8s-io-v1-Event