Kubernetes provides a security relevant chronological set of records documenting the sequence of activities that have affected the system by individual users and actions take on their behalf by the system.
This feature is enabled with the following steps:
1) Audit policy – This is the main audit policy. We don’t specify it as a file with –audit-policy-file, rather we define the following:
a. Log. Metadata – this logs requests metadata but not the request or response itself. We define policies for all of Kubernetes core group resources.
audit/audit-policy.yaml:
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  - level: RequestResponse
    resources:
    - group: ""
    resources: ["pods"]
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: ""
      resources: ["endpoints", "services"]

  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*"
    - "/version"

  - level: Request
    resources:
    - group: ""
      resources: ["configmaps"]
    namespaces: ["kube-system"]

  - level: Metadata
    resources:
    - group: ""
      resources: ["secrets", "configmaps"]

  - level: Request
    resources:
    - group: ""
    - group: "extensions"

  - level: Metadata
    omitStages:
      - "RequestReceived"

b. Audit Backends:  There are also auditing solutions like Falco that make it easy but here we define a Log backend and have the audit events route to the SRS gateway. This includes –audit-log-path for /var/log, --audit-log-maxage for 3 months , --audit-log-maxbackup for a number of 20 log files to be retained, and –audit-log-maxsize of 5MB each.
c. We have additional options specified to truncate logs with –audit-log-truncate-enabled
d. We use fluentd to collect and distribute audit events from log file
We install fluentd in the kube-apiserver node
We create a config file for fluentd
We start fluentd with
fluentd -c /etc/fluentd/config -vv
We start kube-apiserver with the following options:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml –audit-log-path=/var/log/kube-audit and –audit-log-format=json
We check audit for different namespaces in /var/log/audit-*.log