The native k8s events can also be transformed to custom events to suit the need of any other event processing engine. Typically, organizations have their own event gateway and event stores for making them proprietary such as for the use of dial home, network operations center and remote diagnostic sessions. This ability to transform events then let us do without reserving large storage as long as there is some buffering possible from the source.
It is this notion that can be extended to Extract-Transform-Load operations suitable to different downstream systems.
The difference between log forwarding and event forwarding becomes clear when the use of command line options for kube-apiserver is considered. For example, the audit-log-path option dumps the ausit events to a log file that cannot be accessed from within the Kubernetes runtime environment within the cluster. Therefore this option cannot be used with FluentD because that us a containerized workload. On the other hand, the audit-web-hook option allows the service to listen for callbacks from the Kubernetes control plane to the arrival of audur events. These service listening from Falco for this web hook endpoint is running in its own container as a Kubernetes service. The control plane makes only web request per audit event and since the events are forwarded over the http, the Falco service can efficiently handle the rate and latency of traffic.
The performance consideration between the two options is also notable. The log forwarding is the equivalent of running the rail command on the log file and forwarding it over TCP as netcat command. This utilizes the sand amount of data in transfers and uses a TCP connection although it does not traverse as many layers as the web hook. It is also suitable for Syslog drain that enables further performance improvements
The Webhook command is a push command and requires packing and unpacking of data as it traverses up and down the network layers. There is no buffering involved on the service side so there is a chance that some data will be lost as service goes down. The connectivity is also subject to faults more than the Syslog drain. However the http us best suited for message broker intake which facilitates filtering and processing that can significantly improve performance.

The architecture of the Kubernetes has its control plane over network and storage available over infrastructure providers.

The components above are facilitated with the use Pivotal Container Service (PKS) which helps us migrate the same production stack across core infrastructure. Consequently, the security aspects of the production stack are dependent on the PKS and Kubernetes features and we have to reach out to the Kubernetes apiserver for auditing information from the containerized workloads.
The architecture is standard for reviewing any workloads hosted on Kubernetes. In particular, let us note the use of a distributed key-value database within the Kubernetes control plane. This database is the ‘etcd’ and it is used to maintain the cluster. ‘etcd’ is written in Go and uses the Raft consensus algorithm to manage highly-available replicated log.
Any distributed key-value database could do and it may even have benefits if the database can be offloaded from the control plane. If this cluster database could be object storage, it will continue to provide the durability and reliability while bringing some of the storage best practice.
The database is internal to the Kubernetes control plane so it does not really within the scope of this document. However, the events from the Kubernetes execution environment do pass through the layers. K8s events are noted for their format, labels and content. They help with monitoring, troubleshooting and for subsequent analysis from storage.
The native k8s events can also be transformed to custom events to suit the need of any other event processing engine. Typically, organizations have their own event gateway and event stores for making them proprietary such as for the use of dial home, network operations center and remote diagnostic sessions. This ability to transform events then let us do without reserving large storage as long as there is some buffering possible from the source.
It is this notion that can be extended to Extract-Transform-Load operations suitable to different downstream systems.

Audit events originate from the Kube-apiserver usually running on the master VM in the PKS Kubernetes cluster.

There are essentially only two considerations:
First, we define the audit policy and the webhook which is passed as the Yaml file locations to the kube-apiserver in the form of command-line arguments. [These command-line options are explained here: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/]. We can also include these options in the kube-apiserver configuration.

Second, we restart the kube-apiserver to use the specified policy and webhook. Changing the configuration file automatically restarts the kube-apiserver.

The steps to setup auditing so that events can be analyzed later, include the following:

1) ssh admin@<pks-apiserver> # such as "ssh ubuntu@opsman.environment.local"

2) ubuntu@opsmanager-2-5:~$ sudo -i # pks and bosh commands are run from an elevated privilege account

3) pks login -a pks-api.environment.local -u -p -k  # this let us view and use the pks cluster

4) pks cluster <cluster_name> | grep UUID # this lets us get the UUID for the cluster. The convention for naming service instance is usually service-instance_UUID. You can replace the service instance name with whatever format suits the name.

5) bosh vms -d service-instance_874b838b-6391-4c62-991b-3e1528a4b37e # this lets us use the service instance  to display the vms. Usually there will be only one master. The kube-apiserver runs on this master.

6) bosh scp service-instance_874b838b-6391-4c62-991b-3e1528a4b37e master/b9a8aa9f-0e31-4579-8e4b-685c55a80f0e audit-policy.yaml :/var/vcap/jobs/kube-apiserver/config/audit-policy.yaml # we copy the audit policy file locally to the VM where the kube-apiserver runs.

7) bosh ssh service-instance_874b838b-6391-4c62-991b-3e1528a4b37e master/b9a8aa9f-0e31-4579-8e4b-685c55a80f0e -c ' echo "--audit-policy-file=/var/vcap/jobs/kube-apiserver/config/audit-policy.yaml " >> /var/vcap/jobs/kube-apiserver/config/kube-apiserver.yaml' # here we update the configuration of the kube-apiserver with the policy file path. This is the input to the auditing system.

8) bosh ssh service-instance_874b838b-6391-4c62-991b-3e1528a4b37e master/b9a8aa9f-0e31-4579-8e4b-685c55a80f0e -c ' echo "--audit-log-path=/var/vcap/sys/log/kube-apiserver/audit.log" >> /var/vcap/jobs/kube-apiserver/config/kube-apiserver.yaml' # here we update the configuration of the kube-apiserver with the log path. This is the output to the auditing system.

We were discussing Falco for auditing.
The actions of uploading the policies and web hook and requiring the kube-apiserver to restart varies from site to site where the Kubernetes cluster is hosted. If the cluster is hosted on the PKS, the actions taken are different from those for minkube. In this case the BOSH cli is used.

This cli can be used when we have the ip address of the coordinator and the credentials to use it are set via environment variables.
The commands are:
1. ssh ubuntu@<pks-api-server>
2. pks login -a <pks-api-server> -u <admin> -p <password> --skip-ssl-validation
3. pks cluster <cluster-name> # which gives the k8s cluster ID
4. bosh vms -d service-instance_<k8s cluster ID> # which gives the vms for the cluster
5. bosh -d service-instance_<k8s cluster ID> ssh <VM CID> # corresponding the vm for the master

6. Then we run the commands to bosh scp to upload the audit policy
7. the bosh scp command to upload the web hook declaration
8. The bosh scp command to copy the apiserver-config.patch.sh script file which causes the kube-apiserver to restart when the configurations audit-webhook-config-file and audit-dynamic-configuration changes in /etc/kubernetes/manifests/kube-apiserver.yaml

The audit logs then become available at /var/lib/k8s-audit/audit.log

We describe the steps taken to use Falco for auditing on Kubernetes:
1) Deploy Falco to your Kubernetes cluster
2) Define your audit policy and webhook configuration
3) Restart the API Server to enable Audit Logging
4) Observe Kubernetes audit events at Falco

1) can be done with the help of a chart from stable/Falco
helm install --name my-release stable/Falco
Note Rbac is enabled with :

kubectl create -f k8s-with-rbac/falco-account.yaml
serviceaccount "falco-account" created
clusterrole "falco-cluster-role" created
clusterrolebinding "falco-cluster-role-binding" created
k8s-using-daemonset$

A service is created that allows other services to reach the embedded webserver in falco via port 8765:
k8s-using-daemonset$ kubectl create -f k8s-with-rbac/falco-service.yaml
service/falco-service created

2)  The webhook is installed with:
webserver:
   enabled: true
   listen_port: 8765
   k8s_audit_endpoint: /k8s_audit
   ssl_enabled: false
   ssl_certificate: /etc/falco/falco.pem

The Kubernetes audit is configured with a Kube-config file and kube-apiserver as follows:
cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
kind: Config
clusters:
- cluster:
    server: http://<ip_of_falco>:8765/k8s_audit
  name: falco
contexts:
- context:
    cluster: falco
    user: ""
  name: default-context
current-context: default-context
preferences: {}
users: []
EOF
And the kube-server is launched with the following api commands:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig

Rules can be customized with k8s_audit_rules.yaml file.

3)
A script enable-k8s-audit.sh performs the necessary steps of enabling dynamic audit support for the apiserver by modifying the apiserver  command line to add `--audit-dynamic-configuration`, `--feature-gates=DynamicAuditing=true`, etc. arguments, etc.
The same script can be modified to add a default log backend with commandline arguments:--audit-log-path, --audit-log-format, --audit-log-truncate-enabled and --audit-policy-file


4) Kubernetes audit events will then be routed to the falco daemonset within the cluster
Verify that the falco was setup correctly with
kubectl logs -l app=falco

This is a summary of the book: “An Elegant puzzle” written by Will Larson.
The author has a blog that is also a good read. This book is about the discipline of engineering management presented with an engineering driven approach to solving some of the management problems. The emphasis on engineering here is not about formula or theory as much as it is a bias for action, implementation and re-assessment. This approach includes analysis and metrics not necessarily in that order all the time. It proposes fairness in every step of improvement to foster an inclusive environment.
The author presents two common threads that run through every chapter of the book. First, a framework is used to capture the insights presented. This framework can be customized, instrumented, quantitatively evaluated and iterated for continuous improvement in a variety of contexts. Second, the book provides readers with minimal process improvements that do not interfere with their day to day activities. Tools and techniques proposed in this book serve this purpose.
There are seven chapters including Organizations, Tools, Approaches, Culture and Careers.  The chapters are intended for a broad audience.
Organization deals with walking the tightrope between managing risks and sustaining productivity given the winds of change.
Tools is a chapter dedicated to applying systems thinking – a veritable theory that has other books written on this topic. It involves a model-based approach to describing and solving engineering management problems. It benefits the practitioner with incremental progress towards chosen goal.
The Approaches chapter is dedicated to implementing the frameworks described in the Tools chapter. It brings up the differences between the management versus engineering thinking by suggesting that management is in fact a moral profession because it is an opportunity and obligation to the empowerment of those we work with and who work for us. It goes on to suggest that the practice of management has a lasting effect on people.
The chapter on Culture poses it as a double-edged sword where it can vary between mild to extreme indulgence. Some aspects of engineering culture are painstakingly cultivated and preserved such as innovation and accountability. The absence of culture has been brought up as the absence of scaffolding that can result in pitfalls.
The continuous education of workforce is recommended and the representation of diversity in the discipline is sought to be improved with nuanced and results-oriented processes.
The book is careful not to sell any of the proposals as silver bullets and encourages the readers to track the progress of an application over time.
The final chapter on career is about establishing a ladder with continuous replenishment at the base and the opportunity to foster growth by way of change that can be influenced.
As I write this summary, I have to throw in the disclosure that I have never been a manager and have no idea what it brings with formal designation as opposed to practicing without it. The book is guiding and readers can hope to benefit immediately from a results-oriented action plan.

Deployment models for software on Kubernetes
Each deployment model has its advantages and disadvantages. The on-premise model is focused on deploying distributed containers to on-premise servers while securing storage centrally. The cloud-based model is focused on deploying the instance that can meet the monitoring and logging requirements of the cloud environment.
In this section, we cite the challenges and considerations associated with each deployment model for running workloads in production. In the following sections, we describe the container security, Kubernetes deployments, and the network security. Each layer of the deployment may either be self-managed or fully managed service and comes with its best practice.
1. Standalone On-Premise model:
The standalone deployment of any software in Kubernetes comes with a requirement to automate the initialization of the Kubernetes cluster required to host the deployed instance. This installer comes with a documentation that describes the software included in the package, their version and the minimal steps needed to get the instance up and running for the first time. The installation is already secure out of the box in adhering to the product security guidelines and has been analyzed for container image security, web application, network intrusion security among others. The administrator can then secure the product specific features listed in this guide for both the resource requirements of the application and the Kubernetes execution environment of the cluster hosting the application in a two later srack if application over cluster.
2. Cloud-based security model:
The cloud-based deployment of the same software on Kubernetes also comes with a requirement to automate the deployment of the application but on a PKS cluster so that it can be deployed the same to the public or private cloud. PKS allows us to use the same automation to initialize the cluster regardless of the site where it is deployed. However, there are quite a few differences between cloud-based deployment and a standalone deployment that span layers and shared components which require administrator involvement to customize the configuration before first use. 
Both deployments require configuration of the application with administrator involvement before users can begin analyzing streams. This calls out activities such as artifact repository configuration, metrics and monitoring setup, diagnosability and logging configuration as well as settings for scaling the capacity to meet the forecasted demand from the deployed instance.