Today we are going to discuss log indexer deployment to container framework.  We start with a Kubernetes Daemonset that will monitor container logs and forward them to a log indexer. Any log forwarder can be used as the image for the container in the Daemonset yaml specification. Some environment variables and shell commands might be necessary to set and run on the container. There will also be a persistent volume typically mounted at /var/log. The mounted volumes must be verified to be available under the corresponding hostPaths.
In order for logging to be helpful, it is better to have the log sources differentiated. For example, the tools to deploy and monitor the app must be different from the tool to deploy and maintain the container cluster. Although forwarders and indexers can differentiate log stream, it is better to do that at the cluster level.  There are also plugins available from different log indexer product companies which support Docker logging.
The forwarder is also specific to the log product company. We need it only to forward logs. It can be run as a a Daemonset or directly on the Kubernetes nodes. The forwarder is not only a proprietary tool, it is a convenience for deployers to move lots of data reliably and securely using log product maker guidelines. Json driver and journald can be used for integration with Kubernetes. Journald is used to collect logs from those components that do not run inside a container. For example, kubelet and the controller runtime which is usually Docker will write to journald when the host has systemd enabled. Otherwise, they write to .log files in /var/log directory. Klog is the logging library used by such system components.
At this point, it should be important to mention that the collector does not only collect logs.
The log product event collector needs all of the following:
1) logging
2) Metadata/objects
3) Metrics
One indexer is sufficient for a three node cluster with thirty containers generating 1000 messages/second each even when the message size can be a mix of small (say 256 byte) and large (1KB).
Timber is another example of a log product. The use of this product typically entails logstash, elasticsearch, and Kibana where the elasticsearch is for api access and the kibana is for web user interface. Any busybox container image can be used to produce logs which we can use as test data for our logging configuration.
The logrotate tool rotates the log once the side exceeds a given threshold.
The typical strategies for pushing logs include the following:
1) use a node-level logging agent that runs on every node. For example, Stackdriver logging on Google cloud platform and ElasticSearch on conventional Kubernetes clusters.
2) include a dedicated sidecar container for logging in an application pod.
3) Push logs directly in the backend from within an application.